Nix on Kimsufi

I just bought a new Kimsufi server and wanted to setup nixos. It was a struggle. I figured I would document my process for the next traveler.

This process is easy, it took me forever due to some gotchas:

My Kimsufi box was bios only, no uefi
The boot partition had to live on /dev/sda for some reason

I don’t know if this is the case on all Kimsufi boxes but if you are currently stuck and can’t figure out whats wrong, like I was, try those.

The plan

Nixos anywhere is a project for deploying nixos… anywhere. The idea, as I understand it, is to take a box which is already provisioned with linux, build a nixos configuration on a second computer, and then use nixos configuration to take over the box. This option is appealing to me due to it’s universality.

Boot the server in rescue mode
Grab the hardware info
Create a nixos config locally
Do some nixos anywhere magic to replace debian with my config

Grab the hardware info

Once the server is booted into rescue mode I needed two bits of information: the nixos hardware config and the harddisks ids¹. To get the nixos hardware configuration we are going to need to get into a nixos installer some how. Luckily there are kexec-tarballs which will let us hijack the rescue mode. To do this we need to copy over an ssh key so that we can still connect after the hijacking:

ssh-copy-id root@<server>

Then we connect to the server and run

export INSTALLER=https://github.com/nix-community/nixos-images/releases/download/nixos-unstable/nixos-kexec-installer-noninteractive-x86_64-linux.tar.gz
curl -L "$INSTALLER" \
	| tar -xzf- -C /root /root/kexec/run

Once that’s done we are in a nixos installer. As we are going to use disko to generate our partitions we don’t need to generate a filesystem config and can generate our hardware config with:

nixos-generate-config --show-hardware --no-filesystems \
	> hardware.nix

We can also nab the disks ids with the following command.

ls -l /dev/disk/by-id/ \
	| grep wwn \
	| grep -v part

Keep track of which id is /dev/sda, Kimsufi seems to require that the boot partition lives on this drive.

Creating the nixos config

I am going to avoid going too much into detail on how to configure nixos itself. Here are some starter configs I found helpful when learning. Really the only thing I’m going to point out is how I did my disk config. Disko needs some help with their documentation.

Here is my complete disk layout. Most of this is just how I like to do things with zfs. However:

That 1Mb “EF02” partition needs to exist as disko only does gpt and kimsufi only does bios
The boot partition and “EF02” have to be on “/dev/sda” because… reasons I guess.

let
  disk-id = id: "/dev/disk/by-id/${id}";
  d1 = disk-id "wwn-0x5000cca25ed3025e";
  d2 = disk-id "wwn-0x5000cca25ed2e8e8";
  pool = "tank";
in {
  disko.devices = {
    disk = {
      d1 = {
        type = "disk";
        device = d1;
        content = {
          type = "gpt";
          partitions = {
            # This gives grub somewhere to save data
            # this is needed cause disko only does gpt
            boot = {
              size = "1M";
              type = "EF02";
            };
            # Grub can use ext4, but why risk it
            esp = {
              size = "1G";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            zfs = {
              size = "100%";
              content = {
                inherit pool;
                type = "zfs";
              };
            };
          };
        };
      };
      d2 = {
        type = "disk";
        device = d2;
        content = {
          type = "gpt";
          partitions = {
            zfs = {
              size = "100%";
              content = {
                inherit pool;
                type = "zfs";
              };
            };
          };
        };
      };
    };
    zpool = {
      "${pool}" = {
        type = "zpool";
        mode = "mirror";
        rootFsOptions = {
          mountpoint = "none";
          compression = "on";
          xattr = "sa";
          acltype = "posix";
          "com.sun:auto-snapshot" = "false";
        };
        # Take a snapshot of the empty pool
        # this will let us delete darlings
        postCreateHook = """
          zfs list -t snapshot -H -o name \
            | grep -E '^${pool}@blank$' \
            || zfs snapshot ${pool}@blank
        """;

        datasets = {
          "system/root" = {
            type = "zfs_fs";
            mountpoint = "/";
            options.mountpoint = "legacy";
          };
          "local/nix" = {
            type = "zfs_fs";
            mountpoint = "/nix";
            options = {
              mountpoint = "legacy";
              # Nix does not use atime (impure)
              # might as well turn it off
              atime = "off";
            };
          };
          "user/home" = {
            type = "zfs_fs";
            mountpoint = "/home";
            options.mountpoint = "legacy";
          };
        };
      };
    };
  };
}

Debugging steps

A lot of my time was wasted with due to poor ability to debug, here’s what ended up helping me find the root of the problem.

Kvm/ipmi

My biggest mistake here was wasting hours without setting up Kimsufi’s kvm. I would deploy the box and it wouldn’t respond to network requests. It was impossible to differentiate “this box isn’t booting” from “this box has a misconfigured network”. The kvm lets you see the monitor of the server and differentiate these failuer modes. To run the kvm download the Java applet to kvm.jnlp and run:

nix shell \
	nixpkgs#adoptopenjdk-icedtea-web \
	--command javaws kvm.jnlp

Logs

If things are booting and you would rather have a pleasant shell you can always drop the box back into its recovery mode, mount whatever you need to, then check the logs with:

journalctl -D /mnt/var/log/journal

Other potential strategies

OVH supports custom installs through something they call BYOI (Bring Your Own Image). This could be a good fit if you don’t care about disko managing your partitions. It looks like you partition the drives through their webui then give them an image and they load it. You can generate images with Nixos-generators, maybe even whatever images BYOI is expecting?

Other notes

Everyone else went in depth with their network setups, I did not. Whatever nixos does out of the box now worked for me.

You don’t need to do this… I had a bad experience once with a computer changing it’s mind on which drive /dev/sda was so now I try to stick to ids.YMMV.↩︎