Nix on Kimsufi
I just bought a new Kimsufi server and wanted to setup nixos on it. It was a struggle. I figured I would document my process for the next traveler.
This process is easy, it took me forever due to some gotchas:
- My Kimsufi box was bios only, no uefi
-
The boot partition had to live on
/dev/sda
for some reason
I don’t know if this is the case on all Kimsufi boxes but if you are currently stuck and can’t figure out whats wrong, like I was, try those.
The plan
Nixos anywhere is a project for deploying nixos… anywhere. The idea, as I understand it, is to take a box which is already provisioned with linux, build a nixos configuration on a second computer, and then use nixos configuration to take over the box. This option is appealing to me due to it’s universality.
- Boot the server in rescue mode
- Grab the hardware info
- Create a nixos config locally
- Do some nixos anywhere magic to replace debian with my config
Grab the hardware info
Once the server is booted into rescue mode I needed two bits of information: the nixos hardware config and the harddisks ids1. To get the nixos hardware configuration we are going to need to get into a nixos installer some how. Luckily there are kexec-tarballs which will let us hijack the rescue mode. To do this we need to copy over an ssh key so that we can still connect after the hijacking:
ssh-copy-id root@<server>
Then we connect to the server and run
export INSTALLER=https://github.com/nix-community/nixos-images/releases/download/nixos-unstable/nixos-kexec-installer-noninteractive-x86_64-linux.tar.gz
curl -L "$INSTALLER" \
| tar -xzf- -C /root /root/kexec/run
Once that’s done we are in a nixos installer. As we are going to use disko to generate our partitions we don’t need to generate a filesystem config and can generate our hardware config with:
nixos-generate-config --show-hardware-config --no-filesystems \
> hardware.nix
We can also nab the disks ids with the following command.
find /dev/disk/by-id -name "wwn*" \
| grep -v part
Keep track of which id is /dev/sda
, Kimsufi seems to require that the boot
partition lives on this drive.
Creating the nixos config
I am going to avoid going too much into detail on how to configure nixos itself. Here are some starter configs I found helpful when learning. Really the only thing I’m going to point out is how I did my disk config. Disko needs some help with their documentation.
Here is my complete disk layout. Most of this is just how I like to do things with zfs. However:
- That 1Mb “EF02” partition needs to exist as disko only does gpt and kimsufi only does bios
- The boot partition and “EF02” have to be on “/dev/sda” because… reasons I guess.
let
disk-id = id: "/dev/disk/by-id/${id}";
d1 = disk-id "wwn-0x5000cca25ed3025e";
d2 = disk-id "wwn-0x5000cca25ed2e8e8";
pool = "tank";
in {
disko.devices = {
disk = {
d1 = {
type = "disk";
device = d1;
content = {
type = "gpt";
partitions = {
# This gives grub somewhere to save data
# this is needed cause disko only does gpt
boot = {
size = "1M";
type = "EF02";
};
# Grub can use ext4, but why risk it
esp = {
size = "1G";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/boot";
};
};
zfs = {
size = "100%";
content = {
inherit pool;
type = "zfs";
};
};
};
};
};
d2 = {
type = "disk";
device = d2;
content = {
type = "gpt";
partitions = {
zfs = {
size = "100%";
content = {
inherit pool;
type = "zfs";
};
};
};
};
};
};
zpool = {
"${pool}" = {
type = "zpool";
mode = "mirror";
rootFsOptions = {
mountpoint = "none";
compression = "on";
xattr = "sa";
acltype = "posix";
"com.sun:auto-snapshot" = "false";
};
# Take a snapshot of the empty pool
# this will let us delete darlings
postCreateHook = """
zfs list -t snapshot -H -o name \
| grep -E '^${pool}@blank$' \
|| zfs snapshot ${pool}@blank
""";
datasets = {
"system/root" = {
type = "zfs_fs";
mountpoint = "/";
options.mountpoint = "legacy";
};
"local/nix" = {
type = "zfs_fs";
mountpoint = "/nix";
options = {
mountpoint = "legacy";
# Nix does not use atime (impure)
# might as well turn it off
atime = "off";
};
};
"user/home" = {
type = "zfs_fs";
mountpoint = "/home";
options.mountpoint = "legacy";
};
};
};
};
};
}
Debugging steps
A lot of my time was wasted with due to poor ability to debug, here’s what ended up helping me find the root of the problem.
Kvm/ipmi
My biggest mistake here was wasting hours without setting up Kimsufi’s kvm. I
would deploy the box and it wouldn’t respond to network requests. It was
impossible to differentiate “this box isn’t booting” from “this box has a
misconfigured network”. The kvm lets you see the monitor of the server and
differentiate these failuer modes. To run the kvm download the Java applet to
kvm.jnlp
and run:
nix shell \
nixpkgs#adoptopenjdk-icedtea-web \
--command javaws kvm.jnlp
Logs
If things are booting and you would rather have a pleasant shell you can always drop the box back into its recovery mode, mount whatever you need to, then check the logs with:
journalctl -D /mnt/var/log/journal
Other potential strategies
OVH supports custom installs through something they call BYOI (Bring Your Own Image). This could be a good fit if you don’t care about disko managing your partitions. It looks like you partition the drives through their webui then give them an image and they load it. You can generate images with nixos-generators, maybe even whatever images BYOI is expecting?
Other notes
Everyone else went in depth with their network setups, I did not. Whatever nixos does out of the box now worked for me.
- https://mgdm.net/weblog/nixos-on-kimsufi/
- https://acelpb.com/posts/2015-09-26-Install-NixOS-on-So-You-Start-dedicated-server/
- https://www.codejam.info/2015/12/installing-nixos-on-a-kimsufi.html
- https://web.archive.org/web/20160829180041/http://aborsu.github.io/2015/09/26/Install%20NixOS%20on%20So%20You%20Start%20dedicated%20server/
- https://lewo.abesis.fr/posts/install-nixos-on-kimsufi/
-
You don’t need to do this… I had a bad experience once with a computer changing it’s mind on which drive
/dev/sda
was so now I try to stick to ids.YMMV.↩︎