Hardening Home Infrastructure

January 18, 2026·

For the past two years, I’ve been proudly running a “cluster” based on k3s (a lightweight Kubernetes distribution). The setup was minimalistic, yet almost elegant :

One control plane with 8 cores, 64 GB of RAM and a casual 1 TB NVMe SSD;
One worker node with 8 cores (and the performance of a sleepy cow 🐄), 16 GB of RAM and a 1 TB HDD for that authentic “retro datacenter” feel.

Up until now, it worked. Which is to say: nothing was on fire most of the time, but “working” and “being a good idea” are, as it turns out, two very different concepts.

Here’s why this setup slowly but surely started to feel like self-inflicted pain :

Aging OS and painful upgrades The operating system was getting older and increasingly resistant to updates. Since only the cloud provider’s pre-baked images were available, upgrades felt less like maintenance and more like a leap of faith.
Eye-watering costs The overall infrastructure price started to really be prohibitive. We can do better for cheaper price.
High Availability? Never heard of her. One control plane, no redundancy. If it sneezed, the whole cluster caught a cold. Very resilient. While I don’t need enterprise-grade uptime, I’d still like to get some redundancy on my services to limit downtime. This redundancy should also account for disaster recovery. If the whole cluster fails for some reason (like datacenter destruction), then I need to be able to shortly recover by spinning a new cluster, without (or limited) data loss.
Secret management and node provisioning from hell Managing secrets was… inexistent. Provisioning machines involved a delightful ritual of YAML, incantations and screaming at Ansible.

For all of those very good, totally avoidable reasons, I eventually did the only reasonable thing : I burned it down and started over from scratch. I managed to cut costs by almost 40% while getting better performances and improved (not to say existing) redundancy. This also prevents me some emotional damage maintaining this whole mess.

Apparently it’s possible to make things cheaper, faster, and less terrifying at the same time (as long as you’re willing to admit that the original setup was a learning experience and not a long-term strategy).

The new setup uses 3 control planes (6 cores, 18 GB of RAM and 1 TB SSD each) and one satellite node (1 core, 2 GB of RAM and 1 TB old fashion HDD) for now. This is mostly useful for data redundancy at a very low cost.

The overall goal is to automate as much as possible to reduce the maintenance burden.

Cluster Setup

Originally, all my machines were provisioned using Ansible. Which, in hindsight, was a bold experiment in self-inflicted suffering.

Despite all the YAML and “Infrastructure as Code” buzzwords, the syntax is way too complex for my needs. I remember having troubles with the documentation finding what I needed. So the very first thing on my cleanup list was simple: get rid of Ansible. Managing it felt like maintaining a second infrastructure just to keep the first one alive. I could list a mile-long set of issues I see with Ansible but it’s outside the scope of this blog post.

The initial alternative was a rolling-release distro, something like Arch Linux. On paper, it’s great : modern packages, constant updates, and a strong “I know what I’m doing” aesthetic. But once you factor in provisioning multiple machines in a row (and possibly scaling out later) the charm fades quickly. That was supposed to be Ansible’s job. Unfortunately, Ansible was too busy being… Ansible.

Then came NixOS : declarative configuration, highly reproducible systems, atomic upgrades, rollback to a previous configuration when things explode, etc. Basically, the promised land.

One concern is that NixOS doesn’t strictly follow the Filesystem Hierarchy Standard (FHS), which can theoretically wreak havoc with certain applications. As I like pain, I decided to give it a try anyway. Annoyingly enough, it works perfectly fine. No chaos. No disasters. No mysterious breakage. Just some tiny workarounds, and clean, reproducible machines that behave exactly the same every time.

NixOS

To really make things properly, obsessively reproducible, I went all in on Nix Flakes. And no, this isn’t just another shiny Nix feature to feel superior on the Internet, it actually solves real problems. It’s still flagged as experimental but honestly, it’s stable enough to be used safely. With Flakes, everything is pinned :

The exact version of nixpkgs;
All inputs (system configs, modules, overlays, etc.);
And most importantly, a lockfile that freezes the entire universe in time.

That lockfile is the real hero here. It guarantees that rebuilding a machine today, tomorrow, or six months from now will produce the exact same system, not a “close enough” approximation depending on which mirrors felt cooperative that day. No surprises, no drifting configs, no “works on my machine” moments.

Of course, I still want recent packages. I’m not running a museum. So I’m using the unstable channel. Which, in classic Nix fashion, is about as unstable as a well-oiled industrial machine. Honestly, it should just be called rolling-release. Things get updated fast, yes, but they also get tested, and in practice it’s been rock solid. The name alone does more damage to its reputation than the code ever has.

Combined with Flakes, this gives me the best of both worlds : modern software(s), fully deterministic builds, and the comforting knowledge that if an update ever does go sideways, rolling back is literally one command away.

The only actually annoying thing about NixOS is that most cloud providers still pretend it doesn’t exist. It’s hard to find providers with official images, but I was expecting this. NixOS is a bit of a niche system. Thankfully, nixos-anywhere exists. It’s a tool that enables unattended installation of NixOS on remote machines via SSH. It works by using kexec to boot a temporary RAM-based NixOS environment.

The workflow is gloriously pragmatic :

Spin up a machine with something boring and universally supported (say Debian 13 or any other distribution).
Immediately betray it.
Replace everything with NixOS, remotely, without ceremony, using nixos-anywhere.

To keep things sane and avoid duplicating configuration everywhere, I also simplified the setup by storing machine-specific data in a single JSON file. Hostnames, IPs, roles, etc, all the boring but necessary details live there. It’s a bit easier to manipulate JSON data than Nix syntax, especially when automating things in scripts. The Nix configuration uses a regular flake.nix file.

hosts.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "instance_1": {
    "hostname": "instance_1",
    "system": "x86_64-linux",
    "profile": "virtual",
    "role": "server",
    "region": "france",
    "zone": "france-paris",
    "ip": "<ip>",
    "gateway": "<gateway>",
    "maskLength": 24
  }
  // ...
}

flake.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
{
  description = "Infrastructure using NixOS";

  inputs = {
    # NixOS official package source, using the rolling release channel
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    disko = {
      url = "github:nix-community/disko";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    sops = {
      url = "github:Mic92/sops-nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs =
    {
      # self,
      nixpkgs,
      disko,
      sops,
      ...
    }:
    let
      sshKeys = [
        # List of my public SSH keys
      ];
      hosts = builtins.fromJSON (builtins.readFile ./hosts.json);
      allIps = map (host: host.ip) (builtins.attrValues hosts);
      ipsWhitelist = allIps ++ [
        # Extra IPs that I want to whitelist in the firewall settings
      ];
      dnsServers = [
        "8.8.8.8"
        "8.8.4.4"
      ];
      configure =
        {
          ip,
          gateway,
          maskLength,
          hostname,
          system,
          profile,
          role,
          region,
          zone,
          bootstrap,
          ...
        }:
        nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
            disko.nixosModules.disko
            sops.nixosModules.sops
            # And all other custom submodules that I would need
          ];
          specialArgs = {
            inherit hostname;
            inherit role;
            inherit region;
            inherit zone;
            inherit system;
            inherit sshKeys;
            inherit serverAddress;
            inherit ipsWhitelist;
            inherit ip;
            inherit gateway;
            inherit maskLength;
            inherit dnsServers;
          };
        };
    in
    {
      # Inject configuration for each host defined in hosts.json
      nixosConfigurations = builtins.mapAttrs (_: config: configure config) hosts;
    };
}

Since all of this runs on cloud VMs, I obviously configure the machines accordingly (i.e. means headless: no GUI, no display manager, no desktop env, nothing).

system.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
{
  modulesPath,
  ...
}:
{
  # See https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/profiles
  # for available profiles.
  imports = [
    (modulesPath + "/profiles/headless.nix")
  ];

  # Since we can't manually respond to a panic, just reboot.
  boot.kernelParams = [
    "console=ttyS0,115200n8"
    "console=tty0"
    "reboot=acpi" # Use ACPI for reboots instead of BIOS (fixes stop instead of reboot)
  ];
}

Disks

Disk setup during installation is handled automatically with Disko. Disko lets me declare the disk layout in code : partitions, filesystems, mount points; all defined upfront, versioned, and reproducible. Below is a simple example. My actual setup is a bit more complex and uses disk encryption.

disks.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{
  ...
}:
let
  device = "/dev/sda";
in
{
  disko.devices.disk.sda = {
    device = device;
    type = "disk";
    content = {
      type = "gpt";
      partitions = {
        MBR = {
          priority = 0;
          size = "1M";
          type = "EF02";
        };
        ESP = {
          priority = 1;
          size = "500M";
          type = "EF00";
          content = {
            type = "filesystem";
            format = "vfat";
            mountpoint = "/boot";
          };
        };
        root = {
          priority = 2;
          size = "100%";
          content = {
            type = "filesystem";
            format = "ext4";
            mountpoint = "/";
          };
        };
      };
    };
  };

  boot.loader.grub = {
    devices = [ "nodev" ]; # Don't install GRUB to any device for UEFI
    efiSupport = true;
    efiInstallAsRemovable = true;
  };
}

Network

Firewall management is handled the old-fashioned way : iptables. No fancy abstractions, no “next-gen” firewall. Just raw rules and a healthy amount of paranoia. It’s not that I wouldn’t like something a bit more modern. I have to follow NixOS standards. This is the way.

I deliberately keep as few ports open as humanly possible. The default stance is simple : everything is hostile until proven otherwise. By default, that means SSH, and that’s it. Because I still need to get in when things inevitably break. On control plane nodes, I also open HTTP, HTTPS and DNS ports. My domain name is configured to point to all control planes at once, using good old round-robin DNS. Nothing fancy, but it does the job. All the Kubernetes-specific ports are open only between cluster nodes and exposed to a very limited set of my own IPs.

network.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
{
  hostname,
  role,
  ipsWhitelist,
  ...
}:
let
  isServer = role == "server";
in
{
  networking = {
    hostName = hostname;
    firewall = {
      enable = true;
      # Public ports accessible from anywhere
      allowedTCPPorts = (
        [
          22 # SSH
        ]
        ++ (
          # HTTP, HTTPS for servers
          if isServer then
            [
              80 # HTTP
              443 # HTTPS
            ]
          else
            [ ]
        )
      );
      # DNS for servers
      allowedUDPPorts = if isServer then [ 53 ] else []; # DNS
      allowPing = true;

      # Custom rules for restricted ports
      extraCommands = ''
        # Define trusted IPs (only focusing on IPv4)
        TRUSTED_IPS="${builtins.concatStringsSep " " (map (ip: "${ip}/32") ipsWhitelist)}"

        # Allow metrics from trusted IPs only
        for ip in $TRUSTED_IPS; do
          iptables -A nixos-fw -p tcp --dport 10250 -s $ip -j nixos-fw-accept
        done

        ${
          if isServer then
            ''
              # Allow k8s and etcd ports from trusted IPs only
              for ip in $TRUSTED_IPS; do
                iptables -A nixos-fw -p tcp --dport 2379 -s $ip -j nixos-fw-accept
                iptables -A nixos-fw -p tcp --dport 2380 -s $ip -j nixos-fw-accept
                iptables -A nixos-fw -p tcp --dport 6443 -s $ip -j nixos-fw-accept
                iptables -A nixos-fw -p tcp --dport 50443 -s $ip -j nixos-fw-accept
                iptables -A nixos-fw -p udp --dport 3478 -s $ip -j nixos-fw-accept
              done
            ''
          else
            ''
              # Allow Wireguard from trusted IPs only
              for ip in $TRUSTED_IPS; do
                iptables -A nixos-fw -p udp --dport 51820 -s $ip -j nixos-fw-accept
                iptables -A nixos-fw -p udp --dport 51821 -s $ip -j nixos-fw-accept
              done
            ''
        }
      '';
    };
    nameservers = [
      "8.8.8.8"
      "8.8.4.4"
    ];
    # NOTE : Wireguard is already in the default kernel packages on NixOS 21.05+
    wireguard.enable = true; # Flannel backend for k3s
  };
}

Ideally, all of this should sit behind a VPN, and ideally there’d be a proper VIP in front of the control planes (something like kube-vip). Unfortunately, “ideally” and “I actually took time to do it” are two very different states of being. Setting up a clean VPN+VIP combo is not exactly straightforward, and for now, the setup lives in that uncomfortable but familiar place : secure enough, functional, and on the TODO list.

Of course, SSH needs to be configured properly, because leaving it “default” on the Internet is basically an invitation for chaos. Password authentication is disabled. Root login is prohibited.

ssh.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
  ipsWhitelist,
  ...
}: {
  services = {
    fail2ban = {
      enable = true;
      bantime-increment = {
        enable = true;
        factor = "4";
        rndtime = "8m";
      };
      # NixOS already protect SSH with fail2ban when log level is set to VERBOSE so it should be okay
      ignoreIP = [ "127.0.0.1" ] ++ ipsWhitelist;
    };
    openssh = {
      enable = true;
      settings = {
        PermitRootLogin = "no";
        PasswordAuthentication = false;
        LogLevel = "VERBOSE";
        X11Forwarding = false;
      };
    };
  };
}

For certain cloud providers, I had to disable DHCP, because apparently some of them think it’s fun to randomly reassign IPs during system installation.

Kubernetes

I chose k3s for my Kubernetes distribution. It’s lightweight and easy to setup and use. This is the perfect Kubernetes distribution for a personal cluster.

k3s is supported out-of-the-box in NixOS, so I don’t have much to do. I use a predefined secret token for the cluster. I’ll discuss a bit more on that point later. Since I point my domain on all my control planes, I use that address as the server address, so I have some built-in redundancy. Otherwise, I would have a single point of failure pointing at a single node.

k3s.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
  pkgs,
  hostname,
  role,
  region,
  zone,
  serverAddress,
  config,
  clusterTokenPath,
  ...
}:
let
  isServer = (role == "server") || (role == "init");
  clusterInit = role == "init";
  defaultFlags = [
    "--node-label \"topology.kubernetes.io/region=${region}\""
    "--node-label \"topology.kubernetes.io/zone=${zone}\""
  ];
in {
  services = {
    k3s = {
      enable = true;
      package = pkgs.k3s;
      tokenFile = clusterTokenPath;
      role = if isServer then "server" else "agent";
      inherit clusterInit;
      serverAddr = if clusterInit then "" else "https://${serverAddress}:6443";
      extraFlags = (
        defaultFlags
        ++ (
          if isServer then
            [
              "--flannel-backend wireguard-native"
              "--tls-san ${serverAddress}"
            ]
          else
            [ ]
        )
      );
      gracefulNodeShutdown.enable = true;
    };
  };
}

At some point, I thought it would be a great idea to run control plane nodes on HDD-backed machines because they were a bit cheaper. Spoiler alert : it was an absolute disaster. etcd hates slow disks (which is used by k3s for HA). With HDDs, it was constantly choking, timing out, and generally having an existential crisis. k3s nodes endlessly restarting.

Since I switched to SSD nodes (regular SSD, not NVMe ones), nodes have been stable for weeks. Immediate relief. Pure joy.

Moral of the story : don’t cheap out on disks for your control plane. SSDs aren’t an optimization here, they’re a basic requirement if you want your cluster to stay alive and not actively resent you. To be fair, HDDs aren’t evil by definition. On bare metal, with dedicated disks and no noisy neighbors, regular HDDs could be perfectly fine (as long as there isn’t too much disk pressure). In that context, etcd could probably survive, but not in cloud VMs (shared storage, unpredictable latency, etc.).

Persistent Volumes

Some of my applications are stateful, like databases. That means I need persistent volumes in Kubernetes. And since I’ve gone through all this trouble to build an HA setup, it would be nice if my data didn’t vanish the moment a node sneezes.

So the requirements pile up quickly:

Replication across nodes, because data loss is rude;
Regular backups (daily or more) to S3, in case everything catches fire;
Encryption at rest, for both volumes and backups, because plaintext is a crime.

A good candidate that matches all my requirements is Longhorn. It’s easy to install and operate, and it does replication, snapshots, backups, and encryption without demanding a PhD in distributed storage.

I did also look at Rook Ceph. On paper, it looks incredibly powerful. In practice, the learning curve is steep, the setup is heavy, and my cluster would have quickly turned into a storage science experiment. It would have allowed me to mount object storage (S3) as volumes, but with questionable performance and a lot more operational overhead. Given that my current setup is already cheap enough, I really didn’t feel like complicating my life further just to say “I run Ceph”.

Longhorn hits the sweet spot : good enough performance, strong guarantees, sane backups, encrypted data, and most importantly, far less pain.

Of course (because there’s always a catch), Longhorn didn’t work perfectly out of the box. The main issue is that Longhorn very much assumes it’s running on a system that respects the Filesystem Hierarchy Standard (FHS). NixOS politely ignores that assumption and does its own thing. Nothing dramatic though. You also have to remember to install everything needed for disk encryption on the host side. Again, nothing insurmountable. You just need to think about it, and NixOS will do the rest.

volumes.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
{
  pkgs,
  hostname,
  ...
}: {
  # Required by Longhorn for LUKS encrypted volumes
  boot.kernelModules = [ "dm_crypt" ];

  environment.systemPackages = with pkgs; [
    cryptsetup # needed for dm-crypt volumes

    # Add support for NFS filesystem
    nfs-utils # Required for NFS mounts (and Longhorn)

    # Add support for LUKS encryption (and logical volumes)
    cryptsetup
    lvm2
  ];
  boot.supportedFilesystems = {
    nfs = true;
  };
  services = {
    openiscsi = {
      # Required by Longhorn
      enable = true;
      name = "${hostname}-initiatorhost";
    };
    rpcbind.enable = true;
  };
  # Required to avoid issues with Longhorn as NixOS doesn't use FHS paths.
  # See : https://github.com/longhorn/longhorn/issues/2166
  systemd = {
    services.iscsid = {
      serviceConfig = {
        PrivateMounts = "yes";
        BindPaths = "/run/current-system/sw/bin:/bin";
      };
      # Ensure the service waits for the socket
      after = [
        "iscsid.socket"
      ];
      requires = [
        "iscsid.socket"
      ];
    };
    tmpfiles.rules = [
      # Avoid issues with Longhorn because NixOS doesn't use FHS paths.
      # Create a symbolic link /usr/bin/mount -> /run/current-system/sw/bin/mount
      # This avoids issues with RWX volumes
      "L /usr/bin/mount - - - - /run/current-system/sw/bin/mount"
      # Likewise for cryptsetup to LUKS encryption
      "L /usr/bin/cryptsetup - - - - /run/current-system/sw/bin/cryptsetup"
    ];
  };
}

Secrets

There are, unsurprisingly, some secrets to deal with (cluster token, Longhorn encryption key, etc.), and I had absolutely no desire to turn that into yet another operational nightmare. So I went with SOPS. All secrets are encrypted directly my Git the repository. No plaintext files, no “don’t worry, it’s a private repo” coping mechanisms. They’re encrypted using my SSH private keys, which means I can decrypt them locally.

Some of these secrets (like the cluster token) need to be shared across nodes. Which means the nodes themselves also need to be able to decrypt them. So they have their own keys, allowing them to access exactly what they need.

.sops.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
keys:
  - &my_machine age1irdq4jgq134vi2aasxayzgt4uc0fa6k0datbzmzb16pbqawxxy1bfrdz72
  # Remote instances
  - &instance_1 age1y9gw4vsuwqcpeebta8eorvmqrpu1jorg4sd6zjqd20fr03mmdowrt9aos9
  - &instance_2 age1u369wgo46wfm2aw7q9w7zitxzs4ztyfsmum5szp1pzdx644y2dxol77fh7
  - &instance_3 age1zouuedww2a4xljpomtpicawj8qx9g90aa84ogz42dv2x9jqiw0c9kvpsh3
  - &instance_4 age1dc1rgltbeqjj6235uobyio9cswoa9yptr78rqkf0wptayh1q74jv1uyeyp
creation_rules:
  - path_regex: secrets\.(yaml|json|env|ini)$
    key_groups:
      - age:
          - *my_machine
          - *instance_1
          - *instance_2
          - *instance_3
          - *instance_4

In case you wonder, the AGE keys are fake of course. In reality, they are generated from my private keys or the private keys of the nodes (located at /etc/ssh/ssh_host_ed25519_key).

The result is a setup where :

Secrets live safely in version control;
Every machine can decrypt what it’s supposed to;
And the whole thing stays transparent and consistent across the cluster.

It’s clean, it’s practical, and it’s ridiculously easy to set up. All I need to do on my nodes is to tell which private key to use to encrypt the secrets and where the encrypted secrets are located.

sops.nix

{
  ...
}: {
  sops = {
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    defaultSopsFile = ../secrets.yaml;
    secrets = {
      k3s_token = { };
    };
  };
}

Applications

When it comes to deploying applications, I obviously wanted something declarative. Terraform is perfect for this job. As it’s not only for Kubernetes, I can also manage other resources (such as S3 buckets, etc.). The Terraform state lives in an S3 bucket, which means it’s accessible from anywhere and not sitting on some laptop waiting to be accidentally deleted.

On the Kubernetes side of things :

Longhorn gets its recurring backup jobs and storage classes defined declaratively;
HTTPS certificates are handled by cert-manager with Let’s Encrypt, because manually managing certs is a nightmare;
Diun keeps an eye on container image updates and politely tells me when something changed;
Metrics and logs are handled by a LGTM stack, because observability is important, but I’m not interested in paying enterprise prices for it.

I won’t go into more detail here, not because it’s secret or complicated, but because at this point it’s just… infrastructure doing its job. And let’s be honest : that’s not nearly as exciting as the journey it took to get here.

Migration

Since I decided to start over with a clean, healthy, brand-new cluster, that obviously meant one thing : migrating all my data from the old cluster. It was surprisingly fine. It just required a bit of time and bandwidth. I used pv-migrate to move persistent volumes between clusters.

The highly sophisticated migration process went like this :

Create the volumes on the new cluster.
Shut down the applications on the old cluster (because data consistency is still a thing, unfortunately).
Migrate the data. In practice, something like :

pv-migrate \
  --source-kubeconfig $HOME/.kube/<old-config> \
  --source-namespace <old-namespace> \
  --source <old-pvc> \
  --dest-kubeconfig $HOME/.kube/<new-config> \
  --dest-namespace <new-namespace> \
  --dest <new-pvc>

The moment of truth : start the applications on the new cluster.
Be happy (extremely important step, do not skip).

Updates

The whole point of going all-in on NixOS was to make updates painless. The goal wasn’t to build something clever or exotic, it was to have a system that I could :

Keep up to date, both at the OS level and for Kubernetes;
Update without fear;
And roll back instantly if an update decided to get creative.

With NixOS, updating the system is no longer an event. It’s just :

Pull the new inputs;
Rebuild;
Reboot if needed;
Move on with life.

I can either apply the updates manually, or let the system updates itself by regularly pulling from my Git repository.

Automation

Each machine simply points to the Git repository that contains its configuration. The NixOS auto-upgrade feature does the rest.

updates.nix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  hostname,
  ...
}: {
system.autoUpgrade = {
    enable = true;
    flake = "github:<user>/<repository>#${hostname}";
    allowReboot = true;
    rebootWindow = {
      lower = "22:00";
      upper = "23:59";
    };
    randomizedDelaySec = "3h";
    fixedRandomDelay = true;
    dates = "*-*-* 21:00:00";
  };
}

Of course, we could push the madness even further. The next logical step would be to add a GitHub Actions workflow that periodically updates the Flake lockfile, commits the changes, and possibly rolls those updates out to the nodes automatically. We would just need to be careful about what we do and how to avoid taking down the whole cluster.

With a bit of scripting, it’s possible to check the upcoming changes to determine if a restart of the Kubernetes service is required (or worse, a system reboot after a kernel update for example).

check.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
BUILDS_DIR="build"
BUILD_CURRENT="${BUILDS_DIR}/current-${HOSTNAME}"
BUILD_NEW="${BUILDS_DIR}/new-${HOSTNAME}"
CONFIG_PATH=".#nixosConfigurations.${HOSTNAME}.config.system.build.toplevel"

nix build "${CONFIG_PATH}" -o "${BUILD_CURRENT}"
nix flake update
nix build "${CONFIG_PATH}" -o "${BUILD_NEW}"
echo "Expected changes for '${HOSTNAME}' are :"
nix store diff-closures "${BUILD_CURRENT}" "${BUILD_NEW}"

Coordination

When updating, it’s best to do so sequentially to avoid killing the whole cluster (and possibly losing data, my biggest fear). To update a single node, I’ll likely go with the following script and only move on to the next one after the script succeeds.

update.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
TARGET="<user>@${HOSTNAME}"

nix flake update
kubectl cordon "${HOSTNAME}"
kubectl drain "${HOSTNAME}" --ignore-daemonsets --delete-emptydir-data
nixos-rebuild switch \
    --flake ".#${HOSTNAME}" \
    --target-host "${TARGET}" \
    --build-host "${TARGET}"
kubectl uncordon "${HOSTNAME}"

Conclusion

I’m pretty happy with this new setup. For the first time, it actually feels polished, professional, and just… sane. Things work the way they’re supposed to, updates don’t cause panic attacks, and I don’t feel like I’m juggling flaming YAML files anymore. Life is good.

Of course, I’ll see how it holds up over time. This isn’t some mythical “set it and forget it” fantasy. In a year, I might look back, sigh, and decide to tear everything down and start over. Who knows? That’s the fun (or terror) of running your own cluster : the next adventure is always just around the corner.

For now, though, it’s smooth sailing, and that feels incredible. See you later.

Last updated on January 18, 2026

Exploring CUDA, Threading and Async Python - Part 3 Pydantic & Polymorphism