Building a NixOS MiniRack with Raspberry Pi 5s

desktop

Every time I’ve tried to self-host something, I end up with a neglected single-board computer in a dusty corner of my basement on a version of Ubuntu Server. I loved the idea of self-hosting useful apps: an IRC Bouncer, Atuin, or Actual Budget. After a month, I would start to forget the software I installed, or the little configuration quirks I made to make it work. My experiments ended with an apt upgrade or updated Docker image, requiring me to relearn everything I did to make my server work in the first place. I would lose interest quickly. I wanted something I could set up and run with minimal intervention.

In January 2025, I discovered Jeff Geerling’s YouTube channel. If you don’t know who Jeff Geerling is, they cover a smattering of tech subjects on YouTube and on their blog. I dabbled with a few small devices in the past for self-hosting, and my old copy of Kubernetes for Dummies was beckoning.

This time would be different. I’m a huge NixOS fan; I use it on anything I can. Nix allowed me to set up my machines declaratively, and I never felt like there was “random” software floating around I forgot about. I started thinking about how I could use NixOS to manage my self-hosted devices.

Naming Scheme

I wanted a cool naming scheme. I like Jupiter. Jupiter is easy because of its moons. Honestly, I think the naming scheme is a bit of a cliché, but I like it. I even 3D-printed "Jupiter" into the firewall/router tray :).

Figuring out the bare minimum

I wanted to have a place to put my experiments without resorting to something like AWS. It needed a similar developer experience—something where I could throw up my Nix-built Docker images and have a working host on my internal LAN. Finally, I wanted updates to be easier. If I’m running 3+ devices, I am not SSH’ing each time to run a script manually. It had to be updatable.

Why NixOS?

Nix's determinism and declarative configuration made it attractive for managing configuration across multiple nodes. Nix opens the door for caching, or even potentially automated deploys from a binary cache (like Cachix).

There are a few well-known “nix-ops” tools around. I reached for Colmena. I came across it on GitHub before, and it's also a Rust tool (which I always appreciate).

During this project, Raspberry Pis struggled to compile larger pieces of software. Software for aarch64-linux did not always appear in the binary cache for nixos-raspberrypi. Setting up an on-demand beefy build server on AWS to speed up deploys is a great cure for my impatience at the cost of a few dollars. NixOS makes this workflow easy.

Why K3s?

I decided K3s is the way. K3S is a lighter distribution of K8S that I run with embedded etcd. Part of this project was meant to help me learn Kubernetes, and K3S would let me run the Docker images with the workflow I expected.

Kubenix

The one thing unattractive about Kubernetes was the YAML manifest format. Spending far too much time configuring GitHub Actions scarred me. In addition to my hesitancy with YAML, using Kubernetes would mean the project is split between Nix and Kubernetes manifests. It would require extra verification steps, ensuring the parts worked well in isolation and as a whole.

This discrepancy made me wish a Nix-y way of configuring at least the platform to deploy existed. Luckily, I found the Kubenix project. Kubenix allows configuring Kubernetes state with Nix. It is first defined in Nix, then the kubenix command outputs built Kubernetes manifests. Kubenix handles pruning old state, so deployments more accurately reflect what exists on the cluster.

With Kubenix I could have the following:

This way, I enabled configuring the platform for my apps (OS Hosts, Load Balancer, Observability, and Metrics) in one language.

Versatility

Kubernetes comes with a bunch of goodies baked in. HA/Load Balancing/Self-Healing, etc. But above all, I liked the extensibility. If I wanted, I added ArgoCD for an alternate means of deployment with pure Kubernetes manifests (like my website).

Updates

The update flow I settled on is Renovate. Renovate updates Helm chart/docker image versions in Kubenix definitions. I merge those in GitHub CI, update the Nix hashes manually next time I pull/want to update, and run a single colmena build --on @homelab to build everything at once (with my AWS build server if I need it). Updating the machines becomes a ta single colmena apply --on @homelab. Updates resulting in breaking changes imply time spent fixing things. Renovate is nice since it gives a list of changes in the updated versions, so I get some warning. Having control over when updates occur, the bulk application/deployment is a huge improvement over my previous self-hosting attempts. Further improvements can be made with automated Cachix/binary cache deploys, for instance. I haven’t explored this much yet, though, as the current update flow suits my needs.

I extracted relevant bits for this post into nixos-rpi-lab from Jupiter. There’s a lot to cover here, so this is probably "part 1" of a series.

Bill of Materials

For the initial version, listed is what I ended up with this to host three Kubernetes Control nodes. The K3S control plane nodes double as worker nodes. While this uses three nodes, it is possible to run K3S on a single node if one desires. I did not have a 3D printer at this point and went with the GeeekPi rack they sell. The RPi5 piece includes an NVMe SSD adapter.

Architecture

For the first version, the architecture ended up looking like this:

 ┌─ jupiter.lan ──────────────────────────────────────────┐
 │                                                        │
 │    ┌─────────────────┐                                 │
 │    │ Jupiter         │                                 │
 │    │ Router/Firewall │                                 │
 │    └────────┬────────┘                                 │
 │             │                                          │
 │ ┌─ k3s cluster ──────────────────────────────────────┐ │
 │ │           │                                        │ │
 │ │           ▼              ┌─ control plane (k3s) ─┐ │ │
 │ │   ┌───────────────┐      │ ┌───────────────────┐ │ │ │
 │ │   │ MetalLB VIPs  │      │ │ ganymede (leader) │ │ │ │
 │ │   │ 10.10.68.1    │      │ └───────────────────┘ │ │ │
 │ │   └───────────────┘      │ ┌───────────────────┐ │ │ │
 │ │                          │ │ io                │ │ │ │
 │ │                          │ └───────────────────┘ │ │ │
 │ │                          │ ┌───────────────────┐ │ │ │
 │ │                          │ │ europa            │ │ │ │
 │ │                          │ └───────────────────┘ │ │ │
 │ │                          └───────────────────────┘ │ │
 │ └────────────────────────────────────────────────────┘ │
 └────────────────────────────────────────────────────────┘

Templates

The Nix templates consist of two portions: the lab configuration and the secrets configuration. While secrets exist encrypted with age and sops-nix, the secrets configuration is separate to enable storing the encrypted sops secrets in a private repository.

The templates include fixes and QoL improvements specific to configuring a cluster:

Set up nixos-rpi-lab in one spot and secrets in another:

nix flake init -t github:insipx/nixos-rpi-lab#lab

# secrets
nix flake init -t github:insipx/nixos-rpi-lab#secrets

Change the inputs of the lab repository to reference the secrets repository:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    nixos-raspberrypi = {
      url = "github:nvmd/nixos-raspberrypi/main";
    };
    lab-secrets = {
      # if you push it to a git repo, ensure you have
      # nix setup with your GitHub access token to clone
      # or use a git+ssh: path
      # url = "github:you/your-lab-secrets-repo";
      url = "path:/home/hunter2/path/to/secrets";
    };
    # ...
  };
}

This will get you this directory structure for the #lab.

.

├── base/                 # NixOS Config used by everything.
│                         # Modify the user config here.
├──┐deployments/          # Stuff that's deployed to the cluster
│  └── kubenix/           # Kubernetes resources defined with Kubenix
├── hive/                 # NixOS Configurations for each node
└──┐ machine-specific/    # Machine-specific NixOS Configuration
   └── rpi5/              # RPi 5 hardware and kernel settings

#secrets consists of:

├── .sops.yaml    # sops public keys/metadata
├── flake.nix     # devshell with needed tools
├── keyscan.sh    # script to scan & convert public keys to `age`
├── module.nix    # some sops options/enabling K3S secrets

We’re going to focus on how to set up the RPI’s first; leave the secrets configuration alone for now. It will be used for storing the shared k3s_token for cluster authentication.

For setting up the Raspberry Pis, we care about two things:

Setting up the Pis

For the initial installation, I was able to refer to nvmd/nixos-raspberrypi. That's the repository that built support for Raspberry Pi 5s, and includes some examples to get started in addition to this guide.

  1. Build and flash the NixOS Raspberry Pi image onto the SD card.
  2. Modify the configuration in installer_images.nix to allow SSH access to the Pi after the SD image has booted.
  3. Use nixos-anywhere to apply the initial configuration, which includes the NVMe file system configuration.
  4. Remove the SD card to force the Pi to boot from NVMe. This is important for debugging. If something goes wrong, we can insert the original SD image to access a Nix chroot environment and fix it.

The SD image we build is initialInstall and includes all the NixOS configuration modules in base/ and machine-specific/rpi5. This includes the custom kernel from nvmd/nixos-raspberrypi and other quality-of-life packages. This explicitly excludes filesystem.nix, which is set up in the next step with Disko.

On this step, ensure you modify 'base' to include SSH keys and other configuration you might want on the Pis, such as the specific terminfo package for your favorite terminal emulator or a preferred text editor. The SD card will be reusable for any further Raspberry Pis you set up.

Once ready, the commands are roughly:

# build the image, from the #lab flake repository
# `accept-flake-config` ensures use of the binary caches defined in the template.
# thats the nixos-community cache and the nixos-raspberry pi cache.
# if you prefer compiling the kernel on your own, omit the flag.
nix build --accept-flake-config .#sdImages.initialInstall

# extract the zstd to an `.img` file somewhere
zstd -d result/sd-image/nixos-installer-rpi5-kernel.img.zst -o nixos-installer-rpi5-kernel.img

# Flash to an SD card; ensure the `/dev/sdX` path is correct.
sudo dd if=nixos-installer-rpi5-kernel.img of=/dev/sdX bs=4M status=progress conv=fsync

Inserting the SD card and starting the RPI (with a screen attached), you will see a screen that lists a user to log in, an IP address, and other information. In the past, I’ve found the IP by checking DHCP leases for anything new. You should be able to SSH, which is important for the nixos-anywhere command.

nixos-anywhere

Once the SD card is installed, we need to make use of the connected NVMe drive. K3S can be write-heavy; an NVMe will last longer than an SD card. While this setup would work with SD cards, expect performance degradation.

The configuration uses Disko for initial disk setup. The setup is almost completely verbatim to the example in nixos-raspberrypi. The one difference is important, because it enables Longhorn.

Towards the bottom of filesystem.nix, an ext4 volume is defined. The homelab module allows configuring the volume size (on initial install). The size is defined in the rpiHomeLab module. Longhorn does not work with ZFS. Instead, we need to create an ext4 volume on ZFS use it. This can be skipped if the install is a pure SD-card install or if you choose a filesystem other than ZFS. Just make sure Longhorn also supports this other filesystem. I like ZFS for its abilty to automate backups with snapshots, so that's why I'm using it here.

{
  disko.devices.zpool.rpool.datasets."longhorn-ext4" = {
    type = "zfs_volume";
    content = {
      type = "filesystem";
      format = "ext4";
      mountpoint = "/var/lib/longhorn";
      mountOptions = [
        "noatime"
        "discard"
      ];
    };
  };
}

Apply the initial NixOS configuration and set up the NVMe external disk with ZFS by using nixos-anywhere.

nixos-anywhere --flake .#initialInstall root@your_node_ip

This will install and reboot. Then you can remove the SD card and reboot into the NixOS install from the NVMe.

The HomeLab NixOS Module

Much of the per-node configuration exists in the homelab module I created for configuring nodes. The documentation lists available options. Hive configuration has some default values set. An example configuration looks like

{
  rpiHomeLab = {
    networking = {
      interface = "end0";
      hostId = "a1b2c3d4"; # Use a unique value for each node.
      hostName = "worker-01";
      address = "10.10.69.20/22";
    };

    k3s = {
      enable = true;
      leader = false;
      agent = true;
      leaderAddress = "https://10.10.69.10:6443";
      longhorn = true;
      longhornDiskSize = "800G"; # Adjust for this node's available storage.
    };
  };
}

longhornDiskSize should be changed depending on the nodes' available storage. Leave some room for the OS. In this case, 800G gives 800 gigabytes for Longhorn and leaves ~200 gigabytes for the OS. Longhorn itself can be disabled with longhorn = false. This will avoid creating the ext4 ZFS volume.

The module is meant to simplify per-node configuration and is loaded with some fixes to make a K3s setup on Raspberry Pis with ZFS work.

Deploying with Colmena

Per-node configuration exists in the hive. Control nodes are tagged control, and each node gets a homelab tag. Build everything with

colmena build --on @homelab

# Or build a single node with
colmena build --on node1

# Apply/deploy the nodes with
colmena apply --on @homelab

# or for a single node
colmena apply --on node1

Ensure each node has a user with your SSH key on it, setting it in the Colmena configuration.

{
  deployment = {
    targetHost = "node1.lab.lan";
    targetUser = "user";
    tags = [
      "homelab"
      "control"
    ];
  };
}

At this point, Colmena should be able to build/apply to nodes. Ensure that k3s is disabled, however, since K3s requires extra steps to bootstrap the secret correctly. Secrets set to true also will not be used unless k3s.enable = true, so no worries there about exposing anything K3S-related by accident.

Setting up Secrets

In order for k3s to work, it needs a shared token for nodes to join the network. Since in this guide we are setting up three control plane nodes, they will all have access to the server token.

Bootstrapping the K3s Secret and Leader Node

Before we do anything, we need to set up the leader with a secret, then copy the full server token k3s generates from that leader, and replace k3s_token in our SOPS secrets with that token.

Set up SOPs secrets repository

At the start, we initialized a secrets repository but didn’t use it. In .sops.yaml of the secrets repository, add the public keys to encrypt with. It looks like this:

# This example uses YAML anchors which allows reuse of multiple keys
# without having to repeat yourself.
# Also see https://github.com/Mic92/dotfiles/blob/d6114726d859df36ccaa32891c4963ae5717ef7f/nixos/.sops.yaml
# for a more complex example.
keys:
  - &admin age1REPLACE_WITH_YOUR_ADMIN_PUBLIC_KEY
  - &node_1 age1REPLACE_WITH_NODE_1_HOST_KEY # ./keyscan.sh node1
  - &node_2 age1REPLACE_WITH_NODE_2_HOST_KEY # ./keyscan.sh node2
  - &node_3 age1REPLACE_WITH_NODE_3_HOST_KEY # ./keyscan.sh node3
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
      - age:
          - *admin
          - *node_1
          - *node_2
          - *node_3

I store my keys on YubiKeys and use age-plugin-yubikey to get a usable public key from them. SSH keys or other key schemes work too. Sops-nix has an abundance of documentation on the subject.

Scanning for public keys

Before we can encrypt anything, we need to know what to encrypt it to. The secrets flake template includes a keyscan utility. It can be used with nix run .#keyscan -- $HOST. If my $HOST is 192.168.1.100, keyscan may return something like this:

➜ nix run .#keyscan -- 192.168.1.100
skipped key: got ssh-rsa key type, but only ed25519 keys are supported
age1q9420838rcwr3mykp3juf3f5ntwwudndlmeyxmaanr5ekqmgzdvshjph7z

age1q9420838rcwr3mykp3juf3f5ntwwudndlmeyxmaanr5ekqmgzdvshjph7z goes in the keys: section of .sops.yaml, and the name node_1 goes into key_groups. This will indicate to encrypt anything in path_regex with the given public keys. For instance, the YAML would look like the following:

# This example uses YAML anchors which allows reuse of multiple keys
# without having to repeat yourself.
# Also see https://github.com/Mic92/dotfiles/blob/d6114726d859df36ccaa32891c4963ae5717ef7f/nixos/.sops.yaml
# for a more complex example.
keys:
  - &admin age1REPLACE_WITH_YOUR_ADMIN_PUBLIC_KEY
  - &node_1 age1q9420838rcwr3mykp3juf3f5ntwwudndlmeyxmaanr5ekqmgzdvshjph7z #./keyscan.sh 192.168.1.100
  - &node_2 age1REPLACE_WITH_NODE_2_HOST_KEY # ./keyscan.sh node2
  - &node_3 age1REPLACE_WITH_NODE_3_HOST_KEY # ./keyscan.sh node3
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
      - age:
          - *admin
          - *node_1
          - *node_2
          - *node_3

It's important to update the keys after modifying .sops.yaml. Otherwise, the encrypted ciphertext will not be updated. Before we update keys, run sops secrets/homelab.yaml just to have something for SOPS to encrypt.

sops secrets/homelab.yaml # this will open a file with $EDITOR. just save & close.
sops updatekeys

And of course, commit it. Once committed, run nix flake update $secrets-input in the lab repository.

The full flow might look like this:

nix run keyscan -- $one_of_my_node_ips # get the age public key
$EDITOR .sops.yaml # update the key for $node
sops updatekeys # repeat for all nodes
git commit -am "add node_$X key" && git push origin main
cd $lab_repository
nix flake update $secrets_input

Re-installing a node with nixos-anywhere regenerates its host key, so the node will need to be rescanned (with keyscan) and the secrets will need to be re-encrypted (sops updatekeys). This workflow counts on the host key generated during install, so there is no copying keys between hosts. Private keys never leave the host they were generated on.

Once you have the keys, you have to create the secrets.

sops secrets/homelab.yaml

That will open the file with SOPS, and it looks like this:

hello: Welcome to SOPS! Edit this file as you please!
example_key: example_value

# Example comment
example_array:
  - example_value1
  - example_value2
example_number: 1234.56789
example_booleans:
  - true
  - false

Generate the initial K3S token.

The initial secret for the leader may be generated:

k3s token generate
# or without k3s
openssl rand -hex 16

For this guide, let’s use e68487be15c9df0bd91e7353740c44a0 as the secret (do not reuse this for obvious reasons). This is the value that will go into the secrets/homelab.yaml file.

Before encryption, homelab.yaml will look like:

k3s_token: e68487be15c9df0bd91e7353740c44a0

Saving and closing that file will encrypt it. Commit and push, or point the labs repository containing the flake.nix to the absolute path of the secrets repository.

In the hive template, the lab-secrets module should be imported. In the configuration for each node,

{
  lab-secrets.settings.k3s = true;
}

Enables access to those secrets.

For the bootstrap, we are setting up the leader node first. Enable k3s and leader = true for the node you choose to be a leader.

A full leader configuration in Colmena will look like:

{
  # ...
  node1 = _: {
    imports = [
      ./../machine-specific/rpi5
    ]
    ++ commonImports;
    deployment = {
      targetHost = "node1.lab.lan";
      targetUser = "user";
      tags = [
        "homelab"
        "control"
      ];
    };
    rpiHomeLab = {
      networking = {
        hostId = "00000000";
        hostName = "node1";
        address = "10.10.10.10/22";
        interface = "end0";
      };
    };
    rpiHomeLab.k3s.leader = true;
    rpiHomeLab.k3s.enable = true;
    rpiHomeLab.k3s.longhorn = true;
    rpiHomeLab.k3s.longhornDiskSize = "25G";
    lab-secrets.settings.k3s = true;
    services.k3s.extraFlags = [
      "--tls-san node1.lab.lan"
      "--tls-san node1"
      "--tls-san 10.10.10.10"
    ];
  };
}

Run the apply

colmena apply --on node1

Now we need to retrieve the token K3S generates. This will be prepended with a CA hash. It will look like this:

K10<CA hash>::<credentials>

Retrieve it with SSH and copy it:

ssh user@node1

sudo cat /var/lib/rancher/k3s/server/token

Copy the token into k3s_token of the secrets/homelab.yaml. Commit and push.

Enabling K3s

The leader address is in base/default.nix towards the bottom. Only one node is the leader, and it should be the same across all nodes. Set the address to the node that we just set up as the leader.

base/default.nix

{
  rpiHomeLab = {
    k3s.leaderAddress = "https://node1.lab.lan:6443";
  };
}

Setting the rest of the nodes to k3s = enable will allow them to join the cluster. Copy /etc/rancher/k3s/k3s.yaml from the leader node to a local ~/.kube/config to use kubectl with the cluster. Once copied, ensure the localhost address is changed from 127.0.0.1 to point to a node in the cluster.

You made it! The cluster should be working!

Verify the state with kubectl get nodes. You should see a list of nodes and their ready state. For instance, my output looks like the following:

➜ kubectl get nodes
NAME       STATUS     ROLES                       AGE    VERSION
amalthea   Ready      <none>                      250d   v1.35.6+k3s1
callisto   Ready      <none>                      309d   v1.35.6+k3s1
elara      Ready      <none>                      97d    v1.35.6+k3s1
europa     Ready      control-plane,etcd,master   309d   v1.35.6+k3s1
ganymede   Ready      control-plane,etcd,master   309d   v1.35.6+k3s1
io         Ready      control-plane,etcd,master   309d   v1.35.6+k3s1
lysithea   Ready      <none>                      244d   v1.35.6+k3s1

All nodes should be ready. A NotReady node means something failed or was not configured correctly.

Configuring with Kubenix

The templates include configuration for the following:

Configuration exists in deployments/kubenix.

Note that ServiceLB and the default K3S-managed traffic are disabled with the Homelab NixOS module configuration. This is replaced by MetalLB and the Traefik configuration in Kubenix.

Before applying anything, modify the Metal LB Configuration with an IPAddressPool

On my setup, I created a VLAN for 10.10.69.0/22. This allows me to give Metal LB multiple address pools. The DHCP server hands out leases for physical machines in a limited range in the 10.10.69.0/24 subnet. That gives me 10.10.68.0/24 for my internal app endpoints and 10.10.70.0/24 for external. Ensure you include the IP address pools that you want. This should give more than enough space for apps on a home network, but choose the subnet prefix and IP address scheme that suits you.

Applying the Configuration

Build the configuration with

nix run .#kubenix

Render the manifests with

nix run .#kubenix -- render

Apply the manifests with

# This will prune resources that were removed from the previous apply
nix run .#kubenix -- apply --all

After a kubenix apply, things should be running! I like to use the k9s CLI to look at my namespaces/pods.

Running k9s with different namespaces should result in a view of successfully running pods.

k9s -n kube-system
k9s -n longhorn-system
k9s -n metallb-system

desktop

Future posts:

This post is part of a series. It covers initial setup and bootstrap of a k3s cluster deployed with NixOS and Kubenix. Future posts could cover the following: