Kubernetes development cluster bootstrapping with Nix packages

This project aims to provide single dependency Kubernetes clusters for local testing, experimenting and development purposes.

Moving pictures are worth more than thousand words, so here is a short demo:

Nix?

Have you ever heard about Nix, the functional package manager?

In case you haven’t, don’t worry, the important thing is that it provides all the third-party dependencies needed for this project, pinned to a dedicated version. This guarantees stable, reproducible installations.

KuberNix itself is a Rusty helper program, which takes care of bootstrapping the Kubernetes cluster, passing the right configuration parameters around and keeping track of the running processes.

What is inside

The following technology stack is currently being used:

Some other tools are not explicitly mentioned here, because they are no first-level dependencies.

Boot Phases

Components start in four sequential phases. Within each phase, components start in parallel. If any component in a phase fails, subsequent phases are skipped.

graph TD
    A["Phase 1: Infrastructure"] --> B["Phase 2: ControlPlane"]
    B --> C["Phase 3: Controller"]
    C --> D["Phase 4: NodeAgent"]

    A --- etcd
    B --- apiserver
    C --- scheduler & controller-manager & cri["cri-o or containerd (x N)"] & proxy
    D --- kubelet["kubelet (x N)"]

Single Dependency

With Nix

As already mentioned, there is only one single dependency needed to run this project: Nix (with flakes enabled). To setup Nix, simply run:

$ curl https://nixos.org/nix/install | sh

Please make sure to follow the instructions output by the script and that flakes are enabled in your Nix configuration.

Getting Started

Cluster Bootstrap

To bootstrap your first cluster, download one of the latest release binaries or build the application via:

$ make build-release

The binary should now be available in the target/release/kubernix directory of the project. Alternatively, install the application via cargo install kubernix.

After the successful binary retrieval, start KuberNix:

$ kubernix

KuberNix will now take care that the Nix environment gets correctly setup, downloads the needed binaries and starts the cluster. Per default it will create a directory called kubernix-run in the current path which contains all necessary data for the cluster.

When running as a non-root user, KuberNix uses rootlesskit to provide a user namespace where all components run as fake root. This requires unprivileged user namespaces (kernel.unprivileged_userns_clone=1) and cgroup v2 with user delegation. On Ubuntu 23.10+, you may also need to set kernel.apparmor_restrict_unprivileged_userns=0. The cluster runs in its own network namespace, which slirp4netns connects to the outside world. Only the API server port (6443) is published to 127.0.0.1 on the host, so NodePort Services and host ports are not reachable from the host, and the cluster cannot reach services listening on the loopback interface of the host. The --cidr must not overlap with the slirp4netns network 10.0.2.0/24. Kernel modules cannot be loaded explicitly from the user namespace. The kernel loads the netfilter modules for the pod network and kube-proxy on demand, otherwise they have to be loaded on the host (for example iptable_nat and nf_conntrack).

In multi-node rootless mode, all nodes run as direct processes (no container isolation) with --hostname-override for node identity. Running with sudo gives container-based multi-node isolation.

Shell Environment

If everything went fine, you should be dropped into a new shell session, like this:

[INFO ] Everything is up and running
[INFO ] Spawning interactive shell
[INFO ] Please be aware that the cluster stops if you exit the shell
>

Now you can access your cluster via tools like kubectl:

> kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-85d84dd694-xz997   1/1     Running   0          102s

All configuration files have been written to the target directory, which is now the current one:

> ls -1
apiserver/
controllermanager/
coredns/
crio/          # or containerd/ when using --cri-runtime=containerd
encryptionconfig/
etcd/
kubeconfig/
kubelet/
kubernix.env
kubernix.toml
nix/
pki/
policy.json
proxy/
scheduler/

For example, the log files for the different running components are now available within their corresponding directory:

> ls -1 */*.log */*/*.log
apiserver/kube-apiserver.log
controllermanager/kube-controller-manager.log
crio/<node>/crio.log  # or containerd/<node>/containerd.log
etcd/etcd.log
kubelet/<node>/kubelet.log
proxy/kube-proxy.log
scheduler/kube-scheduler.log

The CRI runtime and kubelet data is stored per node, where <node> is the hostname for a single node cluster and node-0, node-1, … in multinode mode. If a component runs inside a node container, then its log file is named after the container runtime, for example crio/node-1/podman.log.

If you want to spawn an additional shell session, simply run kubernix shell in the same directory as where the initial bootstrap happened.

$ kubernix shell
[INFO ] Spawning new kubernix shell in: 'kubernix-run'
> kubectl run alpine --image=alpine -it --rm -- sh
If you don't see a command prompt, try pressing enter.
/ #

This means that you can spawn as many shells as you want to.

Cleanup

The whole cluster gets automatically destroyed if you exit the shell session from the initial process:

> exit
[INFO ] Cleaning up

Please note that the directory where all the data is stored is not being removed after the exit of KuberNix. This means that you’re still able to access the log and configuration files for further processing. If you start the cluster again, then the cluster files will be reused. This is especially handy if you want to test configuration changes.

Restart

If you start KuberNix again in the same run directory, then it will re-use the configuration during the cluster bootstrapping process. This means that you can modify all data inside the run root for testing and debugging purposes. The startup of the individual components will be initiated by JSON files called run.json, which are available inside the directories of the corresponding components. For example, etcd gets started via:

> cat kubernix-run/etcd/run.json

{
  "command": "/nix/store/…-etcd-3.6.13-bin/bin/etcd",
  "args": [
    "--advertise-client-urls=https://127.0.0.1:2379",
    "--client-cert-auth",
    "--data-dir=/…/kubernix-run/etcd/run",
    "--listen-client-urls=https://127.0.0.1:2379",
    "--name=etcd",
    "…"
  ]
}

The generated Nix environment in the nix directory is an exception: it gets regenerated if it differs from the one of the running KuberNix version, for example after an upgrade or when the content of the overlay changed. The multinode base image is tagged with a hash of its inputs (kubernix:base-<hash>), so it gets rebuilt as well. Old images can be removed via podman rmi. All other component configurations and the kubernix.toml are reused as they are, so remove the run root to pick up new defaults, like the runc runtime handler.

Configuration

KuberNix has some configuration possibilities, which are currently:

Please ensure that the CIDR is not overlapping with existing local networks and that your setup has access to the internet. The CIDR will be automatically split up over the necessary cluster components.

OCI Runtimes

Both crun and runc are registered as runtime handlers named crun and runc for CRI-O as well as containerd, each with its own state directory (<cri>/<node>/crun and <cri>/<node>/runc). The --oci-runtime option selects the handler used for pods without a runtime class, while pods can select the other one via a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: runc
handler: runc

If the other runtime cannot be found, then KuberNix warns and does not register its handler, while a missing default runtime is an error. This works in single node, multinode and rootless mode. The container runtime used for the node containers in multinode mode (--container-runtime) is not affected by this option.

Multinode Support

It is possible to spawn multiple worker nodes, too. To do this, simply adjust the -n, --nodes command line argument as well as your preferred container runtime via -u, --container-runtime. The default runtime is podman, but every other Docker drop-in replacement should work out of the box.

Overlays

Overlays provide a method to extend and change Nix derivations. This means, that we’re able to change dependencies during the cluster bootstrapping process. For example, we can exchange the used CRI-O version to use a local checkout by writing this simple overlay.nix:

self: super: {
  cri-o = super.cri-o.overrideAttrs(old: {
    src = ../path/to/go/src/github.com/cri-o/cri-o;
  });
}

Now we can run KuberNix with the --overlay, -o command line argument:

$ kubernix --overlay overlay.nix
[INFO ] Nix environment not found, bootstrapping one
[INFO ] Using custom overlay 'overlay.nix'
these derivations will be built:
  /nix/store/…-cri-o-1.36.2.drv
  building '/nix/store/…-cri-o-1.36.2.drv'...

Using this technique makes it easy for daily development of Kubernetes components, by simply changing it to local paths or trying out new versions.

Additional Packages

It is also possible to add additional packages to the KuberNix environment by specifying them via the --packages, -p command line parameter. This way you can easily utilize additional tools in a reproducible way. For example, when to comes to using always the same Helm version, you could simply run:

$ kubernix -p kubernetes-helm
[INFO ] Nix environment not found, bootstrapping one
[INFO ] Bootstrapping cluster inside nix environment
…
> helm version
version.BuildInfo{Version:"v4.2.0", ...}

All available packages are listed on the official Nix index.

Contributing

You want to contribute to this project? Wow, thanks! So please just fork it and send me a pull request.