The documenation for all hosts lives in docs/. The corresponding nixos configuration is in hosts/.

Live cluster monitoring is at grafana.dos.cit.tum.de:

• DSE-Monitoring — host reachability & backups
• DSE-FPGA — FPGA cards status
• DSE-Switch — ToR switch port state

New to NixOS?

Check out our tutorial.

New admins: Onboarding

1. Install nix (the recommended Multi-user installation is not NixOS, but only a package manager)
2. Enable flake support in nix. This effectively adds the following flags to all your nix <flags> develop-like commands: --extra-experimental-features nix-command --extra-experimental-features flakes
3. Clone the doctor-cluster-config repo, cd into it and run: nix develop. This opens a shell with additional packages available such as inv --list, sops and age.
4. To generate new admin key, run (requires age):

mkdir -p ~/.config/sops/age/
age-keygen -o ~/.config/sops/age/keys.txt

Provide the generated key to a pre-existing admin and wait for him to re-encrypt all secrets in this repo with it. The admin will now:

• add your key in pubkeys.json
• add your name to the name of admins in sops.yaml.nix
• apply those changes via inv update-sops-files

After pulling the re-encrypted secrets you can read them with sops secrets.yml.

Apply config to all servers

Choose a deployment target:

$ inv -l
Available tasks:

  add-server               Generate new server keys and configurations for a given hostname and hardware config
  build-local              Build nixos configurations locally. Use `inv build-local --hosts ryan` to build a single server
  cleanup-gcroots
  deploy                   Deploy to servers
  deploy-doctor            Deploy to doctor
  deploy-host              Deploy to a single host, i.e. inv deploy-host --host 192.168.1.2
  deploy-local             Deploy NixOS configuration on the same machine. The NixOS configuration is
  deploy-ruby              Deploy to riscv server
  document-craig           Dump craigs (switch) config to encrypted docs/hosts/craig.sops
  flake-check              Run nix checks on this repo (may need a aarch64 remote builder configured)
  generate-password        Generate password hashes for users i.e. for root in ./hosts/$HOSTNAME.yml
  generate-ssh-cert        Generate ssh cert for host, i.e. inv generate-ssh-cert astrid
  ipmi-boot-bios           Set the next boot to bios and reboot
  ipmi-powerconsumption    Measure the power consumption of our servers via IPMI. Note that this does not include all servers.
  ipmi-powercycle          Power cycle a host via IPMI
  ipmi-reboot-bmc          Reboot the BMC (IPMI firmware)
  ipmi-serial              Connect to the serial console of a server via IPMI
  print-age-key            Scans for the host key via ssh an converts it to age, i.e. inv scan-age-keys --host <hostname>
  print-tinc-key
  reboot                   Reboot hosts. example usage: fab --hosts clara.r,donna.r reboot
  run                      Run provided command on the given hosts, if no host list is provided, than the command is run on all hosts.
  update-docs              Regenerate docs for all servers
  update-host-keys          Update host ssh keys in corresponding host.yml
  update-lldp-info         Regenerate lldp info for all servers
  update-sops-files        Update all sops yaml and json files according to .sops.yaml rules

Run!

$ inv deploy

Add new users

See the step-by-step guide in docs/ADD_USER.md for adding chair members, students and reviewers.

Add new servers

For installing new servers, see Add servers.

Update system

We use flakes to manage nixpkgs versions. To upgrade use:

$ nix flake update

Then commit flake.lock.

Home-manager

To install home-manager for a user simply run:

$ mkdir -p $HOME/.config/home-manager
$ cd $HOME/.config/home-manager
$ nix flake init --template github:TUM-DSE/doctor-cluster-config#home-manager
$ $EDITOR flake.nix home.nix
$ nix run $HOME/.config/home-manager#switch

This will initiate your home-manager and will generate a file similar to the one in home/.config/nixpkgs/home.nix

Visual Studio Code Server support in NixOS

You can use this to enable support for VS Code Server in NixOS.

An example of the home.nix configured for VS Code support is shown in home/.config/nixpkgs/home.nix.

IPMI

For information about managing servers via IPMI (power control, serial console, etc.), see docs/IPMI.md.

Monitoring

Hosts are monitored here: https://grafana.thalheim.io/d/Y3JuredMz/monitoring?orgId=1

CI

All machines are build by nixbot (the standalone rewrite of buildbot-nix). The resulting builds are uploaded to https://niks3.dos.cit.tum.de from where machines can download them while upgrading.