Nix and NixOS on the DGX Spark

Try DGX Spark playbooks using Nix on DGX OS, or install NixOS on your DGX Spark for the full Nix experience. The repository provides USB images and a NixOS module with settings for DGX Spark systems.

This works on the NVIDIA DGX Spark itself and also on the Asus Ascent GX10.

See my 5 minute lightning talk from Planet Nix for an intro: https://youtu.be/AvK_gi_snJE?si=MPKv3iiuS9B5elIE

Using Nix on DGX OS (Ubuntu)

You can use the dev shells and playbooks in this repo on NVIDIA DGX OS (Ubuntu) without installing NixOS.

Setup

1. Install Nix using the official installer:

   sh <(curl -L https://nixos.org/nix/install) --daemon
   

   Alternatively, you can use the Determinate Nix Installer:

   curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
   

2. Enable flakes and nix-command by adding to /etc/nix/nix.conf:

   experimental-features = nix-command flakes
   

3. Enable the graham33 Cachix cache — see Caching below.

Running on Non-NixOS (e.g. DGX OS)

On non-NixOS systems, Nix-built CUDA applications need nix-gl-host to find the host GPU drivers. The playbook devshells handle this automatically — container-based playbooks don't need it, and Nix-native playbooks wrap their commands with nixglhost so no manual intervention is required.

For other devshells (e.g. cuda), nixglhost is available and can be used to prefix commands manually:

nix develop .#cuda
nixglhost deviceQuery

NixOS on the DGX Spark

[!WARNING] Only DGX OS can boot from the factory firmware. You need to update firmware before installing NixOS.

USB Boot Image

Build the USB image:

nix build .#usb-image
sudo dd if=$(echo result/iso/*.iso) of=/dev/your_usb_disk_device bs=1M status=progress
sync

The image includes two kernel options, selectable from the GRUB boot menu:

• NixOS (default) - NVIDIA's specialised kernel for DGX Spark with full GPU support and working Ethernet
• NixOS (standard-kernel) - Standard NixOS 6.17 kernel (Ethernet has problems)

Booting

Disable Secure Boot in the DGX Spark BIOS and boot from the USB drive.

You can then follow the installation instructions in the NixOS manual: https://nixos.org/manual/nixos/stable/#sec-installation-manual

Using the DGX Spark module

This module provides configurable DGX Spark hardware support with options for kernel selection.

Module configuration options

hardware.dgx-spark = {
  enable = true;                 # Enable DGX Spark hardware support
  useNvidiaKernel = true;        # Use NVIDIA kernel (default: true)
  connectx7Hotplug = true;       # Enable ConnectX-7 hot-plug (default: true)
};

Using NVIDIA kernel

hardware.dgx-spark.enable = true;  # Uses NVIDIA kernel by default

The NVIDIA kernel is a custom build optimised for NVIDIA DGX Spark systems. The kernel configuration is generated from NVIDIA's Debian annotations and compared with NixOS defaults to produce a minimal, maintainable configuration.

The module also enables the DGX Dashboard web interface at http://localhost:11000, providing GPU telemetry and system monitoring.

Using standard NixOS kernel (has some issues with networking)

hardware.dgx-spark = {
  enable = true;
  useNvidiaKernel = false;       # Use standard NixOS 6.17 kernel
};

Kernel configuration management

The kernel configuration is generated from NVIDIA's Debian annotations and stored in kernel-configs/nvidia-dgx-spark-<version>.nix. This terse configuration only contains options that differ from NixOS defaults, reducing verbosity by ~82%.

To regenerate the kernel configuration:

nix run .#generate-kernel-config

This:

1. Fetches the NVIDIA kernel source from GitHub
2. Builds the NixOS baseline kernel config
3. Compares NVIDIA's annotations with NixOS defaults
4. Generates a terse config file with only the differences

Regeneration is needed when:

• NVIDIA kernel version changes (update kernel-configs/nvidia-kernel-source.nix)
• NixOS common-config changes (nixpkgs update)

You can also use a local kernel source for development:

nix run .#generate-kernel-config -- --kernel-source /path/to/NV-Kernels

Importing in other projects

Other projects can import this flake and use the DGX Spark module:

{
  inputs.dgx-spark.url = "github:graham33/nixos-dgx-spark";

  outputs = { nixpkgs, dgx-spark, ... }: {
    nixosConfigurations.mySystem = nixpkgs.lib.nixosSystem {
      modules = [
        dgx-spark.nixosModules.dgx-spark
        {
          # Enable DGX Spark support
          hardware.dgx-spark.enable = true;
          # Optionally use standard kernel: useNvidiaKernel = false;
        }
        # your other modules
      ];
    };
  };
}

Quick start NixOS template

For a complete NixOS configuration template specifically designed for DGX Spark systems, you can use the template:

# Create a new directory for your NixOS configuration
mkdir my-dgx-spark-config
cd my-dgx-spark-config

# Initialise with the DGX Spark template
nix flake init -t github:graham33/nixos-dgx-spark#dgx-spark

This creates a complete NixOS configuration with:

• flake.nix - Flake configuration that imports the DGX Spark module
• configuration.nix - Main system configuration optimised for DGX Spark
• hardware-configuration.nix - Hardware configuration template

Customising the template

After initialising the template, you need to:

1. Generate hardware configuration and update template:

   # Generate hardware config to a temporary location to get the real UUIDs
   sudo nixos-generate-config --root /mnt --dir /tmp/nixos-config
   
   # Copy the real hardware UUIDs and settings from the generated file
   # Replace the placeholder UUIDs in hardware-configuration.nix with actual
   # values from /tmp/nixos-config/hardware-configuration.nix
   

2. Edit configuration.nix to customise:

   • Change hostname from dgx-spark to your preferred name
   • Update username from nixos to your preferred username
   • Add your SSH public keys for remote access
   • Set your timezone and locale preferences
   • Add any additional packages you need

3. Deploy the configuration to /etc/nixos:

   # Copy your configuration to /etc/nixos
   sudo cp -r . /etc/nixos/
   
   # Apply the configuration
   sudo nixos-rebuild switch --flake /etc/nixos#dgx-spark
   

Firmware updates

As noted in #32, only DGX OS can boot from the factory firmware. If NixOS can't boot from the factory firmware, you need to update firmware from DGX OS first.

The module enables fwupd for firmware updates. NVIDIA publishes DGX Spark firmware to the Linux Vendor Firmware Service (LVFS). To check for available updates:

fwupdmgr get-updates

To install available updates:

fwupdmgr update

Playbooks

This repository includes devshells for NVIDIA DGX Spark playbooks from https://build.nvidia.com/spark:

¹ Pre-installed on DGX OS ² Full Nix: Fully reproducible — all dependencies installed via Nix ³ Container: Nix provides podman, but containers are pulled/built at runtime ⁴ Nix + Container: CLI tools packaged via Nix, but containers are managed by the tool at runtime

Caching

Flox CUDA binary cache (recommended)

Flox distributes pre-built CUDA packages for aarch64-linux with NVIDIA's permission. This includes cudatoolkit, nccl, cuDNN, PyTorch, and other CUDA dependencies — dramatically reducing build times.

If you use the DGX Spark NixOS module, the Flox cache is configured automatically as a substituter.

For standalone Nix (e.g. on DGX OS), add to /etc/nix/nix.conf:

extra-substituters = https://cache.flox.dev
extra-trusted-public-keys = flox-cache-public-1:7F4OyH7ZCnFhcze3fJdfyXYLQw/aV7GEed86nQ7IsOs=

Matching the Flox cache with cudaCapabilities

The DGX Spark module sets nixpkgs.config.cudaCapabilities = [ "12.0" "12.1" ] so CUDA code is compiled only for the Spark's GB10 Blackwell GPU. Without this, packages such as ucc compile for all nine architectures nixpkgs supports (sm_75 through sm_121), which can exhaust memory and trigger OOM kills during a rebuild.

The trade-off is that Flox builds its cache with nixpkgs' default (full) capability list. Restricting the list changes the derivation hash of every CUDA-dependent package, so none of them match the Flox cache any more and they build from source instead — including large leaf packages that only pull CUDA in transitively, such as Firefox and Thunderbird.

If you would rather have the cache hits, restore the default list in your own configuration:

{
  hardware.dgx-spark.enable = true;

  # Compile CUDA code for every capability nixpkgs supports, matching the
  # builds in the Flox cache.
  nixpkgs.config.cudaCapabilities = [ ];
}

An empty list does not mean "no capabilities" — it means "unset", so nixpkgs falls back to its default list.

Which setting is better depends on what you build. Keep the module default if you compile CUDA packages that Flox does not ship (the OOM risk is real on a 128 GB Spark); use [ ] if your CUDA packages come from the cache and you want to avoid rebuilding them and their dependents.

[!NOTE] The playbook devShells and packages in this repo are built with the same cudaCapabilities = [ "12.0" "12.1" ] as the module, so they share build results with the system closure rather than rebuilding every CUDA derivation at a second capability list. That still matches neither Flox default above, so they are served from the graham33 Cachix cache instead.

12.0 stays in the list even though the Spark's GB10 is SM 12.1. PyTorch only appends its arch-specific sm_120a/sm_121a gencode flags — which is how the FP8 GEMM in RowwiseScaledMM.cu reaches the Spark — when the existing flags already mention compute_120, and flashinfer gates most of its Blackwell modules on sm120 as well. Plain 12.0 was never wrong either: a cubin built for compute capability X.y runs on any device X.z where z >= y, so sm_120 SASS already executes natively on sm_121.

graham33 Cachix cache

For packages built by this repo (e.g. dgx-dashboard, openshell):

cachix use graham33

Install cachix first if needed: https://docs.cachix.org/installation

See https://nixos.wiki/wiki/CUDA for general CUDA caching details.

nixos-anywhere (Experimental)

[!WARNING] This has not been tested yet. Use at your own risk.

You can install NixOS on a DGX Spark remotely using nixos-anywhere. This is useful for headless setups where you have SSH access to the target machine.

nix run github:nix-community/nixos-anywhere -- --flake github:graham33/nixos-dgx-spark#dgx-spark root@<ip>

This partitions the NVMe disk and installs NixOS with the DGX Spark module enabled. You may want to customise nixos-anywhere/configuration.nix (e.g. to add SSH keys or change the hostname) — clone the repo and point --flake at your local checkout instead.

To test the disk configuration in a VM without installing:

nix run github:nix-community/nixos-anywhere -- --flake .#dgx-spark --vm-test

See the nixos-anywhere documentation for full details and requirements.

License

MIT License - see LICENSE for details.