nix-grpc-store

Status: Stable.

Remote Nix store access over gRPC instead of SSH. nix copy over a WAN link is 5.5x faster than ssh-ng:// because round trips are hidden: path-info queries are batched and NAR downloads are pipelined, so wall time is bandwidth-bound instead of latency-bound.

Why gRPC instead of ssh-ng://?

• Latency-tolerant nix copy. One RPC batches the path queries for a whole closure and one request per connection downloads all NARs as a server-driven stream. Copying 200 small paths at 50 ms RTT takes 1.3 s instead of the 11.8 s a round-trip-per-path client needs.
• Faster handshakes. A TLS handshake is much cheaper than an SSH connection setup (no subprocess, no shell, no SSH key exchange and session negotiation), and connections are multiplexed over HTTP/2, so frequent short-lived store operations start quickly.
• Standard TLS certificates. Authentication uses plain X.509 certs instead of SSH keys, so you can plug into existing PKI — for example a step-ca issuing short-lived client and server certs — and reuse load balancers, service meshes and mTLS policies that already speak HTTP/2.
• Compression. All traffic is zstd-compressed. A whole batch of paths shares a single zstd stream (one compression window across all NARs), unlike ssh's optional zlib, which made the benchmark below slower instead of faster.
• Scales to a build farm. Point builders = or --store at one grpc:// address and the daemons behind it spread the derivations over all nodes, like Hydra's queue runner but for any Nix client and without a database. See docs/farm.md.

Everything that works over ssh-ng:// works here: remote builds, nix copy, path queries, GC. The gRPC layer is a thin tunnel to the nix-daemon on the other side, with dedicated RPCs for the nix copy hot path (path info queries, bulk import, NAR download). docs/latency.md shows the round trips each operation costs as sequence diagrams.

Benchmark

nix copy

nix copy of a 101-path, 411 MB closure from the same server over a wired link with ~47 ms RTT, 10 interleaved runs each.

A static HTTP binary cache stays ahead for pure downloads, but it is download-only. The gRPC store keeps close while also handling uploads, remote builds, GC and mTLS through the same endpoint.

Reproduce with ./scripts/bench-transports.py.

Remote builds

There are two ways to build on a remote machine:

• hook: the machine is listed in builders =. For every derivation Nix spawns a fresh build-remote helper process that opens its own connections, uploads the inputs, runs the build and downloads the outputs.
• direct: the remote store is the target of the build (nix build --store 'grpc://…'). One client process drives the whole build graph over a single long-lived connection and outputs stay on the remote store.

On a chain of tiny builds over a ~50 ms WAN link, grpc is 1.4x faster than ssh-ng:// in hook mode: one streaming RPC submits a derivation and returns the result with its output path infos, where ssh-ng pays per-derivation ssh session setup. In direct mode the transports are comparable, since a single long-lived connection serves the whole build either way:

Reproduce with ./scripts/bench-builds.py.

Install

nix build

This produces result/bin/nix-grpc-daemon for the server and result/lib/nix/plugins/nix-grpc-store-loader.so for the client, which dispatches to the plugin build matching the running Nix version.

For a client outside NixOS, nix build .#nix-with-plugin gives a nix (and nix-build, nix-store, …) with the plugin already loaded. In another flake, nix-grpc-store.lib.wrapNix pkgs pkgs.nix does the same for any nixpkgs nix, for example to put into a CI image.

nix-grpc-store.lib.nixToolsWithPlugin pkgs returns nix, nix-eval-jobs and nix-fast-build built from your pkgs, with the plugin loaded in all of them. Use it in a dev shell, not in environment.systemPackages (that would collide with nix.package):

inputs.nix-grpc-store.inputs.nixpkgs.follows = "nixpkgs";
# ...
devShells.x86_64-linux.default =
  let
    pkgs = nixpkgs.legacyPackages.x86_64-linux;
    tools = nix-grpc-store.lib.nixToolsWithPlugin pkgs;
  in
  pkgs.mkShell {
    packages = [ tools.nix tools.nix-eval-jobs tools.nix-fast-build ];
  };

nix build .#nix-tools-with-plugin builds the same set from this flake's nixpkgs.

Quick start (NixOS)

Add the flake and enable the modules:

# flake.nix inputs
nix-grpc-store.url = "github:Mic92/nix-grpc-store";

# server
imports = [ nix-grpc-store.nixosModules.server ];
services.nix-grpc-daemon.enable = true;
services.nix-grpc-daemon.tls = { certFile = ./server.pem; keyFile = ./server.key; };

# client
imports = [ nix-grpc-store.nixosModules.client ];
programs.nix-grpc-store.enable = true;

The client module ships plugin builds for the supported Nix versions and a loader that picks the one matching the running Nix. The server module runs the daemon as an unprivileged nix-grpc-daemon user and adds it to extra-allowed-users so it can reach the local nix-daemon even when allowed-users is restricted.

Quick start (manual)

On the builder:

nix-grpc-daemon --listen 0.0.0.0:50051

On the client, use nix-with-plugin (or add plugin-files = /path/to/lib/nix/plugins to nix.conf) and use it like any other store URI:

nix store info --store 'grpc://builder:50051?insecure=1'
nix build nixpkgs#hello --store 'grpc://builder:50051?insecure=1'
nix copy --to 'grpc://builder:50051?insecure=1' ./result

The daemon proxies to the local nix-daemon socket, so gRPC clients get whatever store privileges the uid running nix-grpc-daemon has.

TLS and mTLS

Server:

nix-grpc-daemon --listen 0.0.0.0:50051 \
    --tls-cert server.pem --tls-key server.key \
    --client-ca ca.pem              # optional: require client certs

Client:

nix build --store \
  'grpc://builder:50051?ca-cert=ca.pem&client-cert=me.pem&client-key=me.key' ...

Without --client-ca any TLS client can connect. With it, only clients presenting a certificate signed by that CA are accepted.

Trust model

gRPC clients act as the nix-grpc-daemon user, which is not trusted by default. That is enough for nix build --store 'grpc://…', nix copy, path queries and GC: sources and derivations are content-addressed, signed cache paths are accepted, and the server builds everything itself.

This makes an untrusting server usable as a builder without any special setup:

nix build nixpkgs#hello --store 'grpc://builder:50051' --eval-store auto

Evaluation runs locally (--eval-store auto keeps the eval artifacts in the local store instead of round-tripping them to the server), the derivation closure is imported — sources and .drv files are content-addressed, so they pass signature checks — and all builds happen server-side. Nothing unsigned crosses the trust boundary, so trustClients can stay off.

Using the server as a remote builder (builders = grpc://…) is different: there the client schedules builds and uploads input paths it built locally, which are unsigned — something only trusted users may do. Set services.nix-grpc-daemon.trustClients = true for that. Every authenticated client then has trusted-user privileges, so require client certs (tls.clientCaFile).

Access control

With mTLS, --allow 'cn-pattern=role' maps client certificate CNs to roles via glob patterns. The first match wins and unmatched clients are denied. Without any rules every authenticated client keeps full access.

nix-grpc-daemon ... --client-ca ca.pem \
    --allow 'ci-*=trusted' \
    --allow 'cache-mirror=read-only' \
    --allow '*=write' \
    --allow-anonymous read-only    # optional: cert-less clients

• read-only — path queries and NAR downloads (nix copy --from)
• write — additionally imports (signature checking is enforced regardless of --no-check-sigs) and server-side builds (nix build --store 'grpc://…')
• trusted — everything, including the raw worker-protocol tunnel (GC, nix store add, repair) and unsigned imports (still subject to the nix-daemon's trust in the proxy user, see above)

--allow-anonymous ROLE relaxes the client-certificate requirement: cert-less clients connect with that role (e.g. a public read-only cache), certificate holders keep their --allow roles. Naming policies pair well with a CA like step-ca, where provisioners constrain which CNs each token may request.

--trusted-proxy CN-PATTERN is for a TLS-terminating L7 balancer (envoy) in front of the daemon. A peer presenting a certificate whose CN matches is a proxy: the daemon takes the client's identity from the x-forwarded-client-cert header it sets (Subject CN, evaluated against --allow) or a forwarded bearer token, and treats the request as anonymous if neither is present. From any other peer the header is ignored. The proxy must overwrite the header (envoy forward_client_cert_details: SANITIZE_SET) and authenticate to the daemon with its own client cert.

NixOS:

services.nix-grpc-daemon.accessRules = [
  { cn = "ci-*"; role = "trusted"; }
  { cn = "*"; role = "read-only"; }
];
services.nix-grpc-daemon.anonymousRole = "read-only"; # optional

Role needed per use case:

Builds are submitted through a dedicated RPC and run entirely server-side, so the write role suffices. Only operations that tunnel the raw worker protocol need trusted, and repair builds are treated like the rest of the repair surface.

For host-based access, pair this with a CA that issues certs with the host's domain name in the CN, e.g. a step-ca ACME provisioner with forceCN = true: each host obtains its certificate via ordinary ACME (security.acme) and an accessRules glob like *.example.com = read-only grants it substituter access. See tests/acme-substituter-test.nix for a complete, tested NixOS setup (built as the acme-vm check).

OIDC bearer tokens

Instead of (or next to) client certificates the daemon accepts authorization: Bearer <jwt> and maps token claims to the same roles. --oidc-config FILE takes the JSON schema of niks3's --oidc-config, so one file can serve the cache and the daemon. Scopes map to roles: read → read-only, write → write, admin → trusted.

{
  "providers": {
    "github": {
      "issuer": "https://token.actions.githubusercontent.com",
      "audience": "grpc://cache.example.com",
      "rules": [
        { "bound_subject": ["repo:myorg/*"], "scopes": ["write"] },
        { "bound_claims": { "ref": ["refs/heads/main"] }, "scopes": ["admin"] }
      ]
    },
    "k8s": {
      "audience": "grpc://cache.example.com",
      "ca_file": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
      "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
      "bound_claims": { "kubernetes.io.namespace": ["ci"] }
    }
  }
}

Without issuer, it is taken from bearer_token_file (Kubernetes workload identity). jwks_url skips discovery. Signing keys are fetched on first use and cached for an hour, so the daemon starts while the issuer is down and refuses tokens until it comes back. A presented client certificate takes precedence over a token. A verified token that matches no rule is denied, it does not fall back to --allow-anonymous. In logs and metrics the caller shows as oidc:<provider>:<sub>.

Clients pass the token with the token-file URI parameter or $NIX_GRPC_TOKEN_FILE (default: token next to the default client-cert). The file is re-read for every call, so rotating it in place works. TLS is required.

NixOS: services.nix-grpc-daemon.oidc = { providers.github = { … }; };

Scaling out

One daemon is a farm of one: it schedules builds onto itself. More nodes behind a balancer, one of them the scheduler, with deduplicated builds and S3 outputs: see docs/farm.md for NixOS and docs/kubernetes.md for the Helm chart.

Remote builder

nix.buildMachines = [{
  hostName = "grpc://builder:50051";
  protocol = null;
  systems = [ "x86_64-linux" "aarch64-linux" ];
  maxJobs = 64;
}];

TLS uses the system CA bundle and the client key pair from /run/nix-grpc-store or /var/lib/nix-grpc-store by default (see below).

With access control enabled, remote builders need the trusted role. The builder protocol imports unsigned outputs built on the client, which also requires trustClients (see above).

URI parameters

• insecure — plaintext, no TLS (testing only)
• ca-cert — PEM CA to verify the server. Defaults to $NIX_SSL_CERT_FILE, $SSL_CERT_FILE or the system CA bundle
• client-cert, client-key — PEM pair to present for mTLS. Default to $NIX_GRPC_CLIENT_CERT/$NIX_GRPC_CLIENT_KEY, then client.crt/client.key in $XDG_DATA_HOME/nix-grpc-store, then /run/nix-grpc-store, then /var/lib/nix-grpc-store (unreadable candidates are skipped, and none is looked up when token-file is given)
• token-file — OIDC bearer token, re-read per call. Defaults to $NIX_GRPC_TOKEN_FILE, then token in the directories above
• connect-timeout (default 30) — seconds the first call keeps retrying "connection refused" and similar.
• restart-grace (default 120) — once the server has answered, seconds to ride out a worker or scheduler restart, and to wait while the scheduler says no worker can take a build.
• reschedule-retries (default 8) — how often a derivation whose assigned worker went away is handed back to the scheduler.
• max-builds (default 64) — concurrent BuildDerivation streams.
• system — send x-nix-system on every call, not just builds, so a balancer routes input uploads and substitution to a worker of that system. Use one nix.buildMachines entry per system.
• debug — print which CA bundle and client certificate were loaded and turn on gRPC's TCP/TLS handshake tracing on stderr. NIX_GRPC_DEBUG=1 does the same and also reaches the build hook.

Server flags

• --listen ADDR — default 0.0.0.0:50051
• --proxy-socket PATH — nix-daemon socket, default /nix/var/nix/daemon-socket/socket
• --tls-cert, --tls-key, --client-ca — see above
• --allow 'cn-pattern=role', --allow-anonymous ROLE, --trusted-proxy CN-PATTERN — see access control
• --oidc-config FILE — accept OIDC bearer tokens, see access control
• --metrics-listen ADDR — serve Prometheus metrics, disabled if unset
• --worker-name NAME — name in build-log prefixes and nix_grpc_build_info, default hostname
• --role builder,scheduler, --scheduler ADDR, --scheduler-token-file FILE, --advertise IP:PORT, --max-jobs N, --min-free SIZE — see docs/farm.md. The defaults make a single node schedule onto itself
• --niks3 URL [--niks3-token-file FILE] [--niks3-client-cert FILE --niks3-client-key FILE] [--niks3-push CMD] — publish build outputs and uploads to a niks3 cache; authenticate with a bearer token, a client certificate (defaults to --tls-cert/--tls-key), or both
• --log-level info|debug — access log verbosity, default info

Monitoring

All gRPC clients act as the same local user, so activity is attributed to the client certificate CN (cn=- without mTLS). Have your CA put a user name in the CN and logs and metrics are per user.

Access log

The daemon writes one logfmt line per RPC to stderr (journald):

ts=2025-01-15T12:03:41Z level=info event=rpc method=Connect cn=alice peer=ipv4:10.0.0.5:53211 duration_s=1832 bytes_in=52341 bytes_out=812345678

Every line carries the client certificate CN and the peer address, so one user's activity is a grep away:

journalctl -u nix-grpc-daemon | grep cn=alice

Prometheus metrics

With --metrics-listen 127.0.0.1:9464 (NixOS: services.nix-grpc-daemon.metricsListen) the daemon serves /metrics:

Only CA-issued CNs appear as labels, so cardinality stays bounded.

Tests

nix build .#checks.x86_64-linux.vm -L

End-to-end NixOS VM test plus a 256 MiB throughput benchmark against the unix socket and perf counters. See src/pump.hh for design notes on chunk coalescing and flushable zstd.

nix build .#bench-latency -L

VM benchmark measuring nix copy of 200 small paths at an emulated 50 ms RTT. scripts/bench-transports.py and scripts/bench-builds.py benchmark real hosts and scripts/bench-plot.py renders the charts above.