nix-eval-jobs

This project evaluates nix attribute sets in parallel with streamable json output. This is useful for time and memory intensive evaluations such as NixOS machines, i.e. in a CI context. The evaluation is done by a controllable number of worker processes that are restarted when their memory consumption exceeds a certain threshold.

To facilitate integration, nix-eval-jobs creates garbage collection roots for each evaluated derivation (drv file, not the build) within the provided attribute. This prevents race conditions between the nix garbage collection service and user-started nix builds processes.

Why using nix-eval-jobs?

• Faster evaluation by using multiple worker processes
• Memory used for evaluation is reclaimed after nix-eval-jobs finish, so that the build can use it.
• Evaluation of jobs can fail individually

Example

In the following example we evaluate the hydraJobs attribute of the patchelf flake:

$ nix-eval-jobs --gc-roots-dir gcroot --flake 'github:NixOS/patchelf#hydraJobs'
{"attr":"coverage","attrPath":["coverage"],"drvPath":"/nix/store/fmbqzaq8mim1423879lhn9whs6imx5w4-patchelf-coverage-0.18.0.drv","inputDrvs":{"/nix/store/23632hx2c98lbbjld279dx0w08lxn6kp-hook.drv":["out"],"/nix/store/6z1jfnqqgyqr221zgbpm30v91yfj3r45-bash-5.1-p16.drv":["out"],"/nix/store/ap9g09fxbicj836zm88d56dn3ff4clxl-stdenv-linux.drv":["out"],"/nix/store/c0gg7lj101xhd8v2b3cjl5dwwkpxfc0q-patchelf-tarball-0.18.0.drv":["out"],"/nix/store/vslywm6kbazi37q1vbq8y7bi884yc6yx-lcov-1.16.drv":["out"],"/nix/store/y964yq4vz1gsn7azd44vyg65gnr4gpvi-hook.drv":["out"]},"name":"patchelf-coverage-0.18.0","outputs":{"out":"/nix/store/gfni9sbhhwhxxfqziq1fs3n82bvw962l-patchelf-coverage-0.18.0"},"system":"x86_64-linux"}
{"attr":"patchelf-win32","attrPath":["patchelf-win32"],"drvPath":"/nix/store/s38l0fg5ja6j8qpws7slw2ws0c6v0qcf-patchelf-i686-w64-mingw32-0.18.0.drv","inputDrvs":{"/nix/store/6z1jfnqqgyqr221zgbpm30v91yfj3r45-bash-5.1-p16.drv":["out"],"/nix/store/b2p151ilwqpd47fbmzz50a5cmj12ixbf-hook.drv":["out"],"/nix/store/fbnhh18m4jh6cwa92am2sv3aqzjnzpdd-stdenv-linux.drv":["out"]},"name":"patchelf-i686-w64-mingw32-0.18.0","outputs":{"out":"/nix/store/w8r4h1xk71fryb99df8aszp83kfhw3bc-patchelf-i686-w64-mingw32-0.18.0"},"system":"x86_64-linux"}
{"attr":"patchelf-win64","attrPath":["patchelf-win64"],"drvPath":"/nix/store/wxpym6d3dxr1w9syhinp7f058gwxfmd3-patchelf-x86_64-w64-mingw32-0.18.0.drv","inputDrvs":{"/nix/store/6z1jfnqqgyqr221zgbpm30v91yfj3r45-bash-5.1-p16.drv":["out"],"/nix/store/71lv5lsr1y59bv1b91jc9gg0n85kf1sq-stdenv-linux.drv":["out"],"/nix/store/b2p151ilwqpd47fbmzz50a5cmj12ixbf-hook.drv":["out"]},"name":"patchelf-x86_64-w64-mingw32-0.18.0","outputs":{"out":"/nix/store/fkq5428l2xsb84yj0cc6q1lkvsrga7sv-patchelf-x86_64-w64-mingw32-0.18.0"},"system":"x86_64-linux"}
{"attr":"release","attrPath":["release"],"drvPath":"/nix/store/3xpwg8f623dpkh6cblv2fzcq5n99xl0j-patchelf-0.18.0.drv","inputDrvs":{"/nix/store/6z1jfnqqgyqr221zgbpm30v91yfj3r45-bash-5.1-p16.drv":["out"],"/nix/store/9rmihrl9ys0sap6827xyns0y73vqafjx-patchelf-0.18.0.drv":["out"],"/nix/store/am2zqx3pyc1i14f888jna785h0f841sg-patchelf-0.18.0.drv":["out"],"/nix/store/c0gg7lj101xhd8v2b3cjl5dwwkpxfc0q-patchelf-tarball-0.18.0.drv":["out"],"/nix/store/csjiccxbwpfv55m8kqs2xwrkkha14dnq-patchelf-0.18.0.drv":["out"],"/nix/store/jsrnpxdx5vmpnakd9bkb3sk3lgh0k8hm-patchelf-0.18.0.drv":["out"],"/nix/store/k8a51ax83554c67g98xf3y751vjgjs7m-patchelf-0.18.0.drv":["out"],"/nix/store/wq3ncl207isqqkqmsa5ql4fg19jbrhxg-stdenv-linux.drv":["out"]},"name":"patchelf-0.18.0","outputs":{"out":"/nix/store/d0mzprvv3vhasj23r1a6qn8qip0srbc4-patchelf-0.18.0"},"system":"x86_64-linux"}
{"attr":"tarball","attrPath":["tarball"],"drvPath":"/nix/store/c0gg7lj101xhd8v2b3cjl5dwwkpxfc0q-patchelf-tarball-0.18.0.drv","inputDrvs":{"/nix/store/6z1jfnqqgyqr221zgbpm30v91yfj3r45-bash-5.1-p16.drv":["out"],"/nix/store/9d754glmsvpjm5kxvgsjslvgv356kbmn-libtool-2.4.7.drv":["out"],"/nix/store/ap9g09fxbicj836zm88d56dn3ff4clxl-stdenv-linux.drv":["out"],"/nix/store/f1ksgsyplvb0sli4pls6k6vsfvmv519d-autoconf-2.71.drv":["out"],"/nix/store/jf58lcnch1bmpbi2188c59w5zr1cqrx2-automake-1.16.5.drv":["out"]},"name":"patchelf-tarball-0.18.0","outputs":{"out":"/nix/store/72pz5awc7gpwdqxrdsy8j0bvg2n7z78q-patchelf-tarball-0.18.0"},"system":"x86_64-linux"}

The output here is newline-seperated json according to https://jsonlines.org.

The code is derived from hydra's eval-jobs executable.

Further options

USAGE: nix-eval-jobs [options] expr

  --apply                Apply provided Nix function to each derivation. The result of this function will be serialized as a JSON value and stored inside `"extraValue"` key of the json line output.
  --arg                  Pass the value *expr* as the argument *name* to Nix functions.
  --arg-from-file        Pass the contents of file *path* as the argument *name* to Nix functions.
  --arg-from-stdin       Pass the contents of stdin as the argument *name* to Nix functions.
  --argstr               Pass the string *string* as the argument *name* to Nix functions.
  --check-cache-status   Check if the derivations are present locally or in any configured substituters (i.e. binary cache). The information will be exposed in the `cacheStatus` field of the JSON output.
  --constituents         whether to evaluate constituents for Hydra's aggregate feature
  --debug                Set the logging verbosity level to 'debug'.
  --eval-store
            The [URL of the Nix store](@docroot@/store/types/index.md#store-url-format)
            to use for evaluation, i.e. to store derivations (`.drv` files) and inputs referenced by them.

  --expr                 treat the argument as a Nix expression
  --flake                build a flake
  --force-recurse        force recursion (don't respect recurseIntoAttrs)
  --gc-roots-dir         garbage collector roots directory
  --help                 show usage information
  --impure               allow impure expressions
  --include
  Add *path* to search path entries used to resolve [lookup paths](@docroot@/language/constructs/lookup-path.md)

  This option may be given multiple times.

  Paths added through `-I` take precedence over the [`nix-path` configuration setting](@docroot@/command-ref/conf-file.md#conf-nix-path) and the [`NIX_PATH` environment variable](@docroot@/command-ref/env-common.md#env-NIX_PATH).

  --log-format           Set the format of log output; one of `raw`, `internal-json`, `bar` or `bar-with-logs`.
  --max-memory-size      memory per worker in MiB (4GiB by default). workers * max-memory-size is enforced as a budget for all workers combined: jobs are only dispatched while it has room, a worker above its share is restarted after its job, and if the budget is exceeded anyway the largest worker is killed and its job retried alone.
  --meta                 include derivation meta field in output
  --no-instantiate       don't instantiate (write) derivations, only evaluate (faster)
  --option               Set the Nix configuration setting *name* to *value* (overriding `nix.conf`).
  --override-flake       Override the flake registries, redirecting *original-ref* to *resolved-ref*.
  --override-input       Override a specific flake input (e.g. `dwarffs/nixpkgs`).
  --quiet                Decrease the logging verbosity level.
  --reference-lock-file  Read the given lock file instead of `flake.lock` within the top-level flake.
  --repair               During evaluation, rewrite missing or corrupted files in the Nix store. During building, rebuild missing or corrupted store paths.
  --select               Apply provided Nix function to transform the evaluation root. This is applied before any attribute traversal begins. When used with --flake without a fragment, the function receives an attrset with 'outputs' and 'inputs'. When used with a flake fragment, it receives the selected attribute. Examples: --select 'flake: flake.outputs.packages' --select 'flake: flake.inputs.nixpkgs' --select 'outputs: outputs.packages.x86_64-linux'
  --show-input-drvs      Show input derivations in the output for each derivation. This is useful to get direct dependencies of a derivation.
  --show-trace           print out a stack trace in case of evaluation errors
  --verbose              Increase the logging verbosity level.
  --workers              number of evaluate workers

Potential use-cases for the tool

Faster evaluator in deployment tools. When evaluating NixOS machines, evaluation can take several minutes when run on a single core. This limits scalability for large deployments with deployment tools such as NixOps.

Faster evaluator in CIs. In addition to evaluation speed for CIs, it is also useful if evaluation of individual jobs in CIs can fail, as opposed to failing the entire jobset. For CIs that allow dynamic build steps to be created, one can also take advantage of the fact that nix-eval-jobs outputs the derivation path separately. This allows separate logs and success status per job instead of a single large log file. In the wiki we collect example ci configuration for various CIs.

Projects using nix-eval-jobs

• Hydra - The Nix-based continuous build system
• nix-fast-build - Combine the power of nix-eval-jobs with nix-output-monitor to speed-up your evaluation and building process
• nixbot - Standalone Nix CI service for NixOS
• colmena - A simple, stateless NixOS deployment tool
• robotnix - Build Android (AOSP) using Nix, used in their CI

FAQ

How can I check if my package already have been uploaded in the binary cache?

If you provide the --check-cache-status, the json will contain a "cacheStatus" key in its json, with the following values:

Where do evaluation warnings and traces end up?

Messages from builtins.warn and builtins.trace are printed to stderr as usual and additionally attached to the JSON line of the attribute that was being evaluated, as "warnings": [...] and "traces": [...] (omitted when empty). Because Nix evaluates shared values only once, a message from code shared between attributes is reported on whichever attribute forced it first.

How expensive was an attribute to evaluate?

Every JSON line carries "stats": {"wallMs": ..., "allocBytes": ...} with the wall-clock time and the number of bytes allocated on the GC heap while the worker evaluated that attribute. As with warnings, values shared between attributes are only evaluated once and are billed to whichever attribute forced them first, so the numbers depend on scheduling order and are best used to spot outliers rather than for exact accounting.

How can I evaluate nixpkgs?

If you want to evaluate nixpkgs in the same way hydra does it, use this snippet:

$ nix-eval-jobs --force-recurse pkgs/top-level/release.nix

nix-eval-jobs consumes too much memory / is too slow

By default nix-eval-jobs spawns one worker with --max-memory-size 4096 MiB. Workers keep their evaluation cache between attributes, so more memory per worker means fewer restarts and less re-evaluation of shared dependencies. More --workers means more parallelism but also more duplicated evaluation.

workers * max-memory-size is treated as one budget for all workers combined, so you can size it to the memory you actually want to give the run (e.g. a cgroup limit):

• A worker whose RSS exceeds its share is restarted after finishing its current attribute.
• New attributes are only dispatched while the total RSS plus the estimated growth of all in-flight jobs fits the budget, so a few large attributes (e.g. NixOS closures) throttle concurrency instead of overshooting.
• If the budget is exceeded anyway, the largest worker is killed and its attribute is retried alone on a fresh worker; only if it still does not fit is it reported as an error for that attribute. A worker killed externally (e.g. by the kernel OOM killer) is handled the same way.

Rule of thumb: set --workers to the cores you want to use and --max-memory-size to available memory / workers. If single attributes are larger than one share, prefer fewer workers with a larger --max-memory-size over many small ones.

Making a release

Versions follow <nix major.minor>.<revision>. From an up-to-date main run ./dev/create-release.sh (or ./dev/create-release.sh 0 after bumping Nix). It bumps revision in default.nix, merges it via PR, tags and publishes the GitHub release.