npb

A CLI tool to generate reports of the differences in build outcomes between pairs of Nixpkgs commits.

For example, the following report was generated by running this command:

npb --pr 539280

That command may not give you the same result if you run it yourself, but npb's report includes a reproducible command that will:

npb · 26426a9 → 17a97c0

Expand this for a reproducible command.

npb --base 26426a9ad1c5cebe35b163a94152065a7b9d1413 --head 9acc6884ece717d2d20a485da38fb2fc39d575a0 --patch 9acc6884ece717d2d20a485da38fb2fc39d575a0...17a97c09e0d9467dd191a2cfbc800c2a984b5f1b -s aarch64-darwin

Expand this for a legend of all symbols below.

• ✅ = successfully built
• ❌ = failed to build
• 🚫 = dependency failed to build
• ❔ = couldn't try to build
• ⏩ = failed to evaluate
• ➖ = doesn't exist

aarch64-darwin

🚫 → 🚫 · 1 package

• lixStatic

✅ → ✅ · 76 packages (43 unique)

• lixPackageSets.git.colmena
• lixPackageSets.git.lix
• lixPackageSets.git.nil
• lixPackageSets.git.nix-direnv
• lixPackageSets.git.nix-du
• lixPackageSets.git.nix-eval-jobs
• lixPackageSets.git.nix-fast-build
• lixPackageSets.git.nix-init
• lixPackageSets.git.nix-update
• lixPackageSets.git.nixos-anywhere
• lixPackageSets.git.nixos-rebuild-ng.tests.repl = lixPackageSets.latest.nixos-rebuild-ng.tests.repl = lixPackageSets.stable.nixos-rebuild-ng.tests.repl = lixPackageSets.lix_2_94.nixos-rebuild-ng.tests.repl = lixPackageSets.lix_2_95.nixos-rebuild-ng.tests.repl
• lixPackageSets.git.nixos-rebuild-ng
• lixPackageSets.git.nixpkgs-reviewFull
• lixPackageSets.git.nixpkgs-review
• lixPackageSets.git.nurl
• lixPackageSets.latest.colmena = lixPackageSets.stable.colmena = lixPackageSets.lix_2_95.colmena
• lixPackageSets.latest.nil = lixPackageSets.stable.nil = lixPackageSets.lix_2_95.nil
• lixPackageSets.latest.nix-direnv = lixPackageSets.stable.nix-direnv = lixPackageSets.lix_2_95.nix-direnv
• lixPackageSets.latest.nix-du = lixPackageSets.stable.nix-du = lixPackageSets.lix_2_95.nix-du
• lixPackageSets.latest.nix-eval-jobs = lixPackageSets.stable.nix-eval-jobs = lixPackageSets.lix_2_95.nix-eval-jobs
• lixPackageSets.latest.nix-fast-build = lixPackageSets.stable.nix-fast-build = lixPackageSets.lix_2_95.nix-fast-build
• lixPackageSets.latest.nix-init = lixPackageSets.stable.nix-init = lixPackageSets.lix_2_95.nix-init
• lixPackageSets.latest.nix-update = lixPackageSets.stable.nix-update = lixPackageSets.lix_2_95.nix-update
• lixPackageSets.latest.nixos-anywhere = lixPackageSets.stable.nixos-anywhere = lixPackageSets.lix_2_95.nixos-anywhere
• lixPackageSets.latest.nixos-rebuild-ng = lixPackageSets.stable.nixos-rebuild-ng = lixPackageSets.lix_2_95.nixos-rebuild-ng
• lixPackageSets.latest.nixpkgs-reviewFull = lixPackageSets.stable.nixpkgs-reviewFull = lixPackageSets.lix_2_95.nixpkgs-reviewFull
• lixPackageSets.latest.nixpkgs-review = lixPackageSets.stable.nixpkgs-review = lixPackageSets.lix_2_95.nixpkgs-review
• lixPackageSets.latest.nurl = lixPackageSets.stable.nurl = lixPackageSets.lix_2_95.nurl
• lixPackageSets.lix_2_94.colmena
• lixPackageSets.lix_2_94.lix
• lixPackageSets.lix_2_94.nil
• lixPackageSets.lix_2_94.nix-direnv
• lixPackageSets.lix_2_94.nix-du
• lixPackageSets.lix_2_94.nix-eval-jobs
• lixPackageSets.lix_2_94.nix-fast-build
• lixPackageSets.lix_2_94.nix-init
• lixPackageSets.lix_2_94.nix-update
• lixPackageSets.lix_2_94.nixos-anywhere
• lixPackageSets.lix_2_94.nixos-rebuild-ng
• lixPackageSets.lix_2_94.nixpkgs-reviewFull
• lixPackageSets.lix_2_94.nixpkgs-review
• lixPackageSets.lix_2_94.nurl
• lix = lixPackageSets.latest.lix = lixPackageSets.stable.lix = lixPackageSets.lix_2_95.lix

❌ → ✅ · 14 packages (4 unique)

• lixPackageSets.git.nix-serve-ng
• lixPackageSets.lix_2_94.nix-serve-ng
• nix-serve-ng.tests.lix = lixPackageSets.latest.nix-serve-ng = lixPackageSets.stable.nix-serve-ng = lixPackageSets.lix_2_95.nix-serve-ng = haskellPackages.nix-serve-ng.tests.lix = lixPackageSets.git.nix-serve-ng.tests.lix = lixPackageSets.latest.nix-serve-ng.tests.lix = lixPackageSets.stable.nix-serve-ng.tests.lix = lixPackageSets.lix_2_94.nix-serve-ng.tests.lix = lixPackageSets.lix_2_95.nix-serve-ng.tests.lix
• nix-serve-ng = haskellPackages.nix-serve-ng

❔ → ❔ · 6 packages (3 unique)

• lix.tests.misc = lixPackageSets.latest.lix.tests.misc = lixPackageSets.stable.lix.tests.misc = lixPackageSets.lix_2_95.lix.tests.misc
• lixPackageSets.git.lix.tests.misc
• lixPackageSets.lix_2_94.lix.tests.misc

---

• Goals
• Non-goals
• Installation
• Usage
  • -h, --help
  • -C
  • --base
  • --head
  • --pr
  • --patch
  • --no-merge
  • --no-tests
  • -p, --package
  • --allow-unsupported
  • --allow-broken
  • --allow-insecure
  • --overlay
  • --retry
  • -s, --system
  • --clean
  • --completions
  • -V, --version
• Acknowledgments
• License

Goals

npb strives to be

• causal. Derivations are built on both sides of the change, so the report makes it clear which failures are regressions and which were preexisting.

• thorough. By default, the tool finds and runs tests associated with the affected packages, and includes those test outcomes in the report.

• cached. It takes a nontrivial amount of time to run Nix evaluations and failing builds, so the tool caches these; re-runs should almost always take less than one second.

• efficient. Caching is not an excuse to be slow the first time, so on a cold run, the tool automatically chooses a point along the Pareto frontier to parallelize according to how much RAM is available.

• reproducible. Often a change may be on a branch that gets force-pushed later, or there may be uncommitted local changes, so the report always includes a command to reproduce the exact same Git tree on both sides.

• friendly. Users running the tool in a rich terminal should get nice colors and live progress indicators, and users running it over a pipe should get lean but informative progress lines.

Non-goals

npb does not

• fetch remote evaluations.
• manage Git worktrees.
• provide sandboxing.
• post to GitHub.

Installation

npb is in Nixpkgs, so you can run the following command to start a shell within your shell which has npb available on the PATH:

nix-shell -p npb

Usage

-h, --help

To list available CLI arguments:

npb --help

-C

By default, npb assumes the current working directory is a clone of the Nixpkgs repo. If your Nixpkgs clone is somewhere else:

npb -C ~/github/NixOS/nixpkgs

Note that, just like Git's -C, this argument affects how other paths are interpreted, specifically for --patch.

--base

By default, npb compares against master. To compare against a different base:

npb --base origin/release-26.05

--head

By default, npb evaluates on the current working tree. To evaluate on a different revision:

npb --head origin/staging

--pr

To set --base and --head from the current state of an open pull request:

npb --pr 543008

Note that this is an exception to npb's usual rule that a re-run should be fully cached. If you run npb with a --pr argument, and then that PR is updated, and then you run the same npb command again, it will fetch the current state of the PR every time.

--patch

To apply a diff on top of the --head, provide a path to a patch file. For instance, one trick that npb sometimes uses to provide reproducible commands is --patch /dev/stdin with a heredoc:

npb --patch /dev/stdin <<'PATCH'
--- a/pkgs/by-name/he/hello/package.nix
+++ b/pkgs/by-name/he/hello/package.nix
@@ -66,3 +66,3 @@ stdenv.mkDerivation (finalAttrs: {
     mainProgram = "hello";
-    platforms = lib.platforms.all;
+    platforms = [ ]; # bye bye
     identifiers.cpeParts.vendor = "gnu";
PATCH

Note that the path must include a forward-slash. That is to avoid ambiguity because --patch also accepts three-dot GitHub commit comparison expressions:

npb --patch HEAD...974b2de7dc8f59ba4ac9729c452834641f82849f

--no-merge

By default, npb merges the --base and the --head together, then evaluates --base versus that merged result. To instead take the merge-base of --base and --head, and evaluate that versus --head:

npb --no-merge

--no-tests

By default, npb includes all passthru.tests for packages that changed between the two commits. To not include those tests:

npb --no-tests

-p, --package

By default, npb determines the set of non-test packages by finding all changes. To provide a custom set of packages instead:

npb -ppython3{,{11..15}}Packages.dirty-equals

--allow-unsupported

By default, npb sets allowUnsupportedSystem to false. To set it to true instead:

npb --allow-unsupported

--allow-broken

By default, npb sets allowBroken to false. To set it to true instead:

npb --allow-broken

--allow-insecure

By default, npb sets allowInsecurePredicate to a constant false. To set it to a constant true instead:

npb --allow-insecure

--overlay

To apply a Nixpkgs overlay:

npb --overlay 'final: prev: { pypy311Packages = final.lib.recurseIntoAttrs prev.pypy311Packages; }'

This argument can be repeated.

--retry

By default, npb caches failures and does not try to rebuild derivations that it has seen fail before. To retry failed builds anyway:

npb --retry

-s, --system

By default, npb only evaluates for the system of the machine it is running on. To override that and instead build for one or more other systems:

npb -s x86_64-linux -s aarch64-linux -s aarch64-darwin

--clean

Whenever npb encounters a Git tree it hasn't seen before, it evaluates Nixpkgs on that tree and saves the result to disk, roughly 3 megabytes per tree per --system. To free up disk space by deleting the least recently used evaluation results:

npb --clean 100MB

The value passed to this argument must be either a YYYY-MM-DD date or an integer with a suffix, which determines its meaning as follows.

• Delete entries not used for at least some amount of time:
  • s/sec = seconds
  • m/min = minutes
  • h/hr = hours
  • d = days
  • w = weeks
  • mo = months
  • y/yr = years
• Delete least recently used entry until cache is small enough:
  • B = bytes
  • kB = kilobytes
  • KiB = kibibytes
  • MB = megabytes
  • MiB = mebibytes
  • GB = gigabytes
  • GiB = gibibytes
  • TB = terabytes
  • TiB = tebibytes

--completions

To generate shell completions:

npb --completions bash

-V, --version

To print the version:

npb --version

Acknowledgments

This project owes its existence to its primary inspiration, Jörg Thalheim's nixpkgs-review, which serves a similar purpose but makes different design tradeoffs.

The development of this project was AI-assisted. There is no AI-generated text in this README, in CHANGELOG.md, in npb --help, or in npb's generated reports.

License

npb is licensed under the MIT License.