Hermit

Hermit is a deterministic execution environment for x86-64 Linux programs. It runs an unmodified guest under the Reverie ptrace backend and controls sources of nondeterminism including thread scheduling, time, random data, CPUID results, and selected file metadata. That is the fork this repository builds against — both the Cargo git dependency and the workspace submodule pin https://github.com/rrnewton/reverie.git. Reverie's upstream reference repository is facebookexperimental/reverie.

Hermit is useful for repeatable execution, controlled concurrency testing, record/replay experiments, and diagnosing schedule-sensitive failures.

For an introduction to the ideas behind Hermit, read Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding. It explains how controlling thread schedules and other implicit inputs helps reproduce flaky tests and diagnose concurrency bugs. For current support and usage, see the documentation below.

Requirements

Hermit currently supports x86-64 Linux. Building and running it requires:

• Rust nightly through rustup; rust-toolchain.toml selects the repository toolchain automatically.
• Linux user, PID, and mount namespaces.
• Parent-child ptrace and seccomp filter support.
• libunwind and LZMA development packages.
• User-space performance counters for precise scheduler preemption. Hermit can run without them, but CPU-bound workloads receive fewer preemption points.

On Debian or Ubuntu:

sudo apt-get update
sudo apt-get install -y libunwind-dev liblzma-dev

On Fedora or CentOS:

sudo dnf install -y libunwind-devel xz-devel

Install From Source

Clone the maintained fork and install the CLI into Cargo's binary directory, normally ~/.cargo/bin:

git clone --recurse-submodules https://github.com/rrnewton/hermit
cd hermit
cargo install --locked --path hermit-cli
hermit --version

If you already cloned the repository, initialize its submodules before building:

git submodule update --init --recursive

To build without installing:

cargo build --workspace
./target/debug/hermit --version

An optimized workspace build also stages the backend runtime resources:

cargo build --release --workspace
./target/install_pkg/hermit --version

Staging resources does not enable optional backends in the CLI. See Execution Backends for the feature-enabled build.

target/install_pkg/rsrcs/ contains the SaBRe and e9patch rewriters, Detcore backend shared libraries, the DynamoRIO launcher/runtime, and its relocatable client. Hermit finds that directory from its invocation path or executable path. Set HERMIT_INSTALL_DIR only when the resources live under a different prefix. The staging hermit entry is a symlink to target/release/hermit; use a dereferencing copy such as cp -aL target/install_pkg/ DESTINATION when making a standalone installation or archive.

Quick Start

Run a command deterministically by placing hermit run -- before it:

hermit run -- /bin/echo hello

The -- separator is recommended so arguments beginning with - are passed to the guest. The command above prints hello and exits with the guest's status.

Hermit's current defaults are strict and deterministic. --strict is retained as an explicit compatibility spelling for those defaults; it does not enable a stronger mode:

hermit run --strict -- /bin/echo hello

For programs that read the clock, also fix --epoch to reproduce the starting time. Without it, each invocation captures the current host time. See the date example.

Execution Backends

A backend is the low-level mechanism Hermit uses to observe and control a running guest — how it intercepts the program's system calls and CPU events. Different backends make different trade-offs in speed, host requirements, and which programs they support, which is why there is more than one. The default ptrace backend needs no special setup and is the most thoroughly tested; the rest are experimental or specialized. If you are just getting started, use the default and skip the rest of this section. See Architecture for how a backend fits into the whole system.

Hermit accepts --backend=ptrace|dbt|liteinst|sabre|kvm|e9patch as a global option before the subcommand. Backend scope is command-specific: every backend supports run; KVM also supports analyze and bisect, which launch run trials; e9patch also supports record, and SaBRe also supports strace; unsupported combinations fail closed. Omitting the option selects ptrace, preserving the existing behavior:

hermit --backend=ptrace run -- /bin/echo hello

Backend selection fails closed. Hermit never substitutes ptrace after an explicit backend request. LiteInst is an experimental ptrace-hosted hybrid for dynamically linked Linux x86-64 guests. It needs the optional liteinst CLI feature, which third-party-backends includes; a build without it refuses --backend=liteinst:

./scripts/stage-liteinst-runtime.sh dev \
  "$PWD/target/debug/libreverie_liteinst.so" \
  "$PWD/target/liteinst-runtime-build"
cargo build --locked -p hermit --features liteinst --bin hermit
./target/debug/hermit --backend=liteinst run --strict --verify -- /bin/echo hello

The ptrace host owns the sole generic Reverie Detcore Tool and GlobalTool. The standalone manifest enables and statically verifies the preload constructor; Hermit rejects non-runtime or constructor-free overrides before activation. The resulting Reverie preload DSO initializes only the LiteInst patch/helper side; it never installs another Tool in the guest. The host observes the first invocation of each eligible syscall site and installs an instruction-punning hook. Later invocations enter the LiteInst trampoline and return to the same ptrace-owned Detcore lifecycle.

--verify compares captured status and output and applies the Stripped comparison to selected Detcore scheduler messages. A successful result is a useful diagnostic, but it is not strict determinism. Strict verification requires --verify-strict --verify-json REPORT.json, bitwise_parity: true, and nonzero compared-message counts. Guests may create threads and child processes: clone, clone3 and fork run under the ordinary ptrace lifecycle. Installation of new hooks stops at the first task-creating syscall, while existing hooks and tasks keep running. An activated process leader may exec a new dynamically linked program; the new image must retain the inherited runtime environment and activate the LiteInst runtime again. vfork and exec from a thread other than the process leader remain unsupported and are refused. RCB preemption and CPUID/RDTSC interception use the ptrace host and retain its PMU and CPU capability requirements. The default Hermit namespace path is supported; --no-namespace remains an explicit option for trusted guests. The in-guest patch runtime is experimental and continues to receive compatibility and lifecycle improvements. The dbt, sabre, and e9patch selections require optional CLI features as well as runtime resources. Build both with:

cargo build --release --workspace --features hermit/third-party-backends
./target/install_pkg/hermit --backend=dbt run -- /bin/echo hello

Use the staged executable for these backends; a default cargo install does not enable them. KVM requires read-write /dev/kvm access plus its guest-kernel Linux ABI.

SaBRe's measured post-0.2 Stripped envelope, build instructions, and explicit unsupported cases are documented in SaBRe backend compatibility.

The experimental e9patch selection is intentionally a hybrid backend. At startup it loads or generates the main ELF's cached instruction map, then runs e9tool -O0 with exact file-offset matches to install semantics-preserving trampolines at every candidate offset that e9tool recovers as an instruction. The conservative optimizer setting avoids known multi-class rewrite failures. The linear scan can include embedded data, so Hermit reports candidate and recovered counts separately. Partial e9tool coverage fails closed. Hermit does not enable e9patch's B0 fallback because it reserves SIGILL and changes guest signal semantics. The rewritten ELF still runs through Detcore's ptrace backend, which executes the original instructions and covers trapped events in shared libraries, the vDSO, and dynamic code. Raw RDRAND, RDSEED, and TSX in code remain unsupported even when present in the offline map because this initial integration installs empty trampolines. Privilege-bearing executables fail closed rather than losing set-ID or file-capability semantics. This establishes the cached-rewrite pipeline but does not yet reduce ptrace events. The workspace installation package includes rsrcs/e9tool; no separate install is needed when using that package. For a custom installation, put e9tool in PATH or set HERMIT_E9TOOL to its executable. Non-ELF entrypoints, including shebang scripts, skip preprocessing and run through the ptrace correctness path.

A quick determinism check is to run the same virtual random-data read twice:

hermit run -- /bin/sh -c 'od -An -N8 -tx1 /dev/urandom'
hermit run -- /bin/sh -c 'od -An -N8 -tx1 /dev/urandom'

Both invocations should print the same bytes when the command, inputs, and Hermit configuration are unchanged.

Compatibility

The latest compatibility scorecard records results for specific workloads, execution backends, and modes. A passing version probe does not establish support for an application's other workflows. Check the recorded command and configuration, then test the workload you depend on. Run and record/replay have different compatibility limits.

SCORECARD.md lists which tests apply to each backend; it is a selection catalogue, not a pass/fail report. For a larger example, the QEMU boot guide describes deterministic and relaxed Linux boot configurations and their limits.

Key Workflows

A minimal record/replay session is:

hermit record start -- /bin/echo recorded
hermit replay --autopilot

Recordings are stored in $XDG_CACHE_HOME/hermit, normally ~/.cache/hermit. Use --data-dir=DIR or HERMIT_DATA_DIR to select another directory.

Debug Adapter Protocol

The DAP adapter is experimental. End-to-end attach and reverse replay are currently broken with GDB 17.2: an attached session can report that its thread has terminated, and reverse replay fails while loading the adapter's Python extension. No working GDB version is established by this guide. The examples below describe the interface; use ordinary GDB debugging or replay with --autopilot when these adapter failures affect you.

Hermit exposes a GDB remote target with run --gdbserver and replay. hermit-dap starts GDB's Debug Adapter Protocol interpreter with the local system root configured so GDB does not request shared libraries from Hermit's remote server. A GDB build with the DAP interpreter is required. Use --gdb when that executable is not /usr/bin/gdb. To identify the default GDB:

/usr/bin/gdb --interpreter=dap --version

A version string alone does not establish a working DAP session. To build the adapter and start a remote target:

cargo build -p hermit --bin hermit --bin hermit-dap
target/debug/hermit run --gdbserver --gdbserver-port=1234 -- /path/to/program

# Or serve the latest recording without launching GDB itself.
target/debug/hermit replay --serve-only --gdbserver-port=1234

In another terminal, a DAP client can spawn target/debug/hermit-dap and send an attach request containing the executable and remote target. For example, Dapper can run this session from a JSON configuration:

{
  "spawnConfig": {
    "type": "stdio",
    "cmd": "/path/to/hermit/target/debug/hermit-dap"
  },
  "debugRequest": {
    "request": "attach",
    "program": "/path/to/program",
    "target": "127.0.0.1:1234"
  },
  "breakpoints": [
    {
      "type": "source",
      "path": "/path/to/program.c",
      "line": 12
    }
  ],
  "installDefaultExceptionBreakpoints": false
}

dapper proxy --control-port 4711 from-config hermit-dap.json
dapper debug --control-port 4711 continue 1
dapper debug --control-port 4711 stack-trace 1
dapper debug --control-port 4711 step over 1

The same adapter command can be used by an editor or another DAP client. To enable reverse execution for a recording, let the adapter manage the replay by adding the recording arguments to spawnConfig:

{
  "spawnConfig": {
    "type": "stdio",
    "cmd": "/path/to/hermit/target/debug/hermit-dap",
    "args": [
      "--replay", "RECORDING_ID",
      "--data-dir", "/path/to/recordings",
      "--gdbserver-port", "1234"
    ]
  },
  "debugRequest": {
    "request": "attach",
    "program": "/path/to/program",
    "target": "127.0.0.1:1234"
  }
}

When its extension loads successfully, the adapter advertises supportsStepBack and implements DAP stepBack and reverseContinue by restarting the deterministic replay and running forward to the requested earlier source position. This first implementation favors correctness over speed.

dapper proxy --control-port 4711 from-config hermit-dap-replay.json
dapper debug --control-port 4711 step back 1
dapper debug --control-port 4711 reverse-continue 1

Chaos Mode Demonstration

The order_violation guest reads shared state without ensuring that another thread has published it. Build Hermit and the guest, then run the default deterministic schedule twice:

cargo build --release -p hermit --bin hermit
mkdir -p target/chaos-demo
cc -std=c11 -O2 -pthread tests/chaos/order_violation.c \
  -o target/chaos-demo/order-violation

for run in 1 2; do
  target/release/hermit run --max-timeslice=disabled -- \
    ./target/chaos-demo/order-violation
done

Both runs print Hello world!. Chaos mode explores other deterministic schedules; this bounded search reports the guest status for each seed:

for seed in {0..15}; do
  target/release/hermit run --chaos --sched-heuristic=random \
    --max-timeslice=disabled --seed="$seed" -- \
    ./target/chaos-demo/order-violation
  printf 'seed=%s status=%s\n' "$seed" "$?"
done

With this Hermit revision and guest, seed 9 prints ERROR! global_str is null at use. and exits 1. Repeating the exact command reproduces the same failure:

target/release/hermit run --chaos --sched-heuristic=random \
  --max-timeslice=disabled --seed=9 -- \
  ./target/chaos-demo/order-violation

The seed identifies a schedule for a particular Hermit build, guest binary, and configuration. Keep all three fixed when reproducing a failure. This demo disables PMU preemption and relies on thread and syscall scheduling points, so it also works on hosts without accessible performance counters.

Record/replay is less broadly compatible than deterministic run mode. Keep the recording directory, executable, inputs, environment, and Hermit revision unchanged between phases.

Hermit presents deterministic mount identities in /proc/*/mountinfo and the matching mnt_id fields in /proc/*/fdinfo/*. Mount topology, stacking order, and non-identity Linux fields remain visible; Hermit-owned temporary roots are rewritten only when their source object has been proved. The scm_fds fdinfo field passes through because it is the socket's queued-descriptor count, not a host-assigned identity. With run --no-namespace, Hermit snapshots the live mount-ID order on first use and refuses if that order changes during the run, rather than applying an identity map to a different host mount-ID layout. A chroot may expose an order-preserving subset of a captured mount table; those rows retain their positions in the captured identity order, including gaps. If a new mount namespace exposes a mount ID absent from the captured order, a mountinfo read refuses instead of assigning a plausible but unproved identity.

Performance

Hermit's deterministic ptrace backend should generally be budgeted at roughly 3-6x native wall-clock time. This is a planning range, not a benchmark promise: overhead varies with syscall frequency, thread count, PMU availability, and the amount of scheduling and logging enabled.

--strict uses the normal deterministic defaults and has the same performance profile as a default run. Chaos, verify, record/replay, and analyze modes may perform multiple executions or retain additional events, so their total cost can be higher. Benchmark your actual workload on the deployment CPU and kernel.

Architecture

Hermit has three main layers:

1. The hermit CLI validates configuration and creates the guest namespaces, mounts, environment, and process tree.
2. Reverie uses ptrace and seccomp-assisted interception to stop and resume the guest around subscribed syscalls and CPU events.
3. Detcore applies deterministic policy: it virtualizes selected results, serializes threads, models resources and logical time, and records or replays external inputs.

Linux still performs most operations. Hermit is a determinization layer, not a replacement kernel or sandbox. See the architecture guide for the event lifecycle, state ownership, scheduler, resource model, virtual time, and record/replay design.

Troubleshooting

Hosts and container runtimes commonly block namespaces, ptrace, seccomp, or perf_event_open. Start with:

hermit run --namespace-only -- /bin/true
hermit --log=info run --strace-only -- /bin/true

If a sandbox denies a writable proc mount, Hermit retries it read-only and prints a warning after setup. Reads still work, but guest writes to procfs may fail with EROFS. Recordings retain the observed proc mode; replay warns if its mode differs. Older recordings without this information remain readable.

These are diagnostic modes and do not provide normal determinism. The User Guide covers host setup, PMU access, program visibility, hangs, verification differences, and record/replay. The Error Catalog maps stable error text to causes and fixes.

Contributing

Focused contributions are welcome. Before opening a pull request:

1. Fork the repository and create a branch from main.

2. Add a focused regression test for behavior changes.

3. Keep generated manifests and documentation consistent with the source.

4. Run formatting and the broadest tests your Linux host supports:

   cargo fmt --all -- --check
   cargo test -p AFFECTED_PACKAGE
   cargo test --workspace
   

5. Document host-dependent skips or failures instead of weakening the test.

Not every x86-64 Linux machine can run every Hermit test: PMU counters, CPUID faulting, perf_event_open permissions, and user/mount namespaces each gate different test groups, so results differ across bare metal, VMs, containers, and WSL. Before reporting an environment-related failure, see Testing Environments for the support matrix, the list of hardware-sensitive tests, expected failure signatures, and a copy-paste diagnostic and bug-report procedure.

See CONTRIBUTING.md for the pull-request, CLA, issue, style, and licensing guidelines.

More Documentation

• Compatibility results: measured outcomes by workload, backend, and mode.
• Test selection catalogue: which tests apply to each backend.
• User Guide: modes, flags, examples, and troubleshooting.
• Architecture: Reverie, Detcore, scheduling, time, and record/replay internals.
• e9patch Compatibility: measured application envelope, preprocessing classifications, and known limits.
• Error Catalog: errors, triggers, and remediations.
• Per-test Hermit Code Coverage: measure and diff the Hermit/Detcore implementation paths exercised by a test.
• Examples: small programs demonstrating controlled nondeterminism.
• License: BSD 3-Clause.