InfernoSIM

InfernoSIM is an open-source Go sidecar for capturing HTTP traffic and replaying incidents against an isolated service. It supports inbound request capture, outbound dependency stubbing, state-aware replay, fault injection, replay diffs, concurrency pressure, native HTTPS dependency stubbing, semantic request matching, explicit scenarios, OpenAPI release gates, CI reports, encrypted incident bundles, policy-driven privacy controls, explainable matcher healing, generated Testcontainers/Compose/Actions harnesses, Kafka-compatible event replay, AsyncAPI validation, and cross-protocol causal workflows.

v4 adds deterministic reliability testing for tool-using agents: recorded LLM and MCP protocol replay, semantic response faults, an explicit side-effect ledger, consequence assertions, stable stress cases, redacted OpenTelemetry correlation, and CI-native evidence. It remains local-first and needs neither a hosted InfernoSIM service nor a live model during the release gate.

4.0.1 extends this with monitor/state-estimate checks, bound single-use approval assertions, tenant predicates, compensation obligations, shared budgets, crash/restart testing, explicit parallel-call schedules, bounded fault pairs, version comparison, and failure reduction. Start with the runnable 4.0.1 safety and recovery guide. Publication status and validation are tracked in release notes.

v4.1 adds bounded automatic fault/occurrence/order exploration, independent application-state checks, and portable failure reproductions with constraint minimization. The runnable v4.1 discovery guide shows an outbox bug that passes the simulated payment ledger but fails an independent state check. Run bash scripts/discovery-smoke.sh to reproduce the full flow.

Feature guide

v4.2 adds automatic causal-dependency discovery, seeded topological schedules with separate request admission and response release, concurrent HTTP/MCP stdio replay, and individual fault targeting for streamed provider tool calls. Causal failures retain their graph, identity bindings and execution proof through minimization and portable reproduction. Start with the concurrency guide and validation evidence.

Incident to CI in minutes

InfernoSIM already captures and replays incidents. v3.4 productizes the final step: produce a reviewable local test harness, stabilize safe volatile fields, and emit deterministic proof without requiring a hosted service.

# Review a fail-closed matcher proposal learned from repeated observations.
infernosim heal ./incidents/payment-1 \
  --sample ./incidents/payment-2 \
  --sample ./incidents/payment-3

# Generate ordinary, editable Testcontainers-Go source.
infernosim testgen ./incidents/payment-1 \
  --framework go-testcontainers \
  --out ./integration/infernosim

# Or run the dependency simulator directly.
infernosim serve ./incidents/payment-1 \
  --listen 127.0.0.1:19000 \
  --admin-listen 127.0.0.1:19001

See the complete incident-to-test guide, including Kafka, AsyncAPI, workflow verification, self-healing safety boundaries, and the container control API.

Agent incident to reliability gate

v4 can prove how an agent behaves when a tool or model response becomes ambiguous, incomplete, delayed, reset, or semantically malformed. The model is recorded once; CI replays its protocol deterministically and evaluates actual tool-side consequences.

# Inspect the stable baseline and single-fault matrix.
infernosim agent cases ./incidents/refund-agent

# Run your explicit agent test command once per case.
infernosim agent stress ./incidents/refund-agent \
  --report-dir ./artifacts/infernosim-agent \
  -- ./run-agent-tests

# Generate a CI workflow that runs the same assertion engine.
infernosim testgen ./incidents/refund-agent \
  --profile agent --framework github-actions \
  --out ./.github/workflows/infernosim-agent

The release fixture demonstrates tool discovery loss, missing policy fields, malformed LLM arguments, and “effect committed but response lost.” Its safe loop passes all five cases; the unsafe control is rejected in all four fault cases. Reports include an unweighted observed pass-rate surface by fault category and severity—not a generalized model score. See the complete agent reliability guide and raw benchmark evidence.

Install

InfernoSIM v3.4 and later are available from the project Homebrew formula. It builds the tagged source locally so macOS does not require a Gatekeeper bypass:

brew install pranaysparihar/infernosim/infernosim

If you installed the short-lived v3.4 cask before the formula was corrected, migrate once:

brew uninstall --cask infernosim
brew install pranaysparihar/infernosim/infernosim

Release archives and checksums.txt remain available from GitHub for Linux, macOS, and Windows users who do not use Homebrew.

Requirements

• Go 1.26.9 or newer
• Docker with Compose for the container examples
• Linux with NET_ADMIN only for optional transparent replay
• Ollama only for the optional local model-envelope compatibility smoke; it is not required for deterministic agent tests

Build and test

go build -trimpath -o infernosim ./cmd/agent
go test -race ./...
go vet ./...
scripts/agent-smoke.sh
scripts/ollama-smoke.sh  # passes or skips cleanly when Ollama is absent

Release CI also runs fuzz smoke tests, cross-platform builds, multi-architecture container builds, real Testcontainers/Kafka checks, Compose integration profiles, the deterministic incident benchmark, and the safe/unsafe agent reliability oracle. Published assets remain limited to platform archives plus checksums.txt; test JSON, reports, and incident fixtures are never attached. See the release process.

Safety defaults

• Listeners bind to loopback unless another address is explicitly supplied.
• Replay skips POST, PUT, PATCH, and DELETE by default.
• Legacy replay rewrites traffic to --target; it never intentionally calls the captured origin.
• Capture redacts authorization, cookie, API-key, and set-cookie values.
• Raw bodies are fingerprinted but omitted unless --capture-sensitive-data is explicitly supplied or a privacy policy explicitly enables transformed body capture.
• Incident logs and metadata use owner-only file permissions.
• Forward proxies block loopback, link-local, and private destinations unless --allow-private-destinations is supplied.

Do not use --allow-writes, --capture-sensitive-data, --allow-private-destinations, or --insecure-upstream without understanding their scope.

Capture an incident

Start the application, then run:

./infernosim capture \
  --listen 127.0.0.1:18080 \
  --forward 127.0.0.1:8081 \
  --out ./incident-001

This starts an inbound reverse proxy at 127.0.0.1:18080 and an outbound forward proxy at 127.0.0.1:8084. Configure the application process to use the outbound proxy:

HTTP_PROXY=http://127.0.0.1:8084 \
HTTPS_PROXY=http://127.0.0.1:8084 \
./your-application

Exercise the application through the inbound proxy, then press Ctrl-C. The incident directory contains incident.json, inbound.log, and outbound.log.

For a local private dependency and replayable payload bytes:

./infernosim capture \
  --listen 127.0.0.1:18080 \
  --forward 127.0.0.1:8081 \
  --outbound-listen 127.0.0.1:8084 \
  --allow-private-destinations \
  --capture-sensitive-data \
  --out ./incident-001

--capture-sensitive-data can store credentials and PII. Treat such incident bundles as secrets. Existing non-empty bundles are rejected unless --append is explicitly supplied.

Inspect and verify

./infernosim inspect ./incident-001
./infernosim verify ./incident-001

inspect prints the request timeline and discovered state chains. verify reports side effects, missing dependencies, and expired JWTs.

Replay

Safe replay:

./infernosim replay ./incident-001 \
  --target-base http://127.0.0.1:8081 \
  --runs 3

The positional incident directory may appear before the flags. Write requests are skipped unless explicitly enabled:

./infernosim replay ./incident-001 \
  --target-base http://127.0.0.1:8081 \
  --allow-writes \
  --runs 3

Only use --allow-writes with an isolated target whose data may be changed. Inbound-only incidents are supported and produce a weak pass because dependency behavior was not verified.

Replay flags

• --target-base: isolated service receiving captured inbound requests
• --runs: number of replay iterations
• --time-scale: scale captured timing
• --density: compress request gaps
• --min-gap: minimum gap between requests
• --max-wall-time: total replay budget
• --max-idle-time: per-request progress budget
• --max-events: maximum inbound events
• --fanout: concurrent replay workers
• --window: optional SLO completion window
• --inject: dependency latency/timeout/retry rule
• --diff: show status, stable-header, body-hash, and latency changes
• --openapi: validate captured and replayed exchanges against OpenAPI 3.x
• --report-formats: generate junit, sarif, and/or html reports
• --report-dir: choose the report output directory
• --safe-mode: skip writes; enabled by default
• --allow-writes: explicitly disable safe mode
• --stub-listen: primary dependency stub address
• --stub-compat-listen: optional compatibility stub address
• --https-stub: enable native CONNECT/TLS dependency stubbing
• --stub-ca-dir: use an isolated replay CA directory
• --stub-mitm-allow-hosts: allowlist HTTPS dependency hosts

Examples:

./infernosim replay ./incident-001 \
  --target-base http://127.0.0.1:8081 \
  --inject "dep=payments.test latency=+200ms"

./infernosim replay ./incident-001 \
  --target-base http://127.0.0.1:8081 \
  --fanout 8 \
  --window 30s

Replay configuration

An incident may contain replay.yaml:

target: http://127.0.0.1:8081
runs: 3
time_scale: 1.0
safe_mode: true
chaos:
  latency:
    request: 2
    delay: 250ms
state:
  file: ./state.json

Unknown fields and invalid values are rejected. Relative state paths are resolved relative to replay.yaml.

Semantic matching

The matching section allows captured dependency calls to match runtime values without weakening every request:

matching:
  ignored_query_parameters: [timestamp]
  ignored_headers: [X-Request-ID]
  ignored_json_paths: [$.metadata.request_id]
  rules:
    - name: payment
      methods: [POST]
      host_regex: "^payments\\.example\\.test$"
      path_regex: "^/v1/payments/[0-9]+$"
      header_regex:
        X-Tenant: "^tenant-[a-z]+$"
      query_regex:
        mode: "^(sync|async)$"
      jsonpath_regex:
        $.customer.id: "^cust_[0-9]+$"
      compare_headers: true
      compare_json: true

Regexes use Go RE2 syntax. InfernoSIM's deterministic JSONPath subset supports the root $, dotted object keys, and numeric indexes such as $.orders[0].id. Query fields and JSON paths with regex predicates are automatically excluded from exact equality; additional volatile values can be listed in the ignored fields.

Explicit stateful scenarios

Scenarios are evaluated before captured events. A matching step returns its configured response and atomically advances the named scenario:

scenarios:
  - name: authentication
    initial_state: logged_out
    steps:
      - name: login
        state: logged_out
        next_state: logged_in
        match:
          methods: [POST]
          path_regex: "^/login$"
        response:
          status: 200
          headers:
            Content-Type: [application/json]
          body: '{"token":"test-token"}'
      - name: profile
        state: logged_in
        match:
          methods: [GET]
          path_regex: "^/profile$"
        response:
          status: 200
          body: '{"name":"Replay User"}'

Scenario state resets at the beginning of each replay run. See examples/replay-v2.yaml for the v3.2 schema and examples/replay-v3.yaml for templates and descriptor-aware gRPC.

Deterministic dynamic responses

Scenario bodies, headers, trailers, and Protobuf JSON documents can derive values from the runtime request:

templates:
  seed: checkout-ci
  max_output_bytes: 1048576

scenarios:
  - name: orders
    initial_state: ready
    steps:
      - name: create
        state: ready
        match:
          methods: [POST]
          path_regex: "^/orders$"
        response:
          status: 201
          headers:
            Content-Type: [application/json]
            X-Customer: ['{{ jsonPath "$.customer_id" }}']
          body_template: |
            {
              "id": "{{ uuid "order" }}",
              "customer_id": "{{ jsonPath "$.customer_id" }}",
              "created_at": "{{ now }}"
            }

Available functions are jsonPath, proto, header, query, uuid, token, now, nowUnix, toJSON, and default. Generated values are stable for the same seed and request. Templates have bounded source and output sizes and cannot execute programs, access files, read environment variables, or open network connections.

Descriptor-aware gRPC

Load .proto sources or binary FileDescriptorSet files to match Protobuf fields and synthesize typed unary or streaming response frames:

matching:
  grpc:
    proto_files: [grpcapp/echo/echo.proto]
    import_paths: [grpcapp/echo]
  rules:
    - name: echo
      methods: [POST]
      grpc_method: /echo.EchoService/Echo
      protobuf_field_regex:
        $.message: "^hello"
      ignored_protobuf_fields: [$.message]
      compare_protobuf: true

scenarios:
  - name: typed-echo
    initial_state: ready
    steps:
      - name: echo
        state: ready
        match:
          methods: [POST]
          grpc_method: /echo.EchoService/Echo
          protobuf_field_regex:
            $.message: "^hello"
        response:
          status: 200
          headers:
            Content-Type: [application/grpc]
          grpc_status: OK
          protobuf_type: echo.EchoResponse
          protobuf_json: '{"message":"{{ proto "$.message" }} virtualized"}'

Use protobuf_stream instead of protobuf_json to produce multiple response messages. stream_message_delay optionally spaces frames, for example 50ms. Request streams are decoded as an array, so predicates can address fields such as $[0].account_id. Compressed gRPC messages remain available for wire-level replay but are rejected by semantic decoding.

Generate, lint, and explain

Create starting simulations directly from contracts:

infernosim generate --openapi ./openapi.yaml --out replay.yaml
infernosim generate \
  --proto ./proto/payments.proto \
  --import-path ./proto \
  --out replay.grpc.yaml

Generation never overwrites an existing file without --force. Generated examples are deterministic starting points and should be reviewed before use.

Validate configuration structure, template syntax, state reachability, and shadowed matchers:

infernosim lint replay.yaml
infernosim lint --json replay.yaml

Explain matching against every captured outbound call:

infernosim match explain ./incident \
  --request ./request.json \
  --config ./incident/replay.yaml

Standalone capture proxy

Inbound:

./infernosim --mode=inbound \
  --listen 127.0.0.1:18080 \
  --forward 127.0.0.1:8081 \
  --log inbound.log

Outbound with repeatable fault injection:

./infernosim --mode=proxy \
  --listen 127.0.0.1:9000 \
  --log outbound.log \
  --inject "jitter=50ms,drop=5%,status=503,rate=10%" \
  --inject-seed 42

HTTPS MITM

MITM capture is opt-in:

./infernosim --mode=proxy \
  --listen 127.0.0.1:9000 \
  --https-mode mitm \
  --mitm-allow-hosts api.example.test \
  --log outbound.log

The generated CA is stored under ~/.infernosim/ca. The client must trust that CA. Only explicitly allowlisted hosts receive leaf certificates. Local/private MITM additionally requires --allow-private-destinations.

Native HTTPS response stubbing

Replay can terminate application CONNECT/TLS traffic and return captured or scenario responses without contacting the dependency:

stub:
  https:
    enabled: true
    ca_dir: ./.infernosim-ca
    allow_hosts:
      - payments.example.test

Start the application with both proxy variables pointing to the replay stub and trust the printed CA:

HTTP_PROXY=http://127.0.0.1:19000 \
HTTPS_PROXY=http://127.0.0.1:19000 \
SSL_CERT_FILE="$PWD/incident/.infernosim-ca/infernosim-ca.crt" \
./your-application

./infernosim replay ./incident \
  --https-stub \
  --stub-ca-dir ./incident/.infernosim-ca \
  --stub-mitm-allow-hosts payments.example.test

HTTPS stubbing is opt-in and refuses hosts outside the allowlist. It negotiates HTTP/2 with ALPN inside CONNECT and also accepts cleartext HTTP/2 (h2c), so captured gRPC response frames and trailers can be replayed without an HTTP test endpoint. Older captures that contain grpcStatus but no response trailer map are upgraded during replay by emitting the corresponding grpc-status trailer.

Explicit scenarios can return captured gRPC wire data using body_base64, trailers, and grpc_status, or synthesize typed messages with protobuf_json and protobuf_stream when descriptors are configured. body_base64 must contain standard gRPC wire frames, not unframed Protobuf bytes. Payloads larger than the 256 KiB capture bound remain fingerprint-only and cannot be used as captured replay bodies; generated scenario responses use a separate 16 MiB per-message safety bound.

OpenAPI validation and release reports

Validate an incident without replaying it:

./infernosim contract ./incident \
  --spec ./openapi.yaml \
  --report-formats junit,sarif,html

Validate both the baseline and a candidate, including status and content-type drift:

./infernosim replay ./incident \
  --target-base http://127.0.0.1:8081 \
  --openapi ./openapi.yaml \
  --report-formats junit,sarif,html \
  --report-dir ./artifacts

OpenAPI 3.0/3.1 path templates, request/response JSON schemas, local component schema references, required properties, primitive types, arrays, enums, documented statuses, and required response headers are supported. Unsupported external $ref values produce explicit findings instead of being silently accepted.

Kafka, AsyncAPI, and cross-protocol workflows

Capture explicitly selected Kafka-compatible topics with the same local privacy policy model used by HTTP capture:

infernosim kafka capture \
  --brokers 127.0.0.1:9092 \
  --topics payment.authorized \
  --out ./incident/messages.log \
  --privacy-policy ./examples/privacy-kafka.yaml

Validate and replay them with deterministic failure selection:

infernosim kafka validate ./incident \
  --asyncapi ./examples/asyncapi.yaml

infernosim kafka replay ./incident \
  --brokers 127.0.0.1:9092 \
  --asyncapi ./examples/asyncapi.yaml \
  --topic-prefix ci. \
  --drop-every 5 --duplicate-every 7 --reorder-window 3

workflows in replay.yaml declare an ordered causal path across inbound or outbound HTTP, gRPC methods, and Kafka topics. Run infernosim workflow verify ./incident to fail CI on missing, reordered, uncorrelated, or late steps. See examples/replay-v3.4.yaml.

Kafka clients support TLS, mTLS, SASL/PLAIN, and SCRAM. Passwords are read only from an environment variable. v3.4 deliberately claims Kafka-compatible brokers and AsyncAPI 3 JSON payloads—not transparent broker proxying, Avro, or unimplemented messaging systems.

Kafka capture requires a privacy policy unless the unsafe --capture-sensitive-data override is explicit. --direction publish and --direction consume identify how a captured topic relates to the application for causal workflow verification. Policy-based Kafka capture requires capture_bodies: true so the transformed payload remains replayable.

Privacy policies

Privacy policies operate on the copy written to the incident; forwarded traffic is unchanged. Rules can redact, drop, or deterministically tokenize headers, query parameters, and JSON fields:

export INFERNOSIM_TOKEN_KEY='replace-with-at-least-16-random-bytes'
./infernosim capture \
  --forward 127.0.0.1:8081 \
  --privacy-policy ./examples/privacy-policy.yaml \
  --out ./incident

capture_bodies: true stores the transformed body, enabling replay without retaining the original sensitive value. Tokens are tok_ plus a keyed HMAC-SHA256 digest and cannot be reversed. Keep the token key outside the incident. See examples/privacy-policy.yaml.

Encrypted incident bundles v2

Seal a completed incident into an authenticated portable archive:

export INFERNOSIM_BUNDLE_PASSPHRASE='use-a-long-random-passphrase'
./infernosim bundle seal \
  --out incident-001.inferno \
  ./incident-001

./infernosim bundle open \
  --out ./incident-001-opened \
  incident-001.inferno

Bundle v2 uses AES-256-GCM authenticated encryption and derives its key with PBKDF2-HMAC-SHA256 using a random salt and 600,000 iterations. Passphrases are read only from the named environment variable, never a command-line argument. Extraction rejects path traversal, links, oversized files, and non-empty destinations.

Docker examples

scripts/compose-smoke.sh node
scripts/compose-smoke.sh go

The default capture container is unprivileged. Transparent replay requires an explicit root user and NET_ADMIN; do not grant those permissions to ordinary capture deployments.

Development

To contribute, build and validate a focused change before opening a pull request:

go build -trimpath -o infernosim ./cmd/agent
go test -race ./...
go vet ./...
go mod tidy -diff
./infernosim lint examples/replay-v3.yaml

The contributor guide covers deterministic fixtures, fuzzing, container smoke tests, generated Protobuf changes, and security expectations. See SCENARIOS.md, SECURITY.md, upgrade guidance, and the release process for the corresponding operational guidance.

License

MIT. See LICENSE.