spark-infer-bench

Coding-agent workload generation, trace collection and analysis for the Spark-Infer project.

---

Table of contents

• A. What this project does
• B. Architecture
• C. Repository structure
• D. Prerequisites
• E. Installation
• F. The model
• G. Step-by-step first run
• SGLang flags explained
• Docker networking
• H. What success looks like
• I. Trace format
• J. Metrics
• K. Example experiments
• L. Troubleshooting
• M. Manual validation checklist
• Tests
• Git hygiene and large files
• Known assumptions

---

A. What this project does

We want to know what a coding agent actually asks an LLM to do, in enough detail to design an inference engine for it.

The chain we build and measure:

Terminal-Bench              a benchmark of real terminal/coding tasks
      │
      ▼
Pi Coding Agent             reads files, runs tests, edits code, calls the LLM
      │                     many times per task
      ▼
SGLang                      serves the model over an OpenAI-compatible API
      │
      ▼
RadixArk/Qwen3.8-Flash-Next-NVFP4
      │
      ▼
traces                      every LLM request, every tool call, with timing
      │
      ▼
workload analysis           context growth, potential prefix reuse, idle gaps,
                            concurrency, sub-agent structure

Why collect these traces

Local inference for coding agents does not look like chatbot inference:

• Prompts are huge and keep growing. Every tool result is appended to the next request. A task can start at 5K tokens of context and end at 100K.
• Consecutive requests share almost all of their prompt. That is what makes prefix caching and KV reuse potentially worth a lot.
• The agent is often not talking to the model at all. It is running pytest, compiling, installing packages. Those gaps are seconds to minutes long — windows in which an engine could evict, offload to SSD, or prefetch.
• Concurrency is bursty and small. One developer might run 1–5 agents, each of which may spawn sub-agents, rather than 500 independent chat users.

Numbers for each of these are exactly what the Spark-Infer engine work needs, and none of them can be guessed. So we measure them, store them in a stable format, and make the workload replayable against any engine.

What this repository is not

• Not an inference engine.
• Not a fork of SGLang or vLLM. Engines are treated as external services reached over HTTP.
• Not a place where synthetic data may be passed off as results. Synthetic fixtures exist only to test the code; every generated run is stamped "synthetic": true.

---

B. Architecture

  ┌────────────────────────┐
  │  Terminal-Bench task   │   task definition + verifier
  └───────────┬────────────┘
              │ Harbor starts a container and hands over the instruction
              ▼
  ┌────────────────────────┐
  │  Pi coding agent       │   decides what to read, run and edit,
  │  (inside the container)│   and when to call the LLM again
  └───────────┬────────────┘
              │ OpenAI-compatible HTTP
              ▼
  ┌────────────────────────┐
  │  SparkBench proxy      │  ◄── EVERY request/response recorded here
  │  (on the host)         │      timing, tokens, full bodies
  └───────────┬────────────┘
              │ OpenAI-compatible HTTP
              ▼
  ┌────────────────────────┐
  │  SGLang                │   the inference engine
  └───────────┬────────────┘
              ▼
  ┌────────────────────────────────────────┐
  │  RadixArk/Qwen3.8-Flash-Next-NVFP4     │
  └────────────────────────────────────────┘

The components, one by one

Terminal-Bench / Harbor — supplies the tasks and decides pass/fail. Harbor builds a container per task, runs an agent inside it, then runs a verifier. We do not modify Harbor; we register our agent with it.

Pi coding agent — the workload generator. Pi decides what files to read, what commands to run, what to edit, and when to call the LLM again. Its behaviour is the thing we are studying, so we never script it.

SGLang — hosts the model and does the actual inference. External service.

The SparkBench recording proxy — the heart of this repository. It sits between the agent and the engine and records every call at the HTTP boundary. This is the only place that can honestly measure:

It also makes the benchmark agent-agnostic: swap Pi for another agent and the LLM-level trace keeps the same shape.

This repository's pipeline — connects everything and turns raw output into analysis:

raw (immutable)  ──►  parse  ──►  normalized schema  ──►  analysis
                                        │
                                        └──►  replay against any engine

A longer version of this section, including how agents are attributed and how the three kinds of concurrency differ, is in docs/architecture.md.

---

C. Repository structure

spark-infer-bench/
├── README.md                     this file
├── pyproject.toml                package metadata + dependencies
├── .env.example                  template for your local settings
├── .gitignore                    keeps secrets, weights and traces out of git
│
├── configs/
│   ├── models/
│   │   └── qwen38_flash_next_nvfp4.yaml   the research model + reference launch args
│   ├── backends/
│   │   ├── sglang.yaml                     primary backend
│   │   ├── vllm.yaml                       future / unverified
│   │   └── llamacpp.yaml                   future / cannot serve NVFP4
│   ├── agents/
│   │   └── pi.yaml                         how Pi is configured and launched
│   ├── benchmarks/
│   │   └── terminalbench.yaml              Harbor dataset, flags, task ids
│   └── experiments/
│       └── concurrency.yaml                the 1/2/3/5-session sweep
│
├── sparkbench/                   the Python package
│   ├── __init__.py               version + the fixed research model id
│   ├── cli.py                    every `sparkbench …` command
│   ├── util.py                   ids, time, JSONL io, provenance, formatting
│   ├── doctor.py                 read-only environment inspection
│   │
│   ├── config/
│   │   └── loader.py             .env + YAML -> Settings (nothing is hardcoded)
│   │
│   ├── backends/
│   │   ├── base.py               Backend interface + HealthResult
│   │   └── openai_compatible.py  health check, tiny probe, raw POST for replay
│   │
│   ├── agents/
│   │   └── pi.py                 generates Pi's project-local config, runs Pi,
│   │                             streams its JSON events to disk
│   │
│   ├── tracing/
│   │   ├── schema.py             THE normalized trace format + validation
│   │   ├── proxy.py              the recording proxy (ASGI app)
│   │   ├── collector.py          orchestrates one run, writes raw traces
│   │   ├── parser.py             raw (Pi/proxy) -> normalized schema
│   │   ├── prefix.py             potential prefix reuse
│   │   ├── tokenizer.py          model tokenizer, with an honest fallback
│   │   └── synthetic.py          fake traces for testing the pipeline
│   │
│   ├── analysis/
│   │   ├── stats.py              percentiles, describe()
│   │   ├── workload.py           request/token stats, context growth
│   │   ├── tools.py              tool categories, durations, idle gaps
│   │   ├── concurrency.py        agent / request / sub-agent concurrency
│   │   └── summary.py            assembles everything; report + JSON + CSV
│   │
│   ├── replay/
│   │   └── replay.py             re-send a recorded workload to any engine
│   │
│   └── harbor/
│       └── pi_agent.py           SparkBenchPiAgent: runs Pi inside a TB container
│
├── scripts/
│   ├── lib/common.sh             .env loading, helpers (source, do not run)
│   ├── doctor.sh                 sparkbench doctor
│   ├── check_backend.sh          sparkbench backend check
│   ├── start_sglang_qwen38.sh    starts SGLang with the research model
│   ├── start_proxy.sh            starts the recording proxy
│   ├── run_pi_smoke.sh           experiment A: one Pi session, end to end
│   ├── run_terminalbench_smoke.sh experiment B: one Terminal-Bench task
│   ├── run_concurrency.sh        experiment G: 1/2/3/5 session sweep
│   └── patch_harbor_upload_dir.sh optional fix for a known Harbor bug
│
├── examples/
│   ├── simple_repo_task/         a tiny repo with one real bug, for smoke tests
│   │   ├── README.md
│   │   ├── PROMPT.md             the instruction handed to the agent
│   │   ├── textstats/analyze.py  the library (contains the bug)
│   │   └── tests/test_analyze.py one failing test, six passing
│   └── synthetic_traces/
│       └── generate.py           make a fake trace to exercise the pipeline
│
├── traces/
│   ├── raw/                      immutable per-run output (git-ignored)
│   └── processed/                normalized traces (git-ignored)
│
├── results/                      analysis output: summary.json, CSV, PNG
│
├── tests/                        unit tests; no model, no network, no GPU
│   ├── conftest.py               synthetic trace fixtures
│   ├── test_config.py            .env parsing, settings precedence
│   ├── test_schema.py            record derivation + validation
│   ├── test_parser.py            tool categorisation, raw -> normalized
│   ├── test_pi_agent.py          Pi config generation + command line
│   ├── test_prefix.py            common-prefix maths, ratio definition
│   ├── test_analysis.py          workload, tools, gaps, concurrency, summary
│   ├── test_proxy.py             SSE parsing, agent labelling, headers
│   ├── test_replay.py            replay loading and ordering
│   ├── test_harbor_agent.py      container config generation
│   └── test_util.py              ids, JSONL tolerance, formatting
│
└── docs/
    ├── architecture.md           components, data flow, attribution
    ├── trace-format.md           full schema reference with examples
    ├── pi-notes.md               Pi specifics + what to verify
    ├── harbor-notes.md           Harbor specifics + what to verify
    └── metrics.md                every metric defined, with caveats

The files you are most likely to need

---

D. Prerequisites

You do not need a GPU to develop on this repository. Parsing, analysis, prefix maths and the unit tests all run on a laptop against synthetic traces. You need the GPU only to actually serve the model.

---

E. Installation

# 1. clone
git clone https://github.com/bairixie/spark-infer-bench.git
cd spark-infer-bench

# 2. a virtual environment, so nothing is installed system-wide
python3 -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\activate

# 3. the benchmark package, in editable mode so your edits take effect
pip install -e .

# 4. optional extras
pip install -e '.[tokenizer]'        # exact token counts (transformers)
pip install -e '.[plots]'            # PNG figures (matplotlib)
pip install -e '.[dev]'              # pytest
# or everything at once:
pip install -e '.[all]'

# 5. the coding agent (installs globally via npm)
npm install -g --ignore-scripts @earendil-works/pi-coding-agent
pi --version

# 6. the benchmark runner (only needed for Terminal-Bench)
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install harbor
harbor --version

SGLang is not installed by this package: it is large, hardware-specific, and you may already have a build. Install it into the same environment (or a separate one) following NVIDIA/SGLang instructions for your platform, and make sure the build supports this model architecture (qwen4_exp).

Verify the install:

sparkbench --version
sparkbench doctor

---

F. The model

RadixArk/Qwen3.8-Flash-Next-NVFP4

This is the research model for Spark-Infer. A different model has a different tokenizer, a different context window and a completely different KV/state footprint, so its workload is not comparable. If you ever do run something else, sparkbench prints a warning and the run is stamped with the model actually used.

Storage. Allow ~150 GB free for weights plus the Hugging Face cache. Put the cache somewhere with room:

export HF_HOME=/data/hf
huggingface-cli login          # or: export HF_TOKEN=hf_...

Memory. On a unified-memory machine (DGX Spark / GB10) the weights, the KV/state cache and the rest of the system share one pool. That is what --mem-fraction-static controls, and it is the first thing to lower if the server dies at startup.

Why the architecture matters to us. This is not a plain dense-attention model: the gated-delta-net (mamba-like) state is managed separately from the attention KV cache, which is why SGLang has --mamba-* flags. When you later reason about "KV cache" for this model, remember there are two kinds of state.

---

G. Step-by-step first run

Work through these in order. Each step is independent and verifiable — if something fails, you know exactly which layer is at fault. Nothing here has been run for you.

You will end up using three terminals. Keep them open:

Terminal 1   SGLang            (the model server)
Terminal 2   recording proxy   (optional for step 9, required for step 12)
Terminal 3   everything else

Step 1 — Clone the repository

git clone https://github.com/bairixie/spark-infer-bench.git
cd spark-infer-bench

Step 2 — Create a Python environment

python3 -m venv .venv
source .venv/bin/activate

Everything below assumes this environment is active.

Step 3 — Install benchmark dependencies

pip install -e '.[all]'
sparkbench --version

-e (editable) means your edits to sparkbench/ take effect without reinstalling.

Step 4 — Check the system

./scripts/doctor.sh

This only inspects. It never installs or changes anything.

You should see a table of components with ok / warn / MISSING, then either No blockers. or a list of blockers. Roughly:

status  component            detail                         matters for
------  -------------------  -----------------------------  ------------------
ok      Python               3.11.9 (/…/.venv/bin/python)   blocker
ok      python: yaml         6.0.1                          blocker
ok      python: httpx        0.27.0                         blocker
ok      python: uvicorn      0.30.1                         blocker
ok      pi (coding agent)    <some version>                 blocker
ok      docker               Docker version …               needed-for: Terminal-Bench
MISSING harbor               not found                      needed-for: Terminal-Bench
ok      NVIDIA driver / GPUs 1 GPU(s): …                    needed-for: serving the model
warn    disk free (HF cache) 90.0 GiB at /home/…/.cache/…   needed-for: model download

Read it like this:

• blocker and not ok → stop and fix it; nothing will work.
• needed-for: Terminal-Bench missing → steps 1–11 still work fine.
• needed-for: model download warning → fix before downloading weights.
• warn on optional Python packages is fine; you lose exact token counts or plots, and the reports say so.

Step 5 — Authenticate with Hugging Face

Only needed to download the checkpoint.

export HF_HOME=/data/hf            # somewhere with ~150 GB free
huggingface-cli login              # paste a token with access to the repo
# or, non-interactively:
export HF_TOKEN=hf_xxxxxxxxxxxx

Optionally pre-download so the first server start is not also a 135 GB download:

huggingface-cli download RadixArk/Qwen3.8-Flash-Next-NVFP4 --local-dir /data/models/qwen38-nvfp4

Never put a real token in .env.example or any committed file.

Step 6 — Configure .env

cp .env.example .env
$EDITOR .env

A typical single-GPU local setup:

BACKEND=sglang
BASE_URL=http://127.0.0.1:30000/v1
MODEL_ID=RadixArk/Qwen3.8-Flash-Next-NVFP4
TOKENIZER_ID=RadixArk/Qwen3.8-Flash-Next-NVFP4
API_KEY=dummy

PROXY_HOST=0.0.0.0
PROXY_PORT=8900
PROXY_URL=http://127.0.0.1:8900/v1
PROXY_URL_IN_CONTAINER=http://host.docker.internal:8900/v1

# if you pre-downloaded the weights:
# MODEL_PATH=/data/models/qwen38-nvfp4
# HF_HOME=/data/hf

Check what the code actually resolved:

sparkbench config show

Step 7 — Start SGLang (Terminal 1)

Look at the command before running anything:

./scripts/start_sglang_qwen38.sh --dry-run

Then start it for real:

TP_SIZE=1 ./scripts/start_sglang_qwen38.sh

What this does: runs pre-flight checks (is SGLang importable, how many GPUs are visible, is the port free, does TP_SIZE exceed the GPU count), prints the exact command, asks for confirmation, and then runs the server in the foreground on port 30000.

What success looks like: SGLang logs model loading (this takes minutes for a 135 GB FP4 checkpoint — the first run also downloads it), then a line saying the server is ready and listening. Leave it running; Ctrl-C stops it.

Every flag is explained in SGLang flags explained. Start with TP_SIZE equal to your GPU count.

Step 8 — Check the backend (Terminal 3)

sparkbench backend check --smoke

Expected shape (your values will differ):

Backend check
  backend           sglang
  base_url          http://127.0.0.1:30000/v1
  configured model  RadixArk/Qwen3.8-Flash-Next-NVFP4
  endpoint reachable  yes   (HTTP 200)
  models advertised (1):
      RadixArk/Qwen3.8-Flash-Next-NVFP4  <- configured model
  model visible     yes
  tiny generation   ok   (1.23s)
      reply: 'ok'
      usage: {'prompt_tokens': …, 'completion_tokens': …}

  RESULT: backend looks usable

If the model list is empty or shows a different id, do not edit MODEL_ID to match — find out why the server is serving something else.

Step 9 — Run the Pi smoke task (Terminal 3)

./scripts/run_pi_smoke.sh

This checks the backend, starts the recording proxy in the background, copies examples/simple_repo_task to a scratch working directory, runs Pi against it once, then parses and analyses the trace.

The example repository contains a real, small bug: word_frequencies() does not strip punctuation, so exactly one of seven tests fails. The agent is asked to read the code, run the tests, fix the source (not the test), and re-run.

While it runs you should see interleaved output like:

=== run smoke-20260915-101500 ===
  task        simple_repo_task
  agent URL   http://127.0.0.1:8900/v1
  model       RadixArk/Qwen3.8-Flash-Next-NVFP4
  starting Pi ...

  [pi] agent_start
  [proxy] #   1 agent=main     in=4211 out=96  ttft=0.42s lat=3.11s status=200
  [pi] tool  read  textstats/analyze.py
  [pi] tool  done (ok)
  [proxy] #   2 agent=main     in=5820 out=140 ttft=0.55s lat=4.02s status=200
  [pi] tool  bash  python -m pytest -q
  ...

Two things to watch for: [proxy] lines (proof that Pi is really talking to your local model) and the in= numbers growing (context accumulation).

The prompt lives in examples/simple_repo_task/PROMPT.md; override it with --prompt "..." or --prompt-file <path> if you want.

Step 10 — Find the generated trace

sparkbench list
ls traces/raw/<RUN_ID>/

traces/raw/<RUN_ID>/
  run_meta.json      what ran, where, with which versions and git commit
  llm_calls.jsonl    every LLM call with full bodies and timing
  pi_events.jsonl    Pi's JSON event stream
  pi_stderr.log      Pi's stderr
  pi_session.jsonl   copy of Pi's own session file
  proxy.log          proxy output
  pi-home/agent/     the exact Pi config used
  workdir/           the repo copy the agent edited

Quick look:

wc -l traces/raw/<RUN_ID>/llm_calls.jsonl
head -c 400 traces/raw/<RUN_ID>/llm_calls.jsonl

Also worth checking: did the agent actually fix the bug?

cd traces/raw/<RUN_ID>/workdir && python -m pytest -q; cd -

Step 11 — Analyse the trace

Already done by the script, but to repeat it:

sparkbench parse   <RUN_ID>
sparkbench analyze <RUN_ID> --plots

parse turns raw output into the normalized schema at traces/processed/<RUN_ID>.jsonl. analyze prints a report and writes results/<RUN_ID>/ (summary.json, context_growth.csv, prefix.csv, gaps.csv, requests.csv, tool_calls.csv, and PNGs with --plots).

See What success looks like for how to read the report.

Step 12 — Run one Terminal-Bench task

First make Harbor itself work, with no model involved at all:

harbor --version
harbor run -d [email protected] -a oracle       # the reference solution agent

If the oracle does not pass, the problem is Harbor or Docker, not this repository. Fix that first.

Then find a task id and check container→host networking:

harbor run --help                  # confirm the flag names for your version
sparkbench net suggest             # candidate URLs for PROXY_URL_IN_CONTAINER

Now, Terminal 2, start the proxy and leave it running:

./scripts/start_proxy.sh

and in Terminal 3:

TASK_ID=<a-real-task-id> DRY_RUN=1 ./scripts/run_terminalbench_smoke.sh   # look first
TASK_ID=<a-real-task-id> ./scripts/run_terminalbench_smoke.sh

The script verifies the proxy is up, prepares the trace directory, prints the exact harbor run command, asks for confirmation, runs it, then finalizes, parses and analyses.

Harbor writes its own results (ctrf.json, reward.txt) telling you whether the task passed. Harbor's exit code is not the verdict.

Step 13 — Analyse the Terminal-Bench trace

sparkbench analyze <RUN_ID> --plots

Same report as step 11, with task success filled in from the verifier.

A step 0 you can do today, without a GPU

The whole pipeline can be exercised on synthetic data:

python examples/synthetic_traces/generate.py
sparkbench parse   synthetic-001
sparkbench analyze synthetic-001 --plots
pytest

Nothing here touches a model, and every generated run is stamped "synthetic": true. Use it to learn the commands and to check your install before the GPU work starts — never as a source of results.

---

SGLang flags explained

scripts/start_sglang_qwen38.sh builds this command. Defaults come from the model card (configs/backends/sglang.yaml), except --tp.

Anything else can be appended with EXTRA_ARGS="...".

On DGX Spark specifically, expect to revisit: TP_SIZE (GPU count), MEM_FRACTION_STATIC (unified memory is shared with the system), CONTEXT_LENGTH (262K reservations are expensive), and FP4_GEMM_BACKEND (aarch64 kernel availability).

---

Docker networking

When SGLang and the proxy run on the host while Pi runs inside a container (which is what Terminal-Bench does), localhost inside the container is the container, not your machine. So the container cannot reach http://127.0.0.1:8900/v1.

Two settings matter:

1. The proxy must bind all interfaces: PROXY_HOST=0.0.0.0 (the default).
2. The container needs a URL that routes to the host: PROXY_URL_IN_CONTAINER.

Candidates, most portable first:

sparkbench net suggest       # prints candidates for THIS machine

Connectivity check (do this before the first Terminal-Bench run)

With the proxy running:

# from the host — should print JSON
curl -sS http://127.0.0.1:8900/__sparkbench/health

# from inside a container — should print the same JSON
docker run --rm --add-host=host.docker.internal:host-gateway curlimages/curl \
  -sS -m 5 http://host.docker.internal:8900/__sparkbench/health

Whichever URL works goes into PROXY_URL_IN_CONTAINER. Our Harbor agent runs the same probe during setup and fails the run immediately if the container cannot reach the proxy — an empty trace is worse than an error.

Also check your firewall: on Linux, a host firewall can silently drop container→host traffic on port 8900.

---

H. What success looks like

Exact numbers depend on your machine, the task and the model's behaviour, so none are given here. Judge by shape.

sparkbench doctor

A table plus either No blockers. or a blocker list. Missing Harbor/Docker is fine until step 12.

sparkbench backend check

endpoint reachable yes, the configured model listed under "models advertised", model visible yes, and with --smoke a short reply and a usage dictionary. Anything else means the server is not ready.

sparkbench run pi / run_pi_smoke.sh

• [proxy] #N … lines appear — Pi is using your local model.
• in= (prompt tokens) grows over the run — context is accumulating.
• Tool lines show real work: reading files, running pytest.
• At the end: a non-zero number of event lines and a raw trace directory.

Failure mode to watch for: Pi finishes quickly with no [proxy] lines. That means it never called your model (see Troubleshooting).

sparkbench parse

parsed traces/raw/<RUN_ID>
  agents        1
  llm requests  14
  tool calls    22
  wrote 38 records -> traces/processed/<RUN_ID>.jsonl

llm requests 0 is a failure, and the command says so explicitly.

sparkbench analyze

A report with these sections, all populated:

• LLM REQUESTS — request count, total input/output tokens, input:output ratio (expect input-heavy, often 10:1 or more), max context, TTFT and latency statistics, and token_sources — you want server_usage.
• CONTEXT GROWTH — first → last prompt size and a growth factor. For a coding agent this should rise substantially across the run.
• POTENTIAL PREFIX REUSE — sequential and best-earlier percentages. For a conversational agent loop these are typically high. Check the unit line: tokens is a real measurement, chars means no tokenizer was available.
• TOOL CALLS — counts and durations by category. test/build should dominate total time if the task involves running tests.
• IDLE GAPS — how much wall-clock time the agent spent not talking to the model, and how much of it tools explain. Large unexplained time is worth investigating (agent overhead, or a stall).
• CONCURRENCY — for a single sequential agent, request concurrency max is
  1. It only rises in concurrency experiments or with parallel sub-agents.

sparkbench replay

A per-request progress line, then a summary comparing replay TTFT/latency against the originals, and n_failed 0.

---

I. Trace format

Three layers, three different purposes:

Why both raw and processed? Raw is evidence: whatever the tools actually emitted, including full request bodies. Replay needs it, and so will analyses you have not thought of yet. Processed is a stable contract: Pi's event names may change, our schema does not, so figures made today still work on traces collected next semester. Only parser.py bridges the two.

Important fields

llm_request — request_id, agent_id, start_ts, first_token_ts, end_ts, ttft_s, latency_s, input_tokens, output_tokens, context_length, cached_tokens, token_source, params, finish_reason, error, prompt_sha256, body_ref.

tool_call — tool_name, category, command, start_ts, end_ts, duration_s, success.

agent — agent_id, parent_agent_id, agent_type, start_ts, end_ts.

run — task, benchmark, backend, model, machine, versions, git commit, seed, success.

body_ref (e.g. llm_calls.jsonl#7) points back into the raw recording; the processed trace deliberately does not duplicate large prompt bodies.

Example: one raw LLM call (trimmed)

{
  "record_id": "req-000003-9f2c",
  "agent_label": "main",
  "endpoint": "/v1/chat/completions",
  "modified_request": ["stream_options.include_usage"],
  "request": {"model": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "stream": true,
              "n_messages": 7, "body": {"messages": ["..."]}},
  "timing": {"t_start": 1789466400.12, "t_first_token": 1789466400.51,
             "t_end": 1789466404.88, "ttft_s": 0.39, "latency_s": 4.76},
  "response": {"status": 200, "finish_reason": "tool_calls",
               "usage": {"prompt_tokens": 18422, "completion_tokens": 142}},
  "tokens": {"prompt": 18422, "completion": 142, "source": "server_usage"}
}

Example: the same call, normalized

{"record_type": "llm_request", "schema_version": 1,
 "request_id": "req-000003-9f2c", "run_id": "smoke-20260915-101500",
 "agent_id": "main", "agent_attribution": "label", "request_index": 2,
 "model": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "stream": true,
 "start_ts": 1789466400.12, "first_token_ts": 1789466400.51,
 "end_ts": 1789466404.88, "ttft_s": 0.39, "latency_s": 4.76, "decode_s": 4.37,
 "input_tokens": 18422, "output_tokens": 142, "context_length": 18422,
 "token_source": "server_usage", "finish_reason": "tool_calls",
 "body_ref": "llm_calls.jsonl#3"}

modified_request is an honesty field: the proxy adds stream_options.include_usage to streamed requests so the server reports token counts, and every record says whether it did. Disable with sparkbench proxy --no-inject-usage (you then lose token counts on streamed calls).

Full reference: docs/trace-format.md.

---

J. Metrics

In plain language:

Input tokens — how much context was sent to the model for one request. For a coding agent this is the dominant cost: the whole conversation so far, including every file the agent read and every test output.

Output tokens — how much the model generated. Usually far smaller than the input; a 10:1 or 20:1 input:output ratio is normal.

Context length — how much history a request carries. It grows turn by turn, which is why "long context" is not an edge case here but the default.

Potential prefix reuse — how much of a request's prompt is a character-for-character (token-for-token) prefix of an earlier request. If request B is request A plus one appended tool result, almost all of B's prompt is a prefix of A's, so an engine could in principle skip recomputing it. Reported two ways: sequential (versus the immediately preceding request) and best (versus the best matching earlier request, the upper bound for an unbounded radix cache).

This is potential reuse, an upper bound — not a cache hit rate. Whether the engine still holds those blocks depends on its cache size, eviction policy and what else ran in between. Real hit rates come out below these numbers, never above.

TTFT (time to first token) — how long the user waits before anything appears. Dominated by prefill, so it rises with prompt length and with queue depth.

Request latency — the whole request, first byte to last. Includes queueing, so it rises under concurrency even when the model is equally fast.

Tool duration — how long the agent spends outside inference: running tests, building, installing. This is the workload's duty cycle; for many coding tasks the agent spends more time in tools than waiting on the model.

Idle gap — the time between one request finishing and the next starting for the same agent. From the engine's point of view this agent is idle but still holding cache — the window in which offloading or eviction could pay off. The analysis also reports how much of each gap is explained by tool calls.

Agent concurrency — how many agents were alive at once.

Request concurrency — how many LLM requests were in flight at the server at one instant. This is what the engine actually schedules against, and it is measured, never assumed.

Sub-agent parallelism — how many child agents of one session ran at the same time. max_parallel == 1 means the agent spawns children serially; > 1 means genuinely parallel branches.

Keep these three separate: a run with session concurrency 5 may still show request concurrency 2, because agents spend much of their time in tools. That gap between "5 agents" and "2 concurrent requests" is itself a finding.

Full definitions and caveats: docs/metrics.md.

---

K. Example experiments

All of these assume SGLang is running and sparkbench backend check passes.

Experiment 1 — One Pi smoke task

./scripts/run_pi_smoke.sh

Purpose: prove the whole stack works end to end.

Experiment 2 — One Terminal-Bench task

# Terminal 2
./scripts/start_proxy.sh
# Terminal 3
TASK_ID=<task-id> ./scripts/run_terminalbench_smoke.sh

Purpose: a real benchmark task with a verifier that says pass/fail.

Experiment 3 — Three to five Terminal-Bench tasks

./scripts/start_proxy.sh                      # Terminal 2, leave running
for t in task-a task-b task-c task-d task-e; do
  TASK_ID=$t RUN_ID=tb-$t ./scripts/run_terminalbench_smoke.sh
done
sparkbench list

Purpose: enough runs to see whether workload shape is task-dependent. Put the real task ids in configs/benchmarks/terminalbench.yaml (small_task_set) so the set is recorded.

Experiment 4 — Context growth

sparkbench analyze <RUN_ID> --plots
column -s, -t results/<RUN_ID>/context_growth.csv | head -20

Gives request index vs input tokens per agent, the growth factor, mean growth per request, and context_growth.png.

Experiment 5 — Potential prefix reuse

pip install -e '.[tokenizer]'                 # needed for token-level numbers
sparkbench analyze <RUN_ID> --prefix-scope agent    # within one agent
sparkbench analyze <RUN_ID> --prefix-scope run      # across all agents
column -s, -t results/<RUN_ID>/prefix.csv | head -20

--prefix-scope run is the interesting one once sub-agents exist: it shows how much a child could reuse from its parent.

Check the unit line in the report. chars means no tokenizer was loaded and the numbers are only a relative signal.

Experiment 6 — Tool gaps

sparkbench analyze <RUN_ID> --long-gap 5
column -s, -t results/<RUN_ID>/gaps.csv | head -20

Reports every idle window between LLM requests, which tools ran inside it, and how much time tools do not explain. Lower --long-gap to catch shorter windows; raise it to focus on the big ones.

Experiment 7 — Sub-agents

Sub-agents are not required for the first runs, and Pi core ships without them. When you enable them (extension or Pi package):

# configs/agents/pi.yaml
# subagents:
#   enabled: true
#   extension: <name>

sparkbench analyze <RUN_ID> --prefix-scope run

The report's CONCURRENCY section then shows sub-agent count, max parallelism, lifetimes, and whether they ran in parallel or serially; CROSS-AGENT PROMPT OVERLAP shows parent→child and sibling prompt sharing.

To compare, run the same task twice — once with sub-agents disabled, once enabled — and diff the two summary.json files.

Experiment 8 — Concurrency 1 vs 2 vs 3 vs 5

./scripts/run_concurrency.sh                    # sweeps 1 2 3 5
# or one level at a time:
sparkbench run concurrency --level 3 --task examples/simple_repo_task --auto-parse

Each level launches N independent Pi sessions sharing one recording proxy, so request concurrency at the server is measured correctly. Compare levels:

python - <<'PY'
import json, glob
for f in sorted(glob.glob("results/conc*.json")):
    d = json.load(open(f)); a = d["aggregate"]
    print(f"{d['session_concurrency']} sessions: "
          f"max_req_conc={a['request_concurrency_max']} "
          f"mean={a['request_concurrency_mean']:.2f} "
          f"wall={a['wall_clock_s']:.0f}s "
          f"ttft_p95={(a['ttft_s']['p95'] or 0):.2f}s")
PY

The interesting question: how far below N does measured request concurrency sit, and how much does TTFT degrade as N rises?

Experiment 9 — Replay the same trace on SGLang

# as fast as possible, one at a time
sparkbench replay <RUN_ID> --timing immediate --concurrency 1

# stress: same requests, 4 at a time
sparkbench replay <RUN_ID> --timing immediate --concurrency 4

# faithful: reproduce the agent's original think-time
sparkbench replay <RUN_ID> --timing original

# same shape, twice as fast
sparkbench replay <RUN_ID> --timing original --speed 2

Replay re-sends the exact recorded request bodies, so it turns a slow, non-deterministic agent run into a fast, repeatable engine benchmark. Results land in results/replay/….

--timing immediate inflates apparent prefix-cache benefit, because nothing gets evicted during a 45-second pytest that no longer happens. --timing original preserves those idle windows and is the honest comparison.

Adding another backend later

1. Copy configs/backends/sglang.yaml to configs/backends/<engine>.yaml and set base_url / model_id.
2. Start that engine yourself.
3. sparkbench --backend <engine> backend check
4. sparkbench --backend <engine> replay <RUN_ID> --timing original

Because everything goes through the OpenAI-compatible abstraction, no benchmark code changes. Do not claim the NVFP4 checkpoint works on another engine until you have verified it: vLLM is unverified for this checkpoint, and llama.cpp serves GGUF, which NVFP4 is not.

---

L. Troubleshooting

General rule: find the lowest layer that is broken and fix that one. The layers, bottom up: GPU/driver → SGLang → /v1/models → proxy → Pi → Harbor/Docker → analysis. sparkbench doctor and sparkbench backend check tell you where you are.

Hugging Face: permission denied / 401 / 403

The repo may be gated, or your token lacks access.

huggingface-cli whoami
huggingface-cli download RadixArk/Qwen3.8-Flash-Next-NVFP4 --repo-type model --dry-run

Accept any terms on the model page, then re-login. Do not switch to a different model to get around it.

Not enough disk / download stops partway

~135 GB of weights plus cache overhead.

df -h $HF_HOME ${HF_HOME:-~/.cache/huggingface}
huggingface-cli scan-cache

Point HF_HOME at a big volume and re-run the download — it resumes. A truncated download often shows up later as a confusing safetensors error, so verify the download completed before blaming SGLang.

SGLang does not recognise the model / missing qwen4_exp

The architecture is new; your SGLang build must support it.

python -c "import sglang; print(sglang.__version__, sglang.__file__)"
python -m sglang.launch_server --help | head -40

Collect the full traceback — the useful line usually names an unknown architecture or a missing module. Upgrade SGLang to a build that supports this model family. Do not work around it by changing the model.

ModelOpt / FP4 / FlashInfer errors

Symptoms: unknown quantization method modelopt_fp4, missing FP4 kernels, or an import error from flashinfer.

python -c "import modelopt; print(modelopt.__version__)"
python -c "import flashinfer; print(flashinfer.__version__)"
nvidia-smi --query-gpu=name,driver_version --format=csv

These packages are tightly coupled to CUDA/driver versions, and aarch64 wheels lag x86. Try a different FP4_GEMM_BACKEND and record which backend you used when reporting numbers.

CUDA error / GPU not visible

nvidia-smi
echo $CUDA_VISIBLE_DEVICES
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

TP_SIZE must not exceed the visible GPU count — the start script checks this and refuses.

Out of memory at startup or mid-run

In order of effect:

MEM_FRACTION_STATIC=0.70 CONTEXT_LENGTH=65536 MAX_RUNNING_REQUESTS=8 \
  ./scripts/start_sglang_qwen38.sh

CONTEXT_LENGTH is usually the biggest lever: reserving 262K tokens of KV/state is expensive. On unified-memory machines remember the system and the model share one pool — close other memory-hungry processes. If OOM happens mid-run rather than at startup, the agent's context has grown; lower MAX_RUNNING_REQUESTS or CONTEXT_LENGTH.

aarch64 (DGX Spark / GB10) issues

pip may fail to find wheels for GPU-side packages.

uname -m           # aarch64
pip debug --verbose | grep -i tag

Use NVIDIA's ARM builds/containers. sparkbench itself is pure Python and installs anywhere.

Port already in use

lsof -i :30000      # SGLang
lsof -i :8900       # proxy

Change PORT / PROXY_PORT in .env (and keep BASE_URL / PROXY_URL in sync).

/v1/models unavailable

curl -v http://127.0.0.1:30000/v1/models

Connection refused → the server is not up (or is still loading; watch Terminal 1). Hangs → it is still loading the checkpoint. 404 → your BASE_URL is missing or duplicating the /v1 path.

Backend check says the model is not visible

The server is running but advertises a different id. Look at what it lists and at the --model-path you launched with. Do not change MODEL_ID in .env to match unless you genuinely intend to change the research model.

Pi cannot connect / Pi runs but no LLM requests are recorded

The single most common failure. Check in this order:

# 1. is the proxy up?
curl -s http://127.0.0.1:8900/__sparkbench/health

# 2. does the generated Pi config point at the proxy?
cat traces/raw/<RUN_ID>/pi-home/agent/models.json     # baseUrl must be the PROXY

# 3. can the proxy reach the engine?
sparkbench backend check

# 4. what did Pi say?
tail -40 traces/raw/<RUN_ID>/pi_stderr.log

# 5. does Pi see the provider at all?
PI_CODING_AGENT_DIR=$(pwd)/traces/raw/<RUN_ID>/pi-home/agent pi --list-models

Also verify PI_CODING_AGENT_DIR is honoured by your Pi version (see docs/pi-notes.md); if it is not, Pi read your personal ~/.pi config instead and may have called a cloud provider.

Pi hangs at the first tool call

It is waiting for approval. Check the flag:

pi --help | grep -i approve

and fix run.approve_args in configs/agents/pi.yaml.

Requests return HTTP 400 from the server

Often an unsupported field. The compat block in configs/agents/pi.yaml already disables the developer role and reasoning_effort. To see the offending body:

tail -1 traces/raw/<RUN_ID>/llm_calls.jsonl | python -m json.tool | head -40

The proxy records failures too, so the request that failed is in the trace.

Docker cannot reach SGLang / the proxy

See Docker networking and run the two curl checks there. Confirm PROXY_HOST=0.0.0.0, and check any host firewall.

Harbor agent configuration error

harbor run --help
harbor agent list
python -c "import sparkbench.harbor.pi_agent as m; print(m.__file__)"
PYTHONPATH=$(pwd) harbor run -d [email protected] \
  --agent-import-path sparkbench.harbor.pi_agent:SparkBenchPiAgent -m openai/x -n 1

sparkbench must be importable by the Python that runs Harbor — PYTHONPATH is the simplest fix. If the base class or environment.exec differs in your version, only sparkbench/harbor/pi_agent.py needs editing; _exec() already tries several spellings and tells you what to change.

Terminal-Bench container problems

First prove Harbor works without us:

harbor run -d [email protected] -a oracle
docker ps -a
docker system df           # containers/images can fill the disk quickly

If verifiers fail because a tests/ directory got nested, see the known upload_dir issue and scripts/patch_harbor_upload_dir.sh.

Tokenizer mismatch

The report says unit: chars or token_source: estimate.

pip install -e '.[tokenizer]'
python -c "from transformers import AutoTokenizer; \
  AutoTokenizer.from_pretrained('RadixArk/Qwen3.8-Flash-Next-NVFP4', trust_remote_code=True)"

If TOKENIZER_ID differs from MODEL_ID, sparkbench warns — the numbers are then not comparable to the served model.

Analysis shows 0 requests or 0 tool calls

• 0 requests → the proxy was not in the path (see above).
• 0 tool calls → pi_events.jsonl is empty. Check --mode json is being used (traces/raw/<RUN_ID>/run_meta.json records the exact command) and look at pi_stderr.log.

What to send back if you are stuck

Send:

1. the exact command you ran and its full output;
2. sparkbench doctor --json /tmp/doctor.json (attach the file);
3. traces/raw/<RUN_ID>/run_meta.json;
4. the last 50 lines of the relevant log — pi_stderr.log, proxy.log, or the SGLang terminal;
5. head -c 2000 traces/raw/<RUN_ID>/llm_calls.jsonl if it exists;
6. sparkbench config show.

That is almost always enough to identify the layer at fault. Redact tokens.

---

M. Manual validation checklist

Work down the list. Each item is independently checkable, and they are in dependency order — if one fails, the ones below it will too.

[ ] pip install -e '.[all]' succeeds and `sparkbench --version` works
[ ] pytest passes (no model needed)
[ ] synthetic pipeline works:
        python examples/synthetic_traces/generate.py
        sparkbench parse synthetic-001 && sparkbench analyze synthetic-001
[ ] doctor works                      ./scripts/doctor.sh  -> "No blockers."
[ ] SGLang starts                     ./scripts/start_sglang_qwen38.sh
[ ] /v1/models responds               curl http://127.0.0.1:30000/v1/models
[ ] target model is visible           sparkbench backend check --smoke
[ ] proxy starts and is healthy       curl http://127.0.0.1:8900/__sparkbench/health
[ ] Pi connects                       [proxy] lines appear during a Pi run
[ ] Pi can use tools                  tool lines appear; workdir is modified
[ ] raw trace is created              traces/raw/<RUN_ID>/llm_calls.jsonl is non-empty
[ ] processed trace is created        sparkbench parse <RUN_ID>
[ ] trace validates                   sparkbench validate <RUN_ID>
[ ] analyze command works             sparkbench analyze <RUN_ID> --plots
[ ] prefix analysis works             report shows unit "tokens", not "chars"
[ ] tool analysis works               tool categories and durations look sane
[ ] replay works                      sparkbench replay <RUN_ID> --timing immediate --limit 5
[ ] Docker can reach the proxy        the container curl check succeeds
[ ] Harbor works on its own           harbor run -d [email protected] -a oracle
[ ] one Terminal-Bench task runs      ./scripts/run_terminalbench_smoke.sh
[ ] concurrency experiment works      sparkbench run concurrency --level 2 --auto-parse
[ ] subagent experiment works         (only after enabling sub-agents in Pi)

---

Tests

pip install -e '.[dev]'
pytest                       # all tests
pytest -v                    # verbose
pytest tests/test_prefix.py  # one file
pytest -k concurrency        # by name

The tests cover configuration parsing and precedence, schema validation and derived fields, trace parsing and tool categorisation, common-prefix maths and the reuse-ratio definition, tool/gap statistics, concurrency sweep-line maths, summary rendering and CSV output, SSE parsing and agent labelling in the proxy, replay loading and ordering, the generated Pi configuration (including the assertion that Pi is pointed at the proxy and never straight at the engine), and the Harbor container-config generation.

They deliberately use synthetic fixtures only: no model, no network, no GPU, no Docker. Anything that needs those is a manual validation step, which is what section M is for. Synthetic data never leaves the tests as a result.

---

Git hygiene and large files

.gitignore already excludes: .env, auth.json, any hf_token*, model weights (*.safetensors, *.gguf, *.bin), Hugging Face caches, traces/raw/*, traces/processed/*, results/*, .pi-home/, node_modules/, and Python build artefacts.

Never commit: Hugging Face tokens, API keys, model weights, Docker layers, or raw benchmark datasets.

Where to put large traces. Raw traces grow with prompt size — a long Terminal-Bench run can produce tens of MB of llm_calls.jsonl, because full prompts are stored. Keep them out of git and archive them deliberately:

tar czf spark-infer-traces-$(date +%Y%m%d).tar.gz traces/raw/<RUN_ID>

Then store on a lab volume, shared drive, or object storage, and record the archive location in your experiment notes. What belongs in git is the results/<RUN_ID>/summary.json if you want a number in version control — it is small and carries the provenance (git commit, versions, machine).

Reproducibility. Every run records: run id, timestamps, git commit and dirty flag, model and revision, backend and URL, versions of Pi / Harbor / SGLang / Docker / driver, machine and GPUs, task, seed, session concurrency and the full effective configuration. Credentials are never stored.

---

Known assumptions

These are the things that could not be confirmed from documentation alone and should be checked once on your machine. Each is marked [VERIFY] in the source, and each is isolated to a single file so a fix is small.

Environment-dependent things to expect on DGX Spark: aarch64 wheel availability for FP4/FlashInfer/ModelOpt, unified-memory pressure (tune MEM_FRACTION_STATIC and CONTEXT_LENGTH), GPU count for TP_SIZE, and Docker's host-gateway behaviour for container→host networking.

---

License

See LICENSE.