What it does

BenchKit runs established coding, reasoning, knowledge, instruction-following, and long-context benchmarks against local models. It supports OpenAI-compatible servers such as llama.cpp, llama-swap, vLLM, and LM Studio, plus native Ollama.

• Guided terminal UI for connecting, selecting models, running suites, and exploring results
• Headless mode for scripts and CI
• Live streamed responses, reasoning traces, progress, speed, and loop detection
• Direct model evaluation and optional Pi coding-agent evaluation
• Benchmark slicing, tag filters, verifier repair, and choice-order robustness
• JSON, CSV, Markdown, and standalone interactive HTML reports
• Performance profiling and a local dashboard for historical runs

Quick start

BenchKit requires Python 3.11 or newer and uv.

uv sync
cp .env.example .env
uv run benchkit

Set your server in .env, or enter it on the Connect screen:

BENCHKIT_PROVIDER=openai
BENCHKIT_HOST=http://localhost:8080/v1
BENCHKIT_API_KEY=

For Ollama:

BENCHKIT_PROVIDER=ollama
OLLAMA_HOST=http://localhost:11434

No inference server available? Explore the complete UI offline:

uv run benchkit --demo

Run benchmarks

The TUI lets you select models, benchmarks, task limits, and optional run modes. For automation, use headless mode:

# Run the first 20 HumanEval tasks
uv run benchkit --headless --models qwen3:8b --benchmarks humaneval:20

# Run multiple models and benchmarks
uv run benchkit --headless \
  --models qwen3:8b,gemma3:12b \
  --benchmarks sanity,gsm8k:50

# Print prompts, reasoning traces, and responses
uv run benchkit --headless --models qwen3:8b \
  --benchmarks sanity --verbose

Task slices accept 20 for the first 20 tasks, -20 for the last 20, and 40-80 for a range.

Useful commands:

uv run benchkit --list
uv run benchkit --help

Benchmarks

treejs-arena renders each generated page in headless Chromium, so it needs the browser extra first:

uv sync --extra browser
uv run playwright install chromium

The CLI registry is the canonical source for current counts, descriptions, and supported perturbations:

uv run benchkit --list

Advanced runs

Compare raw generation with the stock Pi coding agent (Docker required):

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --harness both

Give incorrect answers one verifier-guided replacement attempt:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --repair-attempts 1

Repairs default to off and can be set from 0–10. One is a practical starting point, but the right depth depends on what you want to measure.

Test whether multiple-choice accuracy survives reordered answer choices:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks mmlu-pro:100 --perturbation choice-order

Decision models served by llama.cpp (/v1/systemone) are detected automatically and run on the multiple-choice and yes/no suites, scored on accuracy plus calibration (Brier, ECE):

llama-server -hf ggml-org/Kev-4B-GGUF
uv run benchkit --headless --models <decision-model> \
  --benchmarks mmlu-pro:100,boolq:100

Behind a proxy that hides this, such as llama-swap (v262+), name the model:

uv run benchkit --headless --models d1-3b,qwen3:8b --benchmarks gpqa \
  --decision-models d1-3b

or mark it once in the llama-swap config:

models:
  d1-3b:
    metadata:
      decision: true

Profile inference performance:

uv run benchkit perf qwen3:8b
uv run benchkit perf qwen3:8b --depths minimal,4k,16k --gen 256 --reps 10

Results

Every run creates a timestamped directory containing:

results/<timestamp>/
├── results.json
├── results.csv
├── results.md
└── results.html

Open results.html for an interactive report with charts and task-level prompts, responses, reasoning traces, timing, and diagnostics.

Browse completed benchmark and performance runs locally:

uv run benchkit history

Development

uv sync
uv run pre-commit install
uv run ruff check .
uv run ruff format .
uv run pytest

See CONTRIBUTING.md for contributor setup and workflow.

License

Apache License 2.0