llm-d-benchmark

This repository provides an automated workflow for benchmarking LLM inference using the llm-d stack. It includes tools for deployment, experiment execution, data collection, and teardown across multiple environments and deployment styles.

[!TIP] We acknowledge many users are still utilizing our previous (now deprecated) library, and to make the transition easier, we still have that library available. It can be found in our v0.5.2 version tag.

Main Goal

Provide a single source of automation for repeatable and reproducible experiments and performance evaluation on llm-d:

• Declarative lifecycle: All infrastructure, workloads, and experiments render into reviewable YAML before provisioning.
• End-to-end automation: A single llmdbenchmark CLI covers standup, benchmarking, result collection, and teardown.
• Reproducibility: A deterministic config merge chain (defaults.yaml to scenario to CLI overrides) captures the exact configuration in each workspace. Any result traces back to its inputs.
• Structured experiments: Built-in Design of Experiments (DoE) support automates parameter sweeps across both infrastructure and workload configurations.
• Multiple harnesses: Swap between inference-perf, guidellm, vllm-benchmark, and others with a CLI flag (-l).
• Post-deployment validation" Per-scenario smoketests verify that deployed pod configurations match what the scenario defines -- resources, parallelism, env vars, probes, routing, and vLLM flags.

Prerequisites

Please refer to the official llm-d prerequisites for the most up-to-date requirements. For the client setup, the provided install.sh will install the necessary tools.

Administrative Requirements

Deploying the llm-d stack requires cluster-level admin privileges, as you will be configuring cluster-level resources. However, the scripts can be executed by namespace-level admin users, as long as the Kubernetes infrastructure components are configured and the target namespace already exists.

Getting Started

Install

The install script supports both uv and the standard python3 -m venv for virtual environment creation. When run interactively, it will prompt you to choose; in non-interactive mode (e.g. curl pipe), it auto-selects uv if your system Python is missing or older than 3.11. You can also pass --uv or --no-uv to skip the prompt.

Quick install (one-liner):

curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/main/install.sh | bash
cd llm-d-benchmark
source .venv/bin/activate
llmdbenchmark --version

Or clone manually:

git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
./install.sh              # or: --uv / --no-uv
source .venv/bin/activate
llmdbenchmark --version

Install a specific branch:

LLMDBENCH_BRANCH=main \
  curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/main/install.sh | bash

The install script auto-detects if the repo is present -- if not, it clones it first. It creates a virtualenv, validates system tools (kubectl, helm, Python 3.11+), and installs the llmdbenchmark package. See Installation for manual install and flags.

[!TIP] The last line of output from llmdbenchmark standup shows the workspace path where all rendered configs, manifests, and results are stored.

Pick your path: with or without Accelerators

Two supported entry points depending on what you have access to:

🖥️ No Accelerators / No Cluster Access - Utilize a Kind Quickstart

Run the full standup -> smoketest -> run -> teardown lifecycle on a local Kind cluster using a simulated inference engine. No accelerators, no cloud account, no cluster operator required. It uses the same cicd/kind-sim scenario that CI runs on every PR, so if it works locally it works in CI.

• Requirements: Docker (or Podman/Colima) with 4 CPUs / 8 GiB RAM and Python 3.11+
• Continue with Quick Start Guide: Quickstart on Kind (or try the simpler EPP+KEDA Saturation Autoscaling guide)

🚀 Access to Compute cluster with Accelerators - full pipeline

Deploy against a Kubernetes cluster with Accelerators (OpenShift, GKE, EKS, CKS, Intel XPU, etc.). Use one of the built-in specs or a well-lit path guide tuned for your hardware.

• Requirements: cluster admin to install infra (or utilize an namespace admin with infra pre-installed), kubeconfig, compute nodes
• Continue below with Choose a specification and Deploy and benchmark

Choose a specification

Every command takes a --spec that selects the configuration for your cluster and GPU type. Specs are Jinja2 templates under config/specification/:

--spec gpu                                      # NVIDIA GPU setup (config/specification/examples/gpu.yaml.j2)
--spec guides/optimized-baseline                # optimized baseline guide (formerly inference-scheduling)
--spec guides/workload-autoscaling              # optimized baseline + WVA autoscaling
--spec guides/epp-keda-saturation               # optimized baseline + direct EPP+KEDA autoscaling (no WVA controller)
--spec examples/multi-model-optimized-baseline  # multi-model optimized baseline: N pools, 1 gateway, 1 shared HTTPRoute
--spec pd-disaggregation                       # prefill-decode disaggregation guide
...
--spec /full/path/to/my-spec.yaml.j2            # custom spec

If the name is ambiguous or not found, the CLI lists all available specs and exits.

Deploy and benchmark (full pipeline)

Stand up the llm-d stack, run a quick sanity benchmark, and tear down:

# Preview what would be deployed (no cluster changes)
llmdbenchmark --spec gpu --dry-run standup

# Deploy for real
llmdbenchmark --spec gpu standup

# Run a sanity benchmark against the deployed endpoint
llmdbenchmark --spec gpu run -l inference-perf -w sanity_random.yaml

# Tear down when done
llmdbenchmark --spec gpu teardown

[!NOTE] --dry-run renders all manifests and logs every command that would execute, without touching the cluster. Use it to review before deploying.

Each command renders Kubernetes manifests from your spec's templates and defaults, then applies them. The workspace directory captures rendered configs, manifests, and results for later inspection.

Deploy multiple models behind one gateway

The multi-model-optimized-baseline scenario is the optimized-baseline guide deployed N times: N models under a single gateway, each with its own EPP + InferencePool + decode Deployment, behind one HTTPRoute with N backendRefs:

# Standup - renders two stacks (qwen3-06b, llama-31-8b), installs shared
# infra once, deploys a per-model Helm release for each.
llmdbenchmark --spec examples/multi-model-optimized-baseline standup -p my-namespace

# Smoketest - runs stack-by-stack (sequential), hitting each pool at its
# routing prefix (/qwen3-06b/v1/models, /llama-31-8b/v1/models).
llmdbenchmark --spec examples/multi-model-optimized-baseline smoketest -p my-namespace

# Run - iterates every stack, each harness pod targets its own pool's endpoint.
llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace

# See what's deployed: list detected endpoints + copy-paste run commands.
llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace --list-endpoints

# Benchmark just one pool (no --endpoint-url needed - auto-resolves):
llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace \
  --stack qwen3-06b \
  -l inference-perf -w sanity_random.yaml

# Teardown - removes both stacks and the shared infra.
llmdbenchmark --spec examples/multi-model-optimized-baseline teardown -p my-namespace

Stack names (qwen3-06b, llama-31-8b) double as path prefixes on the shared HTTPRoute (/qwen3-06b/v1/..., /llama-31-8b/v1/...). Pick short descriptive names in your own scenario - --list-endpoints prints the rendered URLs so you rarely have to type them manually.

Discovering what's deployed (--list-endpoints)

After standup, --list-endpoints detects each pool's routing URL, prints a copy-paste-ready table, and exits without launching any harness pods:

llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace --list-endpoints

📋 Detected endpoints:
  STACK        MODEL                      ENDPOINT URL
  -----------  -------------------------  ------------------------------
  qwen3-06b    Qwen/Qwen3-0.6B            http://10.1.2.3:80/qwen3-06b
  llama-31-8b  unsloth/Meta-Llama-3.1-8B  http://10.1.2.3:80/llama-31-8b

💡 Copy-paste to benchmark one pool:

  # qwen3-06b - Qwen/Qwen3-0.6B
  llmdbenchmark --spec examples/multi-model-optimized-baseline run \
    --namespace my-namespace \
    --endpoint-url http://10.1.2.3:80/qwen3-06b \
    --model Qwen/Qwen3-0.6B \
    -l <harness> -w <workload.yaml> -j <parallel-pods>
  ...

Example Targeting a single pool (--stack)

--stack NAME restricts any lifecycle command to one pool (or a comma-separated subset). Endpoint URL auto-resolves for the selected stack - no need to pass --endpoint-url manually:

# Benchmark qwen3-06b only with guidellm, two parallel harness pods
llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace \
  --stack qwen3-06b \
  -l guidellm \
  -w sanity_random.yaml \
  -j 2

Breakdown of the Example:

• --stack qwen3-06b filters per-stack steps to that pool. Endpoint detection (step 03) runs only for that stack and auto-resolves to http://<gateway>:80/qwen3-06b - including the routing prefix - so every downstream step targets the qwen3-06b InferencePool.
• -l guidellm selects the guidellm harness (workload/harnesses/guidellm-llm-d-benchmark.sh).
• -j 2 launches two guidellm pods hitting the same endpoint. Both pods run the same treatment (-w) but write to distinct result subdirectories ({experiment_id}_1, {experiment_id}_2) on the workload PVC.

Want to compare pools side-by-side? Launch two invocations in parallel shells (different --workspace each):

# Terminal 1 - --workspace is a global option, placed before the subcommand
llmdbenchmark --spec examples/multi-model-optimized-baseline --workspace /tmp/run-qwen run -p my-namespace \
  --stack qwen3-06b \
  -l guidellm -w sanity_random.yaml -j 2 &

# Terminal 2 (or same shell, backgrounded)
llmdbenchmark --spec examples/multi-model-optimized-baseline --workspace /tmp/run-llama run -p my-namespace \
  --stack llama-31-8b \
  -l guidellm -w sanity_random.yaml -j 2

--stack also works on standup, smoketest, and teardown. Same flag, same semantics - restrict execution to the named subset of stacks without editing the scenario YAML. Scenario-wide steps (namespace creation, admin prereqs, shared infra) always run; only the per-stack steps (06+ for standup) are filtered.

# Standup only pool qwen3-06b from the multi-model scenario - shared
# infra (istio, Gateway, model PVC) installs normally, but only
# qwen3-06b's ms/gaie resources get created.
llmdbenchmark --spec examples/multi-model-optimized-baseline standup -p my-namespace \
  --stack qwen3-06b

# Standup two named pools out of a larger scenario:
llmdbenchmark --spec examples/multi-model-optimized-baseline standup -p my-namespace \
  --stack qwen3-06b,llama-31-8b

# Tear down just one pool later, leaving the other running:
llmdbenchmark --spec examples/multi-model-optimized-baseline teardown -p my-namespace \
  --stack qwen3-06b

Unknown stack names fail loudly with a list of valid ones.

When --stack NAME selects exactly one stack, -m/--models scopes to that stack only - sibling stacks keep their scenario-defined models. Handy for "rerun pool A against a different model" without touching pool B:

llmdbenchmark --spec examples/multi-model-optimized-baseline run -p my-namespace \
  --stack qwen3-06b \
  --model meta-llama/Llama-3.2-3B \
  -l inference-perf -w sanity_random.yaml

Without --stack, -m applies to every stack and emits a warning.

Add a third model by copying a stack block in config/scenarios/examples/multi-model-optimized-baseline.yaml and changing name + model. Scenario-wide config (gateway class, shared HTTPRoute, EPP plugin config, Envoy and InferencePool tuning) lives in the top-level shared: block and is inherited by every stack. See the developer guide's Multi-Stack Scenarios section for the merge semantics.

Benchmark an existing endpoint (run-only mode)

Already have a model-serving endpoint running? Skip deployment entirely:

llmdbenchmark --spec gpu run \
  --endpoint-url http://10.131.0.42:80 \
  --model meta-llama/Llama-3.1-8B \
  --namespace my-namespace \
  --harness inference-perf \
  --workload sanity_random.yaml

This uses the same harness, profile rendering, and result collection pipeline -- just without the standup and teardown phases.

[!TIP] run can also be used in debug mode (-d / --debug) which starts the harness pod with sleep infinity so you can exec into it and run commands interactively. See this example.

See workload/README.md for the full experiment file format and all pre-built experiments, as well as advanced functionality. Worked examples for sweeping the EPP plugins config (router.epp.pluginsConfigFile) and the Kubernetes pod scheduler (schedulerName) -- including the kustomize propagation caveat and a dry-run verification one-liner -- live under workload/README.md#sweeping-epp-plugins-config-routerepppluginsconfigfile.

Next Steps

---

Prerequisites

Please refer to the official llm-d prerequisites for the most up-to-date requirements.

System Requirements

• Python 3.11+
• kubectl -- Kubernetes CLI
• helm (>= 4.x) -- Helm package manager
• curl, git -- Standard system tools
• helmfile (>= 1.5) -- Required for modelservice deployments. Older helmfile is incompatible with Helm 4 (it probes helm with the removed helm version --client flag and panics). ./install.sh installs the pinned Helm 4 / helmfile combination for you.
• jq, yq -- Required for template rendering
• kustomize (optional) -- The kustomize deploy path uses kubectl apply -k, which has kustomize built in; the standalone binary is only a convenience
• skopeo, crane (optional) -- Used to resolve :auto image tags; any one of skopeo, crane or podman is enough
• zstd (optional) -- Reads a compressed result set back out of its archive. Without it a run collects uncompressed instead of failing
• oc (optional) -- Required for OpenShift clusters (either kubectl or oc must be present)

Administrative Requirements

[!IMPORTANT] Deploying the llm-d stack requires cluster-level admin privileges for configuring cluster-level resources. Namespace-level admin users can run the tool if Kubernetes infrastructure components are configured and the target namespace already exists. Use --non-admin to skip admin-only steps.

Installation

Quick Install (recommended)

# One-liner -- auto-clones if needed
curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/main/install.sh | bash
cd llm-d-benchmark
source .venv/bin/activate

Or manually:

git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
./install.sh              # or: --uv / --no-uv
source .venv/bin/activate

The install script:

1. Creates a Python virtual environment at .venv/ (via uv or python3 -m venv - see Install)
2. Validates Python 3.11+ and pip
3. Checks for required system tools (curl, git, kubectl or oc, helm, helmfile, jq, yq) and best-effort installs the optional ones (kustomize, skopeo, crane, zstd)
4. Installs the helm-diff plugin (required by helmfile)
5. Installs llmdbenchmark and planner (from llm-d-planner)
6. Verifies all Python packages are importable

Manual Install w/o Install Script

git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
pip install "git+https://github.com/llm-d-incubation/llm-d-planner.git@v0.1.0"

Verify Installation

llmdbenchmark --version

CLI Reference

Global Options

Plan Options

Standup Options

Teardown Options

Experiment Options

Run Options

Smoketest Options

Run post-deployment validation independently against an already-deployed stack.

llmdbenchmark --spec gpu smoketest -p my-namespace
llmdbenchmark --spec gpu smoketest -p my-namespace -s 2   # config validation only

Smoketests also run automatically after standup unless --skip-smoketest is passed. See llmdbenchmark/smoketests/README.md for details on what each step validates.

Environment Variables

Every CLI flag can be set via a LLMDBENCH_* environment variable (see tables above). The priority chain is:

1. CLI flag (highest) -- explicitly passed on the command line
2. Environment variable -- exported in the user's shell
3. Rendered config (lowest) -- defaults.yaml + scenario YAML

This is useful for CI/CD pipelines, .bashrc configuration, or migrating from the original bash-based workflow.

# Example: set common defaults via env vars, override per-run via CLI
export LLMDBENCH_SPEC=guides/optimized-baseline
export LLMDBENCH_NAMESPACE=my-team-ns
export LLMDBENCH_KUBECONFIG=~/.kube/my-cluster

# These use the env vars above; --dry-run overrides nothing, just adds a flag
llmdbenchmark standup --dry-run
llmdbenchmark standup                          # live deploy to my-team-ns
llmdbenchmark standup -p override-ns           # CLI wins over env var

Boolean env vars accept 1, true, or yes (case-insensitive). Active LLMDBENCH_* overrides are logged at startup for debugging.

Quieting the plan-rendering output

standup, smoketest, teardown, run and experiment all render the plan before they do anything else. That render narrates itself in detail -- one line per template, plus image overrides and per-stack banners -- which for a typical scenario is 40+ lines per stack, enough to push the phase output you are actually watching off the screen.

By default those commands now print a two-line summary instead:

✅ Plan rendered: 40 manifest(s) across 1 stack(s) -> /.../workspace/plan
📝 Per-file render detail suppressed (--no-quiet-plan or -v to show; always recorded in /.../workspace/logs)

plan is the exception -- the render narration is that command's output, so it stays verbose by default.

Nothing is thrown away. The suppressed lines are demoted to DEBUG, not dropped, so they are still written to <workspace>/logs/llmdbenchmark-stdout.log. Warnings and errors from the render are never quieted.

# full narration on a lifecycle command (one-off)
llmdbenchmark standup --spec gpu --no-quiet-plan

# ... or for a whole shell / CI job
export LLMDBENCH_QUIET_PLAN=false

# just the summary from `plan`, when you only want the files on disk
llmdbenchmark plan --spec gpu --quiet-plan

# --verbose always wins and shows everything
llmdbenchmark -v standup --spec gpu

Precedence: --quiet-plan / --no-quiet-plan > LLMDBENCH_QUIET_PLAN > per-command default, with --verbose overriding all three.

Compressed output

Output is compressed by default (--no-compress opts out). A result set is dominated by native harness JSON -- per_request_lifecycle_metrics.json alone reaches ~1.5 GB per run -- and the pipeline is generate, compress, copy:

• the harness pod produces every per-result-set artifact (reports, summaries, plots, stage-clipped metrics) before anything is compressed;
• each result set is then compressed in place on the PVC, so the archive rather than the raw tree crosses the apiserver exec tunnel. This is a transfer speedup as much as a storage one, and it composes with --fast-collect;
• the archive is copied down as-is. Nothing is compressed, expanded, or re-analysed on the driver.

A small keep-plain set is left as real files so the collected tree stays usable without touching the archive:

<workspace>/
├── latest -> <user>-<timestamp>/
└── <user>-<timestamp>/
    ├── plan/<scenario>/                       # teardown reads it live
    ├── analysis/<experiment_id>/
    │   └── distributions/*.png                # plain
    └── results/<experiment_id>/
        ├── benchmark_report_v0.2,_*.yaml      # plain
        ├── run_metadata.yaml                  # plain
        └── workspace.tar.zst                  # everything else

Four keep-plain entries, each earning it: the benchmark reports and run_metadata.yaml are what results_store globs off the live filesystem to resolve a run's uid/model/hardware, experiment-summary.yaml is a DoE run's only index, and the plots are the artifact people open (already-compressed bytes, so archiving them buys nothing).

Everything else -- the per-request JSON, logs including the raw Prometheus snapshots, metric summaries, CSV, HTML, traces -- lives in workspace.tar.zst, and every component that reads one of those goes through the archive rather than requiring a plain copy: the cross-treatment overlays, summary extraction, the eval-containers roll-up and per-task reports, the failure validator, and the FMA comparison table. CI's log-dump steps read through util/dump_result_file.sh.

Inspect an archive without expanding it:

tar -I zstd -tf workspace.tar.zst                        # list contents
tar -I zstd -xOf workspace.tar.zst ./logs/stdout.log     # one member to stdout
tar -I zstd -xf workspace.tar.zst                        # expand in place

Grep one member through -xOf as above. Piping the whole archive does not work: the tar padding reads as binary, so plain grep prints nothing and grep -a prints the surrounding tar block rather than the matching line. Level 10 is the default because it is the speed/size knee; --compress-level raises it for archival runs.

zstd must be present in the benchmark image. Images predating it are detected by a probe and fall back to plain collection with a warning, never a failure. Compression is also skipped when the harness did not finish (a wait timeout, or --wait-timeout 0), since deleting files the harness may still be writing is not recoverable.

zstd is needed on the driver too, to read a collected archive back. install.sh installs it best-effort; without it the run collects uncompressed and says so, so a host that cannot supply the package still works.

llmdbenchmark results add <path> and UID lookups behave identically on a compressed and an uncompressed workspace: the plain files stay at results/<experiment_id>/, and plan/, which the store reads the scenario name from, is never touched.

Architecture

The tool operates in three phases, each composed of numbered steps executed by a shared StepExecutor framework.

Config Override Chain

Values flow through a merge pipeline during the plan phase:

Steps read from the rendered config.yaml and never define their own fallback defaults. If a required key is missing from the rendered config, the step raises a clear error. This ensures defaults.yaml is the single source of truth for all default values. Environment variables (LLMDBENCH_*) sit between scenario overrides and CLI flags in the priority chain.

See config/README.md for the full configuration reference, including how to override values.

Deployment Methods

The standup phase supports two deployment paths:

• standalone -- Direct Kubernetes Deployments and Services for each model (step 06)
• modelservice -- Helm-based deployment with gateway infrastructure, GAIE, and LWS support (steps 07-09)

Both paths share steps 00-05 (infrastructure, namespaces, secrets) and step 10 (smoketest).

Standup Steps

Run Steps

Teardown Steps

Project Structure

config/                       Declarative configuration (all plan-phase inputs)
    templates/
        jinja/                Jinja2 templates for Kubernetes manifests
        values/defaults.yaml  Base configuration with all anchored defaults
    scenarios/                Deployment overrides (guides/, examples/, cicd/)
    specification/            Specification templates (guides/, examples/, cicd/)

llmdbenchmark/                Python package
    cli.py                    Entry point, workspace setup, command dispatch
    config.py                 Plan-phase workspace configuration singleton

    interface/                CLI subcommand definitions (argparse)
        commands.py           Command enum (plan, standup, teardown, run, experiment)
        env.py                Environment variable helpers for CLI defaults
        plan.py               Plan subcommand
        standup.py            Standup subcommand
        teardown.py           Teardown subcommand
        run.py                Run subcommand
        experiment.py         Experiment subcommand (DoE orchestration)

    parser/                   Plan-phase template rendering (see parser/README.md)
        render_specification.py   Specification file parsing and validation
        render_plans.py           Jinja2 template rendering engine
        render_result.py          Structured error tracking for renders
        config_schema.py          Pydantic config validation (typo/type detection)
        version_resolver.py       Auto-resolve image tags and chart versions
        cluster_resource_resolver.py  Auto-detect accelerator/network values

    experiment/               DoE experiment orchestration (see experiment/README.md)
        parser.py             Parse experiment YAML (setup + run treatments)
        summary.py            Per-treatment result tracking and summary output

    executor/                 Execution framework (see executor/README.md)
        step.py               Step ABC, Phase enum, result dataclasses
        step_executor.py      Step orchestrator (sequential + parallel)
        command.py            kubectl/helm/helmfile subprocess wrapper
        context.py            Shared state (ExecutionContext dataclass)
        protocols.py          Structural typing (LoggerProtocol)
        deps.py               System dependency checker

    smoketests/               Post-deployment validation (see smoketests/README.md)
        base.py               Health checks, inference tests, pod inspection helpers
        report.py             CheckResult / SmoketestReport tracking
        steps/                Smoketest step implementations (00-02)
        validators/           Per-scenario config validators

    standup/                  Standup phase (see standup/README.md)
        preprocess/           Scripts mounted as ConfigMaps in vLLM pods
        steps/                Step implementations (00-11)

    teardown/                 Teardown phase (see teardown/README.md)
        steps/                Step implementations (00-05)

    run/                      Run phase (see run/README.md)
        steps/                Step implementations (00-11)

    logging/                  Structured logger with emoji support, plus the QuietLogger console-quieting proxy (see logging/README.md)
    exceptions/               Error hierarchy (Template, Configuration, Execution)
    utilities/                Shared helpers (see utilities/README.md)
        cluster.py            Kubernetes connection, platform detection
        capacity_validator.py GPU capacity validation
        huggingface.py        HuggingFace model access checks
        endpoint.py           Endpoint discovery and model verification
        profile_renderer.py   Workload profile template rendering
        kube_helpers.py       Shared kubectl patterns (wait, collect, cleanup)
        cloud_upload.py       Unified cloud storage upload (GCS, S3)
        os/
            filesystem.py     Workspace and directory management
            platform.py       Host OS detection

See module-level READMEs for detailed documentation:

• executor/README.md -- Execution framework and step contribution guide
• smoketests/README.md -- Post-deployment validation and per-scenario config checking
• standup/README.md -- Standup phase details
• run/README.md -- Run phase, benchmark execution, result collection
• teardown/README.md -- Teardown phase details
• experiment/README.md -- DoE experiment orchestration
• parser/README.md -- Plan-phase rendering pipeline
• logging/README.md -- Logger, stream separation, file logging
• utilities/README.md -- Shared utilities, workspace architecture

Well-Lit Path Guides

llm-d-benchmark supports all available Well-Lit Path Guides. Each guide has a corresponding specification:

llmdbenchmark --spec guides/optimized-baseline standup  # Optimized baseline (formerly inference-scheduling)
llmdbenchmark --spec pd-disaggregation standup          # Prefill-decode disaggregation
llmdbenchmark --spec tiered-prefix-cache standup        # Tiered prefix cache
llmdbenchmark --spec precise-prefix-cache-aware standup # Precise prefix cache-aware routing
llmdbenchmark --spec wide-ep-lws standup                # Wide expert-parallel with LWS

[!WARNING] wide-ep-lws requires RDMA/RoCE networking and LeaderWorkerSet (LWS) controller. Verify your cluster has working RDMA HCAs before deploying.

Main Concepts

Model ID Label

Kubernetes resource names derived from model IDs use a hashed model_id_label format: {first8}-{sha256_8}-{last8}. This keeps resource names within DNS length limits while remaining identifiable. The label is computed automatically during the plan phase and used in template rendering for deployment names, service names, and route names. See config/README.md for details.

Scenarios

Cluster-specific configuration: GPU model, LLM, and llm-d parameters. Scenarios are YAML files under config/scenarios/ that override defaults.yaml for a particular deployment context.

Harnesses

Load generators that drive benchmark traffic. Supported: inference-perf, guidellm, vllm benchmarks, inferencemax, and nop (for model load time benchmarking).

(Workload) Profiles

Benchmark load specifications including LLM use case, traffic pattern, input/output distribution, and dataset. Found under workload/profiles.

[!IMPORTANT] The triplet <scenario>, <harness>, <(workload) profile>, combined with the standup/teardown capabilities, provides enough information to reproduce any single experiment.

Experiments

Design of Experiments (DOE) files describing parameter sweeps across standup and run configurations. The experiment command automates the full setup x run treatment matrix -- standing up a different infrastructure configuration for each setup treatment, running all workload variations, tearing down, and producing a summary. See llmdbenchmark/experiment/README.md for the full experiment lifecycle documentation.

Benchmark Report

Results are saved in the native format of each harness, as well as a universal Benchmark Report format (v0.1 and v0.2). The benchmark report is a standard data format describing the cluster configuration, workload, and results of a benchmark run. It acts as a common API for comparing results across different harnesses and configurations. See benchmark-report/README.md for the full schema documentation and Python API.

Analysis

The analysis pipeline generates per-request distribution plots, cross-treatment comparison tables and charts, and Prometheus metric visualizations. Analysis runs both inside the harness container (automatically) and locally via --analyze. For interactive exploration, a Jupyter notebook is also available at docs/analysis/README.md.

Dependencies

• llm-d-infra
• llm-d-modelservice v0.4.14
• inference-perf
• guidellm
• vllm
• inferencemax

News

• KubeCon/CloudNativeCon 2025 North America Talk "A Cross-Industry Benchmarking Tutorial for Distributed LLM Inference on Kubernetes", with the accompanying tutorial
• llm-d-benchmark supports all available Well-Lit Path Guides
• Data from benchmarking experiments is made available on the main project's Google Drive

Topics

• Analysis Pipeline
• Metrics Collection
• Benchmark Report
• Design of Experiments (DoE)
• Lifecycle
• Run
• Agentic evaluation (eval-containers)
• Benchmarking Agent (Agent Core)
• Running eval-containers on OpenShift
• Standup
• Kustomize deploy method
• Benchmarking SGLang
• No-Kubernetes (nok8s) deploy method
• Reproducibility
• Observability
• Quickstart
• Resource Requirements
• Autoscaling: WVA & EPP+KEDA Saturation
• Upstream Versions
• FAQ

Testing

Unit tests live under tests/ and run with pytest:

pytest tests/ -v

For integration testing against a live cluster, util/test-scenarios.sh runs standup/teardown cycles across scenarios:

util/test-scenarios.sh --stable     # Run known-stable scenarios
util/test-scenarios.sh --trouble    # Run scenarios that have had issues
util/test-scenarios.sh --all        # Run all scenarios
util/test-scenarios.sh --ms-only    # Modelservice scenarios only
util/test-scenarios.sh --sa-only    # Standalone scenarios only

See tests/README.md for unit test details.

Developing

• Developer Guide -- How to add new steps, analysis modules, harnesses, scenarios, and experiments
• Package Architecture -- Overview of the llmdbenchmark package structure and submodules

Contribute

• How to contribute, including development process and governance.
• See Developer Guide for how to add new steps, harnesses, scenarios, and analysis modules.
• Join Slack (sig-benchmarking channel) for cross-org development discussion.
• Bi-weekly contributor standup: Tuesdays 13:00 EST. Calendar | Meeting notes | Google group

License

Licensed under Apache License 2.0. See LICENSE for details.