vla-evaluation-harness — KeyStone self-consistency eval

This fork evaluates KeyStone K-noise self-consistency across diffusion / flow-matching VLA models on SimplerEnv. It replicates the encoded observation, draws K candidate action chunks in one batched forward, and selects one with the cluster-medoid selector from the standalone keystone package.

Setup & reproduction

uv sync --python 3.11 --all-extras --dev
source .venv/bin/activate
uv pip install -e /path/to/keystone   # the selector lives in the keystone package
export WORKSPACE=$HOME/projects       # parent dir holding this repo + cache/

The full step-by-step reproduction guide — setup, run commands, config format, and result parsing — lives in the keystone repo: 👉 https://github.com/dywsjtu/keystone/blob/main/examples/README_vla_eval.md

The short version:

./scripts/run_all_sc_sweeps.sh                       # all pairs, K=1 baseline + sweep
python scripts/parse_sc_results.py --results-dir results --merge

Layout

scripts/
  run_sc_sweep.sh            # per-pair K-sweep runner (--ks "1 4 8 16")
  run_all_sc_sweeps.sh       # all pairs sequentially, one GPU
  parse_sc_results.py        # per-pair tables; --merge for mean ± std
  benchmark_sc_latency.py    # in-process wall + memory at K ∈ {1,4,8,16}

configs/model_servers/<model>/
  <suite>.yaml                                       # K=1 baseline (no SC block)
  <suite>_sc_k{4,8,16}_{medoid,cluster_medoid}.yaml  # KeyStone cells

src/vla_eval/model_servers/self_consistency.py       # re-export shim -> keystone

Docs

• docs/architecture.md — harness architecture.
• docs/design-philosophy.md — design principles.
• docs/tuning-guide.md — per-benchmark tuning notes.

Upstream

This fork tracks allenai/vla-evaluation-harness (Apache 2.0). For the base harness setup — building benchmark Docker images, fetching model weights, and vla-eval serve / run mechanics — see the upstream repo; this fork adds the KeyStone self-consistency layer on top. The unmodified upstream README is preserved on the main branch.

License

Apache 2.0