redrob-eval

English · 한국어

Open-source LLM evaluation workbench (Next.js App Router, Apache 2.0): generate verifiable evaluation prompts from parametric templates, compare any model from any source on your own task, settle quality by blind human preference, evolve configurations under a quality floor, and serve self-hosted models on your GPU.

The product is five modules plus settings: Generate · Compare · Analyze · Evolve · Deploy. The primary workflow is Generate a deterministic text benchmark, Compare hosted APIs or your own vLLM endpoints, then Analyze the evidence. Deploy is an optional model source and Evolve starts the improvement loop. Image comparison is intentionally out of the product surface for now; its backend adapters remain for later wiring. Costs, where they appear at all, are % of a baseline, never absolute currency. Provider keys stay server-side.

Findings

• Hand-written routing rules (prompt length, keywords) agreed with dual-model labels only about 25% of the time on some GSM8K slices. Not usable as a production policy. (See docs/learnings.md; dual-eval corpora stay under gitignored eval/routing-runs/.)
• A learned router trained on those labels failed to beat chance on the labels we cared about, and was dropped. The labeling pipeline survived; the router did not.
• Tokenizer fertility is a hard budget constraint on high-fertility languages: it shrinks how many demonstrations fit, so demos_requested and demos_fitted diverge and the effective search space narrows.

Sample results (committed)

Regenerate offline with yarn export:samples (no API keys). Full files live in exports/samples/.

Baseline vs evolved (excerpt):

Dataset: in22-gen-hi-en · Optimizer: gepa · Quality floor: 0.5

Baseline  val quality 0.6000 · val tokens 200 · demos 3/2
Evolved   val quality 0.7000 · val tokens 120 · demos 3/3

Token Δ (val): -80
Quality Δ (val): +0.1000
Relative cost vs baseline: 62.5%

Port is fixed at 3939.

Requirements

• Node.js 20+ (tested on 22)
• Yarn Classic 1.22 (yarn via Corepack is fine)
• An OpenRouter API key (recommended) or another supported provider key

Quick start

git clone https://github.com/redrob-labs/redrob-eval.git
cd redrob-eval
cp .env.example .env
# Edit .env and set OPENROUTER_API_KEY=...
yarn install
yarn verify:phase1
yarn verify:gepa
yarn verify:phase3
yarn export:samples
yarn dev

Open http://localhost:3939, which lands on Compare. The other pages are /evolve, /deploy, /generate and /settings. Restart yarn dev after editing .env.

The workbench follows your operating system's light or dark setting, and changes with it. To pin one instead, use the icon at the right of the title bar or the Appearance card in Settings; the choice is stored in the browser, not in .env. yarn verify:theme prints the contrast of every colour pairing in both palettes, and yarn test fails if dark falls below what light manages.

Nothing else is required for a clean checkout - evaluation runs offline against vendored datasets; only provider API calls leave the machine. CI runs every yarn verify:* plus yarn export:samples and yarn build on each push.

What it does

Typical loop: Generate a verifiable prompt set (or pick a catalog dataset) → Compare to pick a model → Evolve under a quality floor → Deploy what you chose, then compare the served endpoint against the frontier again. For tool-routing SLMs: Deploy (install / measure / serve) → Compare with modality Tool routing.

Generate

Generate is parametric generation of verifiable evaluation prompts together with their verifiers. An item is a template plus a seed rather than a row in a file, and the seed is derived from the generator version, the template id and the instance index rather than chosen. That buys two things a static set cannot have. The items cannot have leaked into pretraining, because they did not exist until someone ran the generator. And cherry-picking becomes structurally impossible rather than discouraged, because any third party can recompute the same seeds from published values and check them, which a conventional seed such as 42 does not allow. Scoring is done by deterministic verifiers, not by a judge model.

Underneath the page are the Redrob Verifiable Task Spec v2, a Python generator, verifiers implemented natively in both languages, and a cross-language conformance suite that fails CI if the two ever disagree.

Python stays optional. yarn install && yarn dev is unchanged and still works on a machine with no interpreter installed; CI has a job that builds with Python removed from PATH to keep it that way. The TypeScript side reads and audits generated sets and runs every declarative verifier natively, and it reaches for Python only through an explicit subprocess call that reports absence with an actionable message instead of failing. /generate behaves the same way: without the CLI it still lists the template catalog, which is read from disk, and says plainly that sampling needs an interpreter rather than failing one button at a time.

# optional: only needed to sample instances or run a study
python -m venv .venv-generate
.venv-generate/Scripts/python -m pip install -e packages/generate   # Windows
# .venv-generate/bin/pip install -e packages/generate               # macOS / Linux

# point the workbench at the venv CLI (repo-root .env; not on PATH by default)
# REDROB_GENERATE_CMD=/absolute/path/to/.venv-generate/.../redrob-generate

redrob-generate emit --template templates/math/linear-equation --count 20 --out /tmp/set
redrob-generate verify --set /tmp/set --outputs answers.jsonl --json
yarn test                                            # the TypeScript half of the conformance suite

See packages/generate/README.md for a worked example, packages/generate/STUDY.md for running a study, and templates/README.md for the template layout. Templates are English-only in substance: hi, hi-Latn and ko locale files exist so the pipeline can be exercised end to end, but they hold the English prompt verbatim and are marked untranslated. A real translation needs native-speaker review, none has had one, and a publishable artifact refuses to build over them.

Compare's four stages

1. Setup - pick a text benchmark (catalog, generated handoff, custom JSONL, or Tool routing) and models across every source (curated, OpenRouter, direct frontier, self-hosted vLLM). Generated sets with deterministic references arrive pre-filled and automatically scored.
2. Run - streams live over SSE. Quality is scored only when the task has reference answers; latency, TTFT and throughput are always measured on this run. No cost column, because published pricing is never real time.
3. Preference - one single-elimination bracket per prompt. Two answers at a time with model names hidden, winner advances, champion takes the prompt. Non-power-of-two fields pad with byes; a model that errored on a prompt loses by walkover. For image, "let the judge decide" hands a match to a vision model and you can still vote the rest. Votes append to eval/tournaments/{runId}/votes.jsonl.
4. Optimize route - name a fast model and a fallback. Every prompt the fast one won or tied becomes a small label, the rest escalate. These land in the same RoutingExample corpus the metric-derived collector fills, so /api/routing/export and the training path are unchanged - the supervision is just human now instead of metric.

Offline check for the bracket and label logic: yarn verify:tournament.

Checklist / video skill scoring (custom goal mode: "checklist" or datasets/video-local/ manifests) evolves a judging prompt + frame_policy for agreement with human graders (QWK), not task accuracy. Frames are sampled in memory only - no video bytes are persisted.

Shared rules:

• Provider keys via server .env only (never sent to the browser)
• Costs are relative percentages of a run baseline - never absolute currency
• Metrics return { score, feedback } text alongside the number
• Train/val may be optimized against; test is reported once and the API refuses reporting test if it was also optimized against

Monorepo layout

Workspace packages are marked "private": true (consumed in-repo; not published to npm).

Attribution - GEPA

This repo reimplements GEPA (Genetic-Pareto) in TypeScript from the paper (arXiv:2507.19457). Cite as agrawal2025gepa. See NOTICE. Implementation: packages/harness/src/lib/optimizer/gepa/ - not a file-by-file port of src/gepa/.

On Evolve, pick a catalog dataset or Custom goal (goal + rubric + input-only JSONL; LLM-as-judge). Seed model runs the candidate prompt; reflect model rewrites it from failure feedback; judge (custom mode) scores answers against your rubric. Offline: yarn verify:gepa, yarn verify:phase3, yarn verify:custom-goal. Export reports via GET /api/optimize/runs/:id?export=md.

Docs

• Primary workflow - Generate → Compare → Analyze, optional Deploy, deterministic reference scoring, and the evidence retained between modules
• Contributing - setup, and the branching model: main is production and only takes releases, develop is what you branch from and target
• Changelog - what each tagged release changed, and why
• Security
• Methodology - routing labels, features, export
• Experiment registry - one record every kind of run shares, what yarn runs gives you, and why storage is a configuration value
• Resumable job queue - run a grid of models x datasets x conditions under a concurrency budget, respect per-provider rate limits, and resume where an interrupted run left off
• Failure analysis - why a model was wrong rather than how often: the shared taxonomy, what yarn failures shows, and which failures a prompt fix would clear
• Statistics - whether a difference between two models is real: paired tests, intervals that behave at small denominators, and a warning when a comparison cannot carry a claim
• Preference - blind brackets, ranking and elimination, and how votes become routing labels
• Multi-turn - scripted conversations for text and tools, scored by capability and by how deep into the conversation a model gets
• Tool routing - the sub-10B tool-calling dataset: 324 tasks balanced across four languages, the verbatim ground-truth integrity gate, and the model registry
• Learnings - living design log
• Decision records - why a design is the way it is, one file per decision, numbered and never rewritten in place. Start with 0001 Generate module foundation, then 0002 Unicode semantics, 0003 List-valued verifier field, 0004 Branching model and 0005 Study runner.
• Running a study - the config format for redrob-generate study, a worked example against the mock provider, and why the stub locales cannot be published
• Sample exports - regenerable report + Pareto

Environment

Set every key at /settings in the app, or via the environment. GPU deploy details (GPU_HOST, GPU_USER, GPU_SSH_KEY, …) never leave the gitignored root .env. See deploy/README.md. Do not put hostnames, usernames, key paths, or API keys in the repo.

Self-hosted relative cost

Self-hosted models have no API $/token. Relative cost uses measured throughput:

relativeCostWeight(m) = 100 * (tok_per_sec_large / tok_per_sec_m)

A large model alone = 100 (GPU-time per token). Run Benchmark on /deploy once the model is serving; it records MEASURED_TOK_PER_SEC on the GPU host and the app attributes it to that model and no other. One model is served at a time, so comparing candidates means deploying each in turn. FP8 vs bf16 must appear in result caveats, so never mix precisions in one table without that note.

Indic large-candidate A/B (Gemma 4 31B vs Qwen3.6 27B): compare on the IN22-Gen slice (in22-gen-hi-en).

Keep .env at the repo root. Next loads it via apps/web/next.config.ts.

API (high level)

Optimize / Evolve

• POST /api/optimize → { runId }
• GET /api/optimize/runs/:id/events - SSE
• GET /api/optimize/runs/:id - meta + result + report
• GET /api/optimize/runs/:id?export=md|json - downloadable report

Compare

• POST /api/compare/run - SSE; dispatches on modality (text | image)
• POST /api/compare/tournament - build one bracket per prompt from a run's answers
• GET /api/compare/tournament/:id - meta + brackets + votes + standings
• POST /api/compare/tournament/:id/vote - record a blind vote and advance
• POST /api/compare/tournament/:id/judge - let the modality's model judge decide one match
• POST /api/compare/tournament/:id/route-policy - preference labels, save rate, optional corpus write

Routing collection

• POST /api/routing/collect → { runId }
• GET /api/routing/runs/:id/events - SSE
• GET /api/routing/export?format=chat|flat - training JSONL

Headless

• POST /api/eval - SSE text eval with the optional router baseline (includeRouter)
• POST /api/preference/runs - K×M preference generation, no UI needed
• GET|POST /api/score - metric fixtures and one-off scoring

Also: /api/models, /api/datasets, /api/image/suites, /api/generate/*, /api/status, /api/probe, …

CLI

yarn verify:phase1   # datasets, splits, relative cost helpers
yarn verify:gepa     # GEPA unit checks (offline)
yarn verify:phase3   # script_policy, demo fit, report (offline)
yarn verify:custom-goal  # custom goal parse + judge JSON (offline)
yarn verify:video    # frame_policy, QWK, rubric lint (offline)
yarn verify:tournament   # preference brackets + preference-derived routing labels (offline)
yarn verify:selfhosted   # self-hosted catalog + relative cost, no currency (offline)
yarn verify:preference-gen  # preference run planning + matrix (offline)
yarn export:samples  # write exports/samples report + Pareto SVG
yarn test            # TypeScript conformance suite for the Generate spec (offline)
yarn generate:spec-types  # regenerate TS types from spec/verifiable-task-v2.schema.json
yarn typecheck
yarn lint
yarn build
yarn probe
yarn datasets:fetch  # regenerate vendored JSON (not needed at runtime)
yarn export:routing

Optional: train routers (Python research)

pip install -r train/requirements-mlp.txt
yarn train:mlp
yarn train:event-detect

See train/README.md.

Tips

• Start with 5-20 samples while iterating
• Exact-match datasets default to threshold 0.99
• Oracle on the Pareto chart is the training-target upper bound for the labeling rule
• High tokenizer fertility shrinks demos that fit - watch demos_requested vs demos_fitted on Evolve reports

Author

Built by Janghoon Lee (이장훈)

License

Apache-2.0. Copyright 2026 Redrob. See LICENSE and NOTICE.