EvoGen-Harness

Learning Where and How to Evolve Image-Generation Harnesses

Same model. Smarter harness.

Paper   ·   Project & Gallery   ·   Quick Start   ·   Documentation   ·   中文

Jiabin Luo¹   Yinan Liu²   Chunlei Meng³   Yufei Guo^1,*
1 Peking University   ² Beijing University of Technology   ³ Fudan University
_{* Corresponding author}

EvoGen-Harness is the official implementation of our generator-agnostic framework for evolving the persistent system around a frozen image generator. Rather than optimizing only prompts or a fixed workflow, it exposes five responsibilities—Policy, Tools, Skills, Middleware, and Memory—to targeted improvement.

TRACE (Trajectory-Relative Attribution and Coordinated Evolution) answers two questions: where should the harness change, and how should it change? It aggregates visual evidence across stochastic executions, uses attribution to focus edit proposals, re-localizes residual failures, and keeps only validated updates. No-Patch is an explicit option when persistent change is not justified.

Overview

Figure 1 from the technical report. The image generator stays frozen; the external harness is what evolves.

See the evolution

Figure 4: successive updates repair different visual constraints without changing generator weights. Explore the interactive comparisons for a closer look.

Paper results

Overall scores from Tables 1–3 of the paper. Higher is better; Δ is an absolute score difference, not a percentage improvement.

The strongest evaluated baseline in these three tables is Nano Banana. The main EvoGen-Harness configuration uses frozen FLUX.1-dev, GPT-4.1, and OWLv2 + NVILA internal verification. The paper also reports 87.9–91.4% responsibility attribution recall, 94.8% No-Patch accuracy, and 1.9% regression.

These are paper-reported results, not outputs of the CPU quick start below. The comparisons are system-level, not a claim of equal generator backbones or zero additional compute. See Tables 4–5 and 9 for same-backbone comparisons, inference cost, and evolution cost.

Quick start

1. Install the lightweight development environment

Run commands from the repository root. Python 3.11 is the reference version for the supplied GPU environment; the lightweight tools also run on newer Python versions.

git clone https://github.com/King-play/EvoGen-Harness.git
cd EvoGen-Harness
conda create -n genharness python=3.11 -y
conda activate genharness
pip install --upgrade pip
pip install -r requirements.txt

2. Check the code and the supplied data—no GPU or API key

python -m gen_harness.cli --help
python -m pytest -q -rs
python scripts/check_release.py
python scripts/prepare_evolution.py --output runs/quickstart/evolution_tasks.jsonl

The last command validates and combines the included 500 target / 100 held-out / 100 preservation prompts. It does not download or re-sample data. Counts, checksums, disjointness, and the exact-match overlap scope in the supplied manifest are checked before writing a working copy. Details: evolution data.

3. Estimate an evolution run before spending compute

python -m gen_harness.cli self-evolve-run \
  --harness examples/visual_harness \
  --tasks runs/quickstart/evolution_tasks.jsonl \
  --output-dir runs/quickstart/evolution \
  --backend-config configs/backends/flux1_dev_diffusers.json \
  --inspection-config configs/inspection/composite_owlv2_nvila_strong.json \
  --policy-extractor-config configs/policy_extractors/openai_chat_policy_extractor.json \
  --harness-localizer-config configs/patch_proposers/openai_chat_harness_localizer.json \
  --patch-proposer-config configs/patch_proposers/openai_chat_patch_proposer.json \
  --trace-dataset-protocol paper_p2 \
  --seeds 0,1,2,3 --max-rounds 3 --dry-run-budget

This is a budget-only check: no image generation, no paid API calls, and no harness updates. It checks the 500/100/100 split contract, not the full model environment. Do not simply remove --dry-run-budget before completing runtime setup and the evolution workflow.

Run with real models

Use the GPU installation and environment guide to install the supplied inference dependencies, set model paths, and configure the LLM endpoint. Then start with a small real generation run:

python -m gen_harness.cli generate \
  --dataset geneval2 \
  --harness examples/visual_harness \
  --tasks data/geneval2/geneval2_tasks_prompt_only_smoke32_2026_09_04.jsonl \
  --output-dir runs/geneval2_smoke \
  --backend-config configs/backends/flux1_dev_diffusers.json \
  --inspection-config configs/inspection/composite_owlv2_nvila_strong.json \
  --policy-extractor-config configs/policy_extractors/openai_chat_policy_extractor.json \
  --harness-localizer-config configs/patch_proposers/openai_chat_harness_localizer_fastloop.json \
  --patch-proposer-config configs/patch_proposers/openai_chat_patch_proposer_fastloop.json \
  --seeds 0 --limit 1

This command calls real models and may incur API costs. --limit 1 is a functional smoke test, not a benchmark result. examples/visual_harness is the supplied harness configuration; this release does not identify it as the historical final paper checkpoint.

The two LLM configuration pairs are intentionally distinct: the unsuffixed files are used for persistent TRACE evolution, and the *_fastloop.json files for transient per-request repair. All generation uses internal verification; official benchmark scores are computed separately after generation.

Benchmark evaluation

Persistent evolution, generation-time verification, and official scoring are separate stages. The complete three-benchmark workflows are collected below and in the evaluation guide.

export EVOGEN_EVAL_HARNESS="artifacts/my_run/final_harness"

python -m gen_harness.cli build-tasks \
  --dataset geneval2 \
  --benchmark-data /path/to/geneval2_data.jsonl \
  --output data/geneval2/tasks.jsonl

python -m gen_harness.cli generate \
  --dataset geneval2 \
  --harness "$EVOGEN_EVAL_HARNESS" \
  --tasks data/geneval2/tasks.jsonl \
  --output-dir runs/geneval2 \
  --backend-config configs/backends/flux1_dev_diffusers.json \
  --inspection-config configs/inspection/composite_owlv2_nvila_strong.json \
  --policy-extractor-config configs/policy_extractors/openai_chat_policy_extractor.json \
  --harness-localizer-config configs/patch_proposers/openai_chat_harness_localizer_fastloop.json \
  --patch-proposer-config configs/patch_proposers/openai_chat_patch_proposer_fastloop.json

python -m gen_harness.cli export-eval \
  --dataset geneval2 \
  --experiences runs/geneval2/gen_harness_generations.jsonl \
  --benchmark-data /path/to/geneval2_data.jsonl \
  --output runs/geneval2/image_map.json

python -m gen_harness.cli build-tasks \
  --dataset t2icompbenchpp \
  --dataset-dir /path/to/T2I-CompBench++ \
  --output data/t2icompbenchpp/tasks.jsonl

python -m gen_harness.cli generate \
  --dataset t2icompbenchpp \
  --harness "$EVOGEN_EVAL_HARNESS" \
  --tasks data/t2icompbenchpp/tasks.jsonl \
  --output-dir runs/t2icompbenchpp \
  --backend-config configs/backends/flux1_dev_diffusers.json \
  --inspection-config configs/inspection/composite_owlv2_nvila_t2icompbenchpp.json \
  --policy-extractor-config configs/policy_extractors/openai_chat_policy_extractor.json \
  --harness-localizer-config configs/patch_proposers/openai_chat_harness_localizer_fastloop.json \
  --patch-proposer-config configs/patch_proposers/openai_chat_patch_proposer_fastloop.json

python -m gen_harness.cli export-eval \
  --dataset t2icompbenchpp \
  --tasks data/t2icompbenchpp/tasks.jsonl \
  --experiences runs/t2icompbenchpp/gen_harness_generations.jsonl \
  --manifest-dir runs/t2icompbenchpp/eval_manifests \
  --sample-root runs/t2icompbenchpp/eval_samples \
  --copy

python -m gen_harness.cli build-tasks \
  --dataset wise \
  --benchmark-data /path/to/wise.jsonl \
  --output data/wise/tasks.jsonl

python -m gen_harness.cli generate \
  --dataset wise \
  --harness "$EVOGEN_EVAL_HARNESS" \
  --tasks data/wise/tasks.jsonl \
  --output-dir runs/wise \
  --backend-config configs/backends/flux1_dev_diffusers.json \
  --inspection-config configs/inspection/composite_owlv2_nvila_strong.json \
  --policy-extractor-config configs/policy_extractors/openai_chat_policy_extractor.json \
  --harness-localizer-config configs/patch_proposers/openai_chat_harness_localizer_fastloop.json \
  --patch-proposer-config configs/patch_proposers/openai_chat_patch_proposer_fastloop.json

python -m gen_harness.cli export-eval \
  --dataset wise \
  --tasks data/wise/tasks.jsonl \
  --experiences runs/wise/gen_harness_generations.jsonl \
  --output runs/wise/official_input.jsonl

Workflows and documentation

Repository layout

gen_harness/             Core implementation and CLI
configs/                 Generator, verifier, policy, localizer, and proposer configs
examples/visual_harness/ Supplied five-responsibility harness
examples/tasks/          Small task-schema and usage examples
data/evolution/          P2 500/100/100 files, provenance, and checksums
data/geneval2/           Prompt-only benchmark inputs and smoke subsets
scripts/                 Verifier adapters and release helpers
tests/                   Unit and optional integration tests
assets/readme/           Locally bundled figures, badges, and light/dark logos
documentation/           Setup, evolution, evaluation, and release notes
artifacts/               Documented interface for separately archived real runs

The existing GitHub Pages site remains in docs/ when this code is merged into the live repository. This code archive deliberately does not replace that directory.

Citation

@misc{luo2026evogenharness,
  title         = {{EvoGen-Harness}: Learning Where and How to Evolve Image-Generation Harnesses},
  author        = {Jiabin Luo and Yinan Liu and Chunlei Meng and Yufei Guo},
  year          = {2026},
  eprint        = {2610.00383},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.00383}
}

Also available as BibTeX and CITATION.cff.