BenchTrace Code

This repository contains the code for BenchTrace, a benchmark for evaluating self-evolution ability in LLM agents. BenchTrace consists of a snapshot-reflection dataset of 1,821 annotated episodes spanning six tasks, paired with two evaluation suites: Reflection Evaluation and Evolution Evaluation.

📊 Dataset: huangjh16/BenchTrace on Hugging Face

Repository Structure

benchrogue_code/
├── data_generation/        # Dataset construction pipeline
├── evaluation/
│   ├── reflection/         # Reflection Evaluation
│   └── evolution/          # Evolution Evaluation
└── analysis/               # Analysis and plotting scripts

Note on Jericho game files: Zork and other Infocom game data (ROMs and .qzl saved-game files) are proprietary and are not included in this repository for copyright reasons. To run the Jericho evaluation, obtain the game files separately (see the Jericho project) and place them under evaluation/evolution/jericho/.

Each of the three top-level modules contains a subdirectory per task:

• jericho — Jericho text adventure games
• alfworld — ALFWorld embodied household tasks
• babyai — BabyAI grid-world navigation
• bundled_web_shopping — Bundled Web Shopping
• group_travel_planning — Group Travel Planning
• science_world — ScienceWorld

Requirements

Install dependencies:

pip install -r requirements.txt

An api_key.json file is required in the root directory with the following format:

{"openai": "<your-openai-key>", "anthropic": "<your-anthropic-key>"}

Modules

1. data_generation/

Scripts for constructing the BenchTrace snapshot-reflection dataset.

Usage (example for Jericho):

python data_generation/jericho/ai_annotate.py
python data_generation/jericho/build_dataset.py

2. evaluation/

evaluation/reflection/

Scripts for running and scoring Reflection Evaluation — measuring whether a model can correctly answer detection, localization, and diagnosis questions given an episode snapshot.

Usage:

python evaluation/reflection/jericho/run_reflect_task.py --model Qwen/Qwen3-32B
python evaluation/reflection/jericho/score_reflect_task.py

evaluation/evolution/

Scripts for running and scoring Evolution Evaluation — measuring whether an agent avoids a target failure instance after being exposed to relevant past snapshots.

Baselines included:

Usage:

python evaluation/evolution/jericho/run_evotest.py --game balances --model Qwen/Qwen3-32B
python evaluation/evolution/score_evol_eval.py

3. analysis/

Scripts for reproducing figures and analysis results in the paper.

Usage:

python analysis/plot_cascade_crossenv.py
python analysis/correlation_analysis.py

Citation

@inproceedings{benchrogue2026,
  title     = {BenchTrace: Benchmarking Self-Evolution in LLM Agents},
  booktitle = {Proceedings of ACL},
  year      = {2026}
}