BenchTrace is a benchmark for evaluating the self-evolution ability of LLM agents. It provides a snapshot–reflection dataset of 1,821 annotated episodes spanning six agentic tasks, paired with two evaluation suites (Reflection Evaluation and Evolution Evaluation).
| Task | Description | Episodes |
|---|---|---|
alfworld | ALFWorld embodied household tasks | 451 |
babyai | BabyAI grid-world navigation | 446 |
bundled_web_shopping | Bundled web shopping | 316 |
scienceworld | ScienceWorld interactive science | 275 |
jericho | Jericho text-adventure games | 242 |
group_travel_planning | Group travel planning | 91 |
| Total | 1,821 |
Each task folder contains an all.json (full split) plus per-category JSON files (e.g. per game, level, or task type).
Each episode in episodes has:
| Field | Description |
|---|---|
id | Unique episode identifier |
game | Environment / game instance |
task_type | Task category within the environment |
model | Agent model that produced the episode |
won | Whether the episode succeeded |
snapshot | Episode snapshot (trajectory context for reflection) |
failure_instances | Annotated failures: detection, localization, and diagnosis labels |
from datasets import load_dataset
ds = load_dataset("huangjh16/BenchTrace", "jericho")
Or load the raw JSON directly:
import json
data = json.load(open("jericho/all.json"))
print(data["n_episodes"], len(data["episodes"]))
BenchTrace episodes are derived from several agentic environments, each under its own license. When using this dataset, please respect the upstream terms and cite the original works:
zork1) are not redistributed in this dataset.Original annotations in BenchTrace are released under CC-BY-4.0.
@inproceedings{benchrogue2026,
title = {BenchTrace: Benchmarking Self-Evolution in LLM Agents},
booktitle = {Proceedings of ACL},
year = {2026}
}
BenchTrace is a benchmark for evaluating the self-evolution ability of LLM agents. It provides a snapshot–reflection dataset of 1,821 annotated episodes spanning six agentic tasks, paired with two evaluation suites (Reflection Evaluation and Evolution Evaluation).
| Task | Description | Episodes |
|---|---|---|
alfworld | ALFWorld embodied household tasks | 451 |
babyai | BabyAI grid-world navigation | 446 |
bundled_web_shopping | Bundled web shopping | 316 |
scienceworld | ScienceWorld interactive science | 275 |
jericho | Jericho text-adventure games | 242 |
group_travel_planning | Group travel planning | 91 |
| Total | 1,821 |
Each task folder contains an all.json (full split) plus per-category JSON files (e.g. per game, level, or task type).
Each episode in episodes has:
| Field | Description |
|---|---|
id | Unique episode identifier |
game | Environment / game instance |
task_type | Task category within the environment |
model | Agent model that produced the episode |
won | Whether the episode succeeded |
snapshot | Episode snapshot (trajectory context for reflection) |
failure_instances | Annotated failures: detection, localization, and diagnosis labels |
from datasets import load_dataset
ds = load_dataset("huangjh16/BenchTrace", "jericho")
Or load the raw JSON directly:
import json
data = json.load(open("jericho/all.json"))
print(data["n_episodes"], len(data["episodes"]))
BenchTrace episodes are derived from several agentic environments, each under its own license. When using this dataset, please respect the upstream terms and cite the original works:
zork1) are not redistributed in this dataset.Original annotations in BenchTrace are released under CC-BY-4.0.
@inproceedings{benchrogue2026,
title = {BenchTrace: Benchmarking Self-Evolution in LLM Agents},
booktitle = {Proceedings of ACL},
year = {2026}
}