Long-Horizon Terminal-Bench (LHTB)

LHTB is a 46-task benchmark for measuring how well LLM agents sustain useful work in a containerized terminal over hundreds of steps. Unlike short-horizon coding benchmarks where an agent writes one artifact and stops, LHTB drops the agent into a stateful environment and grades it with hidden, rebuild-from-artifact verifiers — self-reported progress does not count.

• 📝 Blog: https://zli12321.github.io/LHTB/
• 🏆 Leaderboard: https://huggingface.co/datasets/IntelligenceLab/LHTB-leaderboard
• 📄 Paper: https://arxiv.org/abs/2607.08964
• 💻 Code: https://github.com/zli12321/LHTB

⚠️ Harness update — continue-until-timeout

LHTB adds one behavior on top of Harbor: for long-horizon tasks the agent keeps working until the task timeout instead of ending the moment it declares the task complete. After the agent stops, the harness runs the hidden verifier and — if it hasn't fully passed — resumes the same session with the verifier feedback, repeating until the timeout elapses or the verifier passes.

This is controlled by continue_until_timeout = true in the [agent] block of a task's task.toml, set on 30 of the 46 tasks. Stock upstream Harbor ignores this flag, so those tasks run single-shot there and score lower. To reproduce the official LHTB numbers, use the modified Harbor bundled in the GitHub repo (harbor/) — see harbor/README.md for the exact diff and harbor/skills/apply-lhtb-patches/PATCH.md to apply the same patch to any other Harbor version.

What's in this dataset

Each of the 46 tasks ships as a Harbor / Terminal-Bench 2.0 task folder under tasks/:

tasks/<task-id>/
├── task.toml          # metadata, timeouts, resources, continue_until_timeout
├── instruction.md     # agent-facing prompt
└── environment/       # Dockerfile (+ task assets)

Held out on purpose. The tests/ (hidden verifiers) and solution/ (oracle solutions) directories are intentionally not published here to keep the benchmark contamination-resistant. Request access or run the official harness for scoring; see the GitHub repo.

Dataset viewer

The tasks config (data/tasks.jsonl, one row per task) powers the dataset viewer. Columns:

from datasets import load_dataset

ds = load_dataset("IntelligenceLab/Long-Horizon-Terminal-Bench", "tasks", split="test")
print(ds[0]["task_id"], ds[0]["category"])
print(ds[0]["instruction"])

Running the benchmark

Use Harbor with the task folders:

uv tool install harbor          # or: pip install harbor
export DOCKER_DEFAULT_PLATFORM=linux/amd64   # many images are amd64-only
harbor run -c your_config.yaml

See the GitHub README for full instructions.

Citation

@misc{li2026longhorizonterminalbenchtestinglimitsagents,
      title={Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading}, 
      author={Zongxia Li and Zhongzhi Li and Yucheng Shi and Ruhan Wang and Junyao Yang and Zhichao Liu and Xiyang Wu and Anhao Li and Yue Yu and Ninghao Liu and Lichao Sun and Haotao Mi and LeoweiLiang},
      year={2026},
      eprint={2607.08964},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.08964}, 
}

License

Apache License 2.0.