ClawBench Dataset

ClawBench is an open benchmark for AI web agents — the systems that drive a real browser to complete a user's task end-to-end. It scores agents on real, everyday online tasks (booking flights, ordering groceries, submitting job applications) across live websites.

|💻 Github | 🏆 Leaderboard | 📖 Paper | 🌐 Website |

🚀 What's New

• [2026.05.12] Added the V2 corpus (130 newer tasks across 63 platforms) and 7 new models judged with deepseek/deepseek-v4-pro — see snapshot below. Companion V2 traces released at TIGER-Lab/ClawBenchV2Trace.
• [2026.05.04] Reorganized to the clawbench-eval package. Single command for both V1 and V2: clawbench run --corpus v2 --model <m> --harness hermes.
• [2026.04.18] Published NAIL-Group/ClawBenchV1Trace — full 5-layer execution traces (recording, actions, HTTP, agent messages, interception) for every V1 run.
• [2026.04.06] Paper preprint up: arXiv:2604.08523 — Can AI Agents Complete Everyday Online Tasks?

🏆 Leaderboard

Live results — pulled from leaderboard/results.csv in this repo. Filter by corpus (v1 / v2 / all) and submit your model in the interactive Space:

V2 snapshot — refreshed 2026-05-12 (full scoring logic: eval/scoring.md)

Intercepted (sort key) = fraction whose final HTTP request matched the per-task URL/method schema — Stage 1, deterministic, no judge. Reward = additionally requires an LLM judge (default deepseek/deepseek-v4-pro) — Stage 2. Rows ranked by Intercepted DESC, Reward as tiebreak. V2 is Hermes-only; alternative harnesses are evaluated separately. Partial = batch attempted < 130 V2 tasks; rates are over attempted, not over 130.. Companion traces in TIGER-Lab/ClawBenchV2Trace. See scoring.md, live leaderboard Space.

Submit a result → run clawbench-eval on your model and open a PR to leaderboard/results.csv — one row per (model × harness × corpus).

Companion datasets (raw traces): NAIL-Group/ClawBenchV1Trace (V1 runs) · TIGER-Lab/ClawBenchV2Trace (V2 runs, rolling) — recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json per model run.

Dataset Structure

Columns

Additional Files

shared/
  alex_green_personal_info.json   # Shared dummy user profile used across all tasks
extra_info/
  004/grocery_list.json           # Task-specific context (32 tasks have extra info)
  007/meal_plan.json
  043/pet_info.json
  ...

• shared/alex_green_personal_info.json — A comprehensive dummy user persona (Alex Green) including personal details, address, work history, education, financial information, and preferences. All tasks share this identity.
• extra_info/ — Task-specific supplementary files referenced by the extra_info column. 32 of 153 tasks include additional context such as grocery lists, job links, meeting details, etc.

eval_schema

The eval_schema field configures the request interceptor — a mechanism that blocks the final HTTP request matching the specified URL pattern and method, preventing irreversible actions (checkout, form submission, etc.) from reaching the server. This allows safe evaluation on live websites.

{
  "url_pattern": "taskrabbit\\.(com|ca)/(api/v\\d+/jobs|book/\\d+/confirm)",
  "method": "POST"
}

Task Categories (metaclass)

Usage

from datasets import load_dataset

ds = load_dataset("TIGER-Lab/ClawBench", split="test")
print(ds[0])

Citation

@article{zhang2026clawbench,
  title={ClawBench: Can AI Agents Complete Everyday Online Tasks?},
  author={Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen},
  journal={arXiv preprint arXiv:2604.08523},
  year={2026}
}