CUAVerifierBench: A Human-Annotated Benchmark for Computer-Using-Agent Verifiers

Universal Verifier paper: The Art of Building Verifiers for Computer Use Agents

Dataset Summary

CUAVerifierBench is an evaluation benchmark for verifiers of computer-using agents (CUAs) — i.e. judges that read an agent's trajectory (screenshots + actions + final answer) and decide whether the task was completed correctly. Where benchmarks like WebTailBench measure agents, CUAVerifierBench measures the judges that score those agents.

Each row pairs a Fara-7B agent trajectory with one human reviewer's verdict, plus the verdicts produced by the Universal Verifier (MMRubricAgent) and several legacy verifiers. Researchers can use the dataset to:

• Compute verifier–human agreement (Cohen's κ, accuracy, F1) on a fixed corpus of trajectories
• Study disagreement between judges and how it changes when reviewers see the verifier's output (the "UV-informed" stage)
• Iterate on new verifier prompts/architectures against a frozen ground-truth set

Splits

Both configs (trajectories, annotations) carry the same two splits:

The two splits share the same column schema. The internal split was annotated in a single UV-blind stage, so its informed_* fields and human_process_score are empty / null.

Dataset Structure

The dataset is exposed as two HuggingFace configs that are joinable on task_id:

Reviewer identities are anonymized as Judge1 … JudgeN using a single shared map across both splits — the same human always gets the same Judge ID.

Storing screenshots only in trajectories (rather than duplicating across judges) cuts the on-disk size roughly in half.

Config: trajectories

About the legacy verifiers. mm_is_success and verifier_is_success come from the original WebTailBench verifier suite used in the Fara-7B tech report (a 3-judge ensemble: text-only task verifier, multimodal grounded verifier, early rubric agent). The entire suite has since been deprecated and replaced by the Universal Verifier (MMRubricAgent) in microsoft/fara. They are included only for backwards-compatible analysis against numbers from the original Fara-7B paper. New work should use uv_rubric_score / uv_outcome_success.

Config: annotations

UV-blind vs. UV-informed. Reviewers labeled each trajectory in two stages: first without seeing any verifier output (human_* and *_comment fields), then after being shown the Universal Verifier's verdict (informed_* fields).

Loading

Each config is loaded separately and joined on task_id. Pass either fara7b_om2w_browserbase or internal as the split:

from datasets import load_dataset

split = "fara7b_om2w_browserbase"  # or "internal"
trajs = load_dataset("microsoft/CUAVerifierBench", "trajectories", split=split)
anns  = load_dataset("microsoft/CUAVerifierBench", "annotations",  split=split)

# Per-judge analysis: join in pandas
import pandas as pd
df = anns.to_pandas().merge(trajs.to_pandas(), on="task_id")

# Or look up trajectories on demand:
by_id = {r["task_id"]: r for r in trajs}
print(by_id[anns[0]["task_id"]]["screenshots"][0])  # PIL.Image

Dataset Creation

Source trajectories

Trajectories were generated by running Fara-7B on the public Online-Mind2Web task set, executed inside a Browserbase-hosted Chromium instance. Each trajectory contains the screenshots the model saw, the structured actions it issued, and the final answer it submitted.

Annotation protocol

Each task was independently reviewed by ~2 human annotators in two stages:

1. UV-blind (agnostic) — Reviewers read the instruction and trajectory and assign outcome / process labels and a continuous process score, without seeing any verifier output.
2. UV-informed — Reviewers are then shown the Universal Verifier's verdict and asked whether they agree, with free-text justifications.

Reviewer identities are anonymized as Judge1…Judge6.

Universal Verifier outputs

For each trajectory we also include the verdicts of the Universal Verifier (code is released at https://github.com/microsoft/fara) and two legacy verifiers, so users can directly compute verifier–human agreement.

Considerations for Using the Data

Intended Use

• Evaluating CUA verifiers against human judgment
• Studying inter-annotator agreement and the effect of showing model verdicts to humans
• Developing new judge prompts / architectures for trajectory evaluation

Limitations

• 106 tasks is a relatively small corpus; results should be reported with confidence intervals
• All trajectories come from a single agent (Fara-7B); verifier behavior on trajectories from other agents may differ
• Tasks inherit the temporal validity and domain biases of Online-Mind2Web

Licensing

MIT License

Citation

If you use CUAVerifierBench in your research, please cite the Universal Verifier paper:

@article{UniversalVerifier2026,
  title={The Art of Building Verifiers for Computer Use Agents},
  journal={arXiv preprint arXiv:2604.06240},
  year={2026},
  url={https://arxiv.org/abs/2604.06240v1}
}

Contributions

Created by Microsoft Research AI Frontiers.