README
CAGUI: Chinese Android GUI Benchmark
A real-world Chinese Android GUI benchmark designed to evaluate GUI agent models on two complementary capabilities:
- Grounding โ understanding individual GUI components and linking them to semantics.
- Agent โ planning and executing multi-step actions to complete user goals on Chinese Android apps.
See AgentCPM-GUI for more details.
๐ Key Features
| Aspect | Grounding | Agent |
|---|---|---|
| Objective | GUI widgets grounding / OCR text | Follow natural-language instructions to operate an app |
| Data | 2 ร JSONL files (cap.jsonl, ocr.jsonl) + screenshots | Per-episode JSON + step-level screenshots |
| Actions | n/a | Tap, long-tap, text-input, scroll, etc. (result_action_type, see here) |
๐๏ธ Repository Structure
CAGUI
โโ CAGUI_agent
โ โโ domestic/
โ โโ <episode_id>/
โ โโ <episode_id>.json # episode file
โ โโ <episode_id>_0.jpeg # step-0 screenshot
โ โโ <episode_id>_1.jpeg
โ โโ ...
โโ CAGUI_grounding
โโ code/
โ โโ cap.jsonl # function to point & text to point
โ โโ ocr.jsonl # bbox to text
โโ images/
โโ cap/
โ โโ 0.jpeg
โ โโ ...
โโ ocr/
โโ 0.jpeg
โโ ...
๐ Data Format
1. Agent episodes (CAGUI_agent/domestic/<episode_id>/<episode_id>.json)
Each file is a list of steps:
| Field | Type | Description |
|---|---|---|
episode_id | str | Unique id |
episode_length | int | Total steps |
step_id | int | Step index (0-based) |
instruction | str | user goal |
image_path | str | Relative path to screenshot |
image_width / image_height | int | Raw resolution |
ui_positions | str (JSON list) | Normalised [[y, x, h, w], โฆ] UI element boxes |
result_action_type | int | Action code, see here |
result_action_text | str | Text typed (if any) |
result_touch_yx / result_lift_yx | str | Normalised touch coords, [-1,-1] if no touch |
duration | float | null | Action time (s) |
2. Grounding annotations (cap.jsonl, ocr.jsonl)
One JSON object per line:
| Field | Example | Description |
|---|---|---|
task | "bbox2function" / "bbox2text" | Sub-task type |
image | "grounding_eval/dataset/images/0.jpeg" | Screenshot path |
id | 0 | Unique int id |
abs_position | "<x1, y1, x2, y2>" | Pixel-level bbox |
rel_position | "<x1, y1, x2, y2>" | Normalised bbox |
text | "UIๅ
็ด ๆฏไธไธช่ๅๆ้ฎโฆ" | Target description / OCR string |
๐ License
CAGUI is released under the CC-BY-NC 4.0 license for non-commercial research. Screenshots originate from publicly available Chinese apps and are used under fair-use for research purposes only. Remove them if local regulations require.
โ๏ธ Citation
This benchmark is used for evaluating AgentCPM-GUI. If our model and benchmark are useful for your research, please cite:
@article{zhang2025agentcpmgui,
title={Agent{CPM}-{GUI}: Building Mobile-Use Agents with Reinforcement Fine-Tuning},
author={Zhong Zhang and Yaxi Lu and Yikun Fu and Yupeng Huo and Shenzhi Yang and Yesai Wu and Han Si and Xin Cong and Haotian Chen and Yankai Lin and Jie Xie and Wei Zhou and Wang Xu and Yuanheng Zhang and Zhou Su and Zhongwu Zhai and Xiaoming Liu and Yudong Mei and Jianming Xu and Hongyan Tian and Chongyi Wang and Chi Chen and Yuan Yao and Zhiyuan Liu and Maosong Sun},
year={2025},
journal={arXiv preprint arXiv:2506.01391},
}
chinese app
gui agent
gui grounding
Contributors
zhong-zhang
9 commits
Similar repos
README
CAGUI: Chinese Android GUI Benchmark
A real-world Chinese Android GUI benchmark designed to evaluate GUI agent models on two complementary capabilities:
- Grounding โ understanding individual GUI components and linking them to semantics.
- Agent โ planning and executing multi-step actions to complete user goals on Chinese Android apps.
See AgentCPM-GUI for more details.
๐ Key Features
| Aspect | Grounding | Agent |
|---|---|---|
| Objective | GUI widgets grounding / OCR text | Follow natural-language instructions to operate an app |
| Data | 2 ร JSONL files (cap.jsonl, ocr.jsonl) + screenshots | Per-episode JSON + step-level screenshots |
| Actions | n/a | Tap, long-tap, text-input, scroll, etc. (result_action_type, see here) |
๐๏ธ Repository Structure
CAGUI
โโ CAGUI_agent
โ โโ domestic/
โ โโ <episode_id>/
โ โโ <episode_id>.json # episode file
โ โโ <episode_id>_0.jpeg # step-0 screenshot
โ โโ <episode_id>_1.jpeg
โ โโ ...
โโ CAGUI_grounding
โโ code/
โ โโ cap.jsonl # function to point & text to point
โ โโ ocr.jsonl # bbox to text
โโ images/
โโ cap/
โ โโ 0.jpeg
โ โโ ...
โโ ocr/
โโ 0.jpeg
โโ ...
๐ Data Format
1. Agent episodes (CAGUI_agent/domestic/<episode_id>/<episode_id>.json)
Each file is a list of steps:
| Field | Type | Description |
|---|---|---|
episode_id | str | Unique id |
episode_length | int | Total steps |
step_id | int | Step index (0-based) |
instruction | str | user goal |
image_path | str | Relative path to screenshot |
image_width / image_height | int | Raw resolution |
ui_positions | str (JSON list) | Normalised [[y, x, h, w], โฆ] UI element boxes |
result_action_type | int | Action code, see here |
result_action_text | str | Text typed (if any) |
result_touch_yx / result_lift_yx | str | Normalised touch coords, [-1,-1] if no touch |
duration | float | null | Action time (s) |
2. Grounding annotations (cap.jsonl, ocr.jsonl)
One JSON object per line:
| Field | Example | Description |
|---|---|---|
task | "bbox2function" / "bbox2text" | Sub-task type |
image | "grounding_eval/dataset/images/0.jpeg" | Screenshot path |
id | 0 | Unique int id |
abs_position | "<x1, y1, x2, y2>" | Pixel-level bbox |
rel_position | "<x1, y1, x2, y2>" | Normalised bbox |
text | "UIๅ
็ด ๆฏไธไธช่ๅๆ้ฎโฆ" | Target description / OCR string |
๐ License
CAGUI is released under the CC-BY-NC 4.0 license for non-commercial research. Screenshots originate from publicly available Chinese apps and are used under fair-use for research purposes only. Remove them if local regulations require.
โ๏ธ Citation
This benchmark is used for evaluating AgentCPM-GUI. If our model and benchmark are useful for your research, please cite:
@article{zhang2025agentcpmgui,
title={Agent{CPM}-{GUI}: Building Mobile-Use Agents with Reinforcement Fine-Tuning},
author={Zhong Zhang and Yaxi Lu and Yikun Fu and Yupeng Huo and Shenzhi Yang and Yesai Wu and Han Si and Xin Cong and Haotian Chen and Yankai Lin and Jie Xie and Wei Zhou and Wang Xu and Yuanheng Zhang and Zhou Su and Zhongwu Zhai and Xiaoming Liu and Yudong Mei and Jianming Xu and Hongyan Tian and Chongyi Wang and Chi Chen and Yuan Yao and Zhiyuan Liu and Maosong Sun},
year={2025},
journal={arXiv preprint arXiv:2506.01391},
}
chinese app
gui agent
gui grounding
Similar repos
Linked in READMEs
Contributors
zhong-zhang
9 commits