SeerRay-Lab/Xiaomi-GUI-0

[Technical Report] An End-to-End Multimodal GUI Agent for Real Mobile Environments

Python

86

71 commits

updated Sep 18, 2026

See the code

README

Xiaomi-GUI-0

An End-to-End Multimodal GUI Agent for Real Mobile Environments

Trained and evaluated in a real-device closed loop — closing the benchmark-to-reality gap.

Project Page arXiv HuggingFace GitHub

📰 News

✨ What Is Xiaomi-GUI-0?

High benchmark scores do not reliably predict performance on real devices, where account states, permission dialogs, payment authentication, and risk-control mechanisms continually reshape the state distribution a GUI agent encounters. To close this gap, Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop:

  • 📱 Real-device-dominant infrastructure: hundreds of physical phones, tablets, and in-vehicle cockpits, complemented by sandboxes, so collection, training, and evaluation share one real-deployment distribution.
  • 🔁 Error-driven data flywheel: failure trajectories from real rollouts are converted into corrected actions, reflective rationales, and recovery demonstrations.
  • 🎯 Progressive three-stage training: SFT → Step RL → Agentic RL incrementally builds basic interface operation, long-horizon planning, and error recovery.
  • 🧪 RealMobile benchmark: 100 real-device tasks across 14 live apps, scored by fine-grained sub-goals, with 57% spanning multiple applications.

📊 Main Results

Success: fraction of fully completed tasks; Progress: mean fraction of completed sub-goals per task.

ModelRealMobile SuccessRealMobile ProgressAndroidWorld
Gemini 3.1 Pro85.0%89.6%
Gemini 3.1 Flash58.0%72.4%
Claude Opus 4.760.0%74.8%
Seed 2.0 Pro80.0%88.1%
Seed 1.865.0%82.4%70.7%
UI-TARS-273.3%
UI-TARS-1.524.0%40.5%64.2%
UI-Venus-1.5-30B-A3B21.0%44.6%77.6%
GUI-Owl-1.5-32B-Instruct22.0%40.6%69.8%
GUI-Owl-1.5-32B-Thinking31.0%51.7%69.8%
Step-GUI-8B15.0%32.8%67.7%
MAI-UI-8B33.0%50.8%70.7%
Xiaomi-GUI-0-30B-A3B72.0%85.8%78.9%

🧪 The RealMobile Benchmark

RealMobile is built from real user traffic, hand-crafted for reproducible evaluation, and executed on physical devices against live applications rather than emulators. Each task is scored through fine-grained sub-goals that award partial credit, and most tasks span multiple applications.

DomainTasksAvg. AppsMulti-App RatioFocus
Foundation101.3010%Basic GUI operations: clicking, scrolling, inputting, and navigating across interfaces.
Safety & Reflection161.3131%Refusing unsafe or irreversible operations, and recognizing infeasible goals to stop or skip.
Memory & Knowledge331.7358%Retaining information across steps and applying external knowledge to complete tasks.
Complex Reasoning & Planning412.4978%Long-horizon planning, multi-source aggregation, and adaptive decision-making.
Overall1001.9357%
Application frequency Applications per task

🗂️ Repository Layout

The agent is shipped as a product and evaluated by two complementary benchmarks. Each directory has its own README with full setup and usage.

DirectoryWhat it is
guiness/Desktop client that runs the agent on a real phone. PySide6 application with a Kotlin Android companion; connects over WiFi (no ADB) or USB; interactive GUI plus a batch evaluation runner.
android_world_eval/Dynamic emulator benchmark built on AndroidWorld. The agent sees screenshots only (no accessibility tree); 116 tasks across 20 applications, with checkpoint and resume.
realmobile/Real-phone Chinese-application benchmark. Scores recorded trajectories by fine-grained sub-goals, verified with XPath and code rules over the UI hierarchy XML, with PaddleOCR for on-screen text.
demo/Sample trajectories. A few AndroidWorld episodes (task.json plus per-turn screenshots). Data only, no code.

📚 Reference

If you find the resources in this repository helpful, please cite as:

@misc{cao2026xiaomigui0technicalreport,
      title={Xiaomi-GUI-0 Technical Report}, 
      author={Wanxia Cao and Chengzhen Duan and Pei Fu and Pengzhi Gao and Niu Lian and Fazhan Liu and Hui Liu and Heng Qu and Qinzhuo Wu and Zhehao Yu and Tongbo Chen and Shiqi Cui and Anan Du and Shukai Jia and Yuanfa Li and Wei Liu and Yike Liu and Wenchao Lu and Zhenbo Luo and Haoyuan Sun and Jiatong Sun and Cheng Tan and Yajie Wang and Changqiao Wu and Tao Xiong and Jiahui Yang and Yuxuan Yuan and Ruoceng Zhang and Shaojie Zhang and Jian Zhu and Jian Luan and Cong Zou},
      year={2026},
      eprint={2606.31410},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.31410}, 
}

Contributors

gpengzhi

57 commits

yzhyuzhehao

9 commits

qinzhuowu

4 commits

claude

1 commits

SeerRay-Lab/Xiaomi-GUI-0

[Technical Report] An End-to-End Multimodal GUI Agent for Real Mobile Environments

Python

86

71 commits

updated Sep 18, 2026

See the code

README

Xiaomi-GUI-0

An End-to-End Multimodal GUI Agent for Real Mobile Environments

Trained and evaluated in a real-device closed loop — closing the benchmark-to-reality gap.

Project Page arXiv HuggingFace GitHub

📰 News

✨ What Is Xiaomi-GUI-0?

High benchmark scores do not reliably predict performance on real devices, where account states, permission dialogs, payment authentication, and risk-control mechanisms continually reshape the state distribution a GUI agent encounters. To close this gap, Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop:

  • 📱 Real-device-dominant infrastructure: hundreds of physical phones, tablets, and in-vehicle cockpits, complemented by sandboxes, so collection, training, and evaluation share one real-deployment distribution.
  • 🔁 Error-driven data flywheel: failure trajectories from real rollouts are converted into corrected actions, reflective rationales, and recovery demonstrations.
  • 🎯 Progressive three-stage training: SFT → Step RL → Agentic RL incrementally builds basic interface operation, long-horizon planning, and error recovery.
  • 🧪 RealMobile benchmark: 100 real-device tasks across 14 live apps, scored by fine-grained sub-goals, with 57% spanning multiple applications.

📊 Main Results

Success: fraction of fully completed tasks; Progress: mean fraction of completed sub-goals per task.

ModelRealMobile SuccessRealMobile ProgressAndroidWorld
Gemini 3.1 Pro85.0%89.6%
Gemini 3.1 Flash58.0%72.4%
Claude Opus 4.760.0%74.8%
Seed 2.0 Pro80.0%88.1%
Seed 1.865.0%82.4%70.7%
UI-TARS-273.3%
UI-TARS-1.524.0%40.5%64.2%
UI-Venus-1.5-30B-A3B21.0%44.6%77.6%
GUI-Owl-1.5-32B-Instruct22.0%40.6%69.8%
GUI-Owl-1.5-32B-Thinking31.0%51.7%69.8%
Step-GUI-8B15.0%32.8%67.7%
MAI-UI-8B33.0%50.8%70.7%
Xiaomi-GUI-0-30B-A3B72.0%85.8%78.9%

🧪 The RealMobile Benchmark

RealMobile is built from real user traffic, hand-crafted for reproducible evaluation, and executed on physical devices against live applications rather than emulators. Each task is scored through fine-grained sub-goals that award partial credit, and most tasks span multiple applications.

DomainTasksAvg. AppsMulti-App RatioFocus
Foundation101.3010%Basic GUI operations: clicking, scrolling, inputting, and navigating across interfaces.
Safety & Reflection161.3131%Refusing unsafe or irreversible operations, and recognizing infeasible goals to stop or skip.
Memory & Knowledge331.7358%Retaining information across steps and applying external knowledge to complete tasks.
Complex Reasoning & Planning412.4978%Long-horizon planning, multi-source aggregation, and adaptive decision-making.
Overall1001.9357%
Application frequency Applications per task

🗂️ Repository Layout

The agent is shipped as a product and evaluated by two complementary benchmarks. Each directory has its own README with full setup and usage.

DirectoryWhat it is
guiness/Desktop client that runs the agent on a real phone. PySide6 application with a Kotlin Android companion; connects over WiFi (no ADB) or USB; interactive GUI plus a batch evaluation runner.
android_world_eval/Dynamic emulator benchmark built on AndroidWorld. The agent sees screenshots only (no accessibility tree); 116 tasks across 20 applications, with checkpoint and resume.
realmobile/Real-phone Chinese-application benchmark. Scores recorded trajectories by fine-grained sub-goals, verified with XPath and code rules over the UI hierarchy XML, with PaddleOCR for on-screen text.
demo/Sample trajectories. A few AndroidWorld episodes (task.json plus per-turn screenshots). Data only, no code.

📚 Reference

If you find the resources in this repository helpful, please cite as:

@misc{cao2026xiaomigui0technicalreport,
      title={Xiaomi-GUI-0 Technical Report}, 
      author={Wanxia Cao and Chengzhen Duan and Pei Fu and Pengzhi Gao and Niu Lian and Fazhan Liu and Hui Liu and Heng Qu and Qinzhuo Wu and Zhehao Yu and Tongbo Chen and Shiqi Cui and Anan Du and Shukai Jia and Yuanfa Li and Wei Liu and Yike Liu and Wenchao Lu and Zhenbo Luo and Haoyuan Sun and Jiatong Sun and Cheng Tan and Yajie Wang and Changqiao Wu and Tao Xiong and Jiahui Yang and Yuxuan Yuan and Ruoceng Zhang and Shaojie Zhang and Jian Zhu and Jian Luan and Cong Zou},
      year={2026},
      eprint={2606.31410},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.31410}, 
}

Contributors

gpengzhi

57 commits

yzhyuzhehao

9 commits

qinzhuowu

4 commits

claude

1 commits

Languages

Python

61.1%

HTML

17.4%

JavaScript

15.2%

Kotlin

4.8%