LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects

🚀 News

• 🏆 [2025/12/02] Our survey has been accepted to TMLR 2025! See the latest version: LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects.
• 🥳 [2025/04/29] Our survey has been updated for v2! See LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects for the paper! Welcome to follow our work.
• 🤩 [2025/03/13] We have updated the Paper List again! We will update once a week from now on. We welcome your continued interest in our work!
• 🎉 [2025/02/21] We have updated the Paper List! The new version of our survey will be released later.
• 💥 [2025/01/07] Our survey is released!
• ✨ [2024/12/29] We create this repository to maintain a paper list on LLM-Powered Phone GUI Agents. More papers are coming soon!

📑 Citation

If you find our survey useful for your research and applications, please cite using this BibTeX:

@article{liu2025llm,
  title={Llm-powered gui agents in phone automation: Surveying progress and prospects},
  author={Liu, Guangyi and Zhao, Pengxiang and Liu, Liang and Guo, Yaxuan and Xiao, Han and Lin, Weifeng and Chai, Yuxiang and Han, Yue and Ren, Shuai and Wang, Hao and others},
  journal={arXiv preprint arXiv:2504.19838},
  year={2025}
}

📖 Introduction

🔥 Must-read papers for LLM-Powered Phone GUI Agents.

We greatly appreciate any contributions via PRs, issues, emails, or other methods.

🔖 General Overview

A comprehensive taxonomy of LLM-powered phone GUI agents in phone automation. Note that only a selection of representative works is included in this categorization.

🪧 Milestones

Milestones in the development of LLM-powered phone GUI agents. This figure divides advancements into four primary parts: Prompt Engineering, Training-Based Methods, Datasets and Benchmarks. Prompt Engineering leverages pre-trained LLMs by strategically crafting input prompts, to perform specific tasks without modifying model parameters. In contrast, Training-Based Methods, involve adapting LLMs via supervised fine-tuning or reinforcement learning on GUI-specific data, thereby enhancing their ability to understand and interact with mobile UIs.

📝 Table of Content (ToC)

• LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects
  • 🚀 News
  • 📑 Citation
  • 📖 Introduction
  • 🔖 General Overview
  • 🪧 Milestones
  • 📝 Table of Content (ToC)
  • 🔍 Paper List
    • Frameworks and Components of Phone GUI Agents
      • Single-Agent Frameworks
      • Multi-Agent Frameworks
      • Plan-Then-Act Frameworks
    • LLMs for Phone Automation
      • Prompt Engineering
        • Text-Based Prompt
        • Multimodal Prompt
      • Training-Based Methods
        • Task-Specific LLM-based Agents
        • Supervised Fine-Tuning Methods for Phone GUI Agents
        • Reinforcement Learning Methods for Phone GUI Agents
    • Datasets and Benchmarks
      • Datasets
      • Benchmarks

🔍 Paper List

Frameworks and Components of Phone GUI Agents

Single-Agent Frameworks

Multi-Agent Frameworks

Plan-Then-Act Frameworks

LLMs for Phone Automation

Prompt Engineering

Text-Based Prompt

Multimodal Prompt

Training-Based Methods

Task-Specific LLM-based Agents

Supervised Fine-Tuning Methods for Phone GUI Agents

Reinforcement Learning Methods for Phone GUI Agents

Datasets and Benchmarks

Datasets

Benchmarks

• Ivy Tendril: Open-source GUI software factory and agent harness that handles parallel Git worktrees for you, complete with programmatic verifications and review loops