Awesome UI Agent

This is a collection of research papers for UI Agent, which includes models, tools, and datasets. And the repository will be continuously updated to track the frontier of UI Agent or related fields.

Welcome to follow and star!

Table of Contents

• Awesome UI Agent
  • Table of Contents
  • Overview of UI Agent
  • Papers
    • Models
      • 2026
      • 2025
      • 2024
      • 2023
    • Tools
    • Datasets
  • Related Repositories
  • Contributing
  • License

Overview of UI Agent

UI Agent aims to build a generalist agent that can interact with various user interfaces (UIs) in different environments, such as mobile apps, web pages, and PC applications. The agent can understand the UIs through vision-language models and interact with them to complete tasks. The agent can be applied to various scenarios, such as mobile device operation, web browsing, and game playing. The agent can be trained in a simulated environment or with real-world data. The agent can be evaluated in terms of task completion rate, efficiency, and generalization ability.

The research on UI Agent is still in its early stage, and there are many challenges to be addressed, such as the scalability of the agent, the robustness of the agent, and the interpretability of the agent. The research on UI Agent is interdisciplinary, involving computer vision, natural language processing, reinforcement learning, human-computer interaction, and software engineering. The research on UI Agent has the potential to revolutionize the way we interact with computers and improve the efficiency and usability of computer systems.

Papers

format:
- [title](paper link) [links]
    - author1, author2, and author3...
    - year
    - publisher
    - key 
    - code 
    - experiment environment

Models

2026

• PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

  • Nicolas Bougie, Xiaotong Ye, Gian Maria Marconi, Narimasa Watanabe
  • Key: persona-conditioned UI/UX evaluation, contrastive reflection fine-tuning, reflective prompt evolution
  • ExpEnv: WiserUI-Bench, UIClip/BetterApp, WebDevJudge, LabintheWild, LabintheWild-UX, UICrit, UXCar

• UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning

  • Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Xu Zhang, et al.
  • Key: multi-perspective instruction reasoning, adaptive instruction selection, GUI grounding
  • ExpEnv: ScreenSpot-Pro, UI-I2E-Bench, MMBench-GUI L2, AndroidWorld, OS-World

• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration

  • Gaole Dai, Shiqi Jiang, Ting Cao, Yuqing Yang, Yuanchun Li, Rui Tan, Mo Li, Lili Qiu
  • Key: proactive reward design, reasoner-actor collaboration, robust GUI-agent optimization
  • ExpEnv: AndroidWorld, AndroidLab, MobileAgentBench, OSWorld, OSWorld-Chrome

• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning

  • Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, Mengwei Xu
  • Key: self-supervised reinforcement learning, K-step GUI transition, label-efficient GUI automation
  • ExpEnv: AndroidControl, GUI Odyssey, AndroidWorld, ScreenSpot-v2, ScreenSpot-Pro

• Go-Browse: Training Web Agents with Structured Exploration

  • Apurva Gandhi, Graham Neubig
  • Key: structured exploration, graph-based website traversal, web-agent data construction
  • ExpEnv: WebArena, VisualWebArena, Online-Mind2Web

• WALT: Web Agents that Learn Tools

  • Viraj Prabhu, Yutong Dai, Matthew Fernandez, Krithika Ramakrishnan, Jing Gu, Yanqi Luo, et al.
  • Key: tool discovery from websites, tool-augmented web agents, long-horizon web interaction
  • ExpEnv: VisualWebArena, WebArena, Online-Mind2Web

• Safe and Scalable Web Agent Learning via Recreated Websites

  • Hyungjoo Chae, Jungsoo Park, Alan Ritter
  • Key: recreated websites, verifiable synthetic environments, safe web-agent learning, self-evolving training
  • ExpEnv: web agent benchmarks on executable recreated websites with deterministic programmatic rewards

• Continual GUI Agents

  • Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng
  • Key: continual GUI agents, reinforcement fine-tuning, anchoring rewards, domain and resolution shift
  • ExpEnv: continual GUI-agent evaluations under shifting domains and resolutions

2025

• CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs

  • Gucongcong Fan, Chaoyue Niu, Chengfei Lv, Fan Wu, Guihai Chen
  • Key: Mobile Agents, Large Language Models, UI Exposure Reduction, Cloud-local Collaboration
  • ExpEnv: Mobile Agents

• UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

  • Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Aojun Zhou, Hongsheng Li
  • Key: Multi-modal Large Language Models, GUI Agents, Self-Improving Framework, Mobile Agents
  • ExpEnv: Mobile GUI environments / Mobile Agents

• BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent

  • Shaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang, Jiahui Yang, Xin Du, ShiqiCui, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan
  • Key: Large Vision-Language Models, GUI Agents, Reinforcement Fine-Tuning, Human-GUI Interaction, Cognitive-inspired Reasoning Framework
  • ExpEnv: GUI environments / Mobile/Desktop GUI Agents

• Programming with Pixels: Towards Generalist Software Engineering Agents

  • Pranjal Aggarwal, Sean Welleck
  • Key: UI-agent, software engineering, visual perception, IDE interaction, Reinforcement Learning.
  • ExpEnv: PwP (Programming with Pixels) environment, PwP-Bench-Lite (benchmark)

• E-commerce UI/UX Optimization via Generative AI, MDD, and Multi-Agent Reinforcement Learning

  • Authors not provided in abstract
  • Key: multi-agent reinforcement learning, UI/UX optimization, Generative AI, E-commerce.
  • ExpEnv: E-commerce applications

• CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks

  • Songqin Nong, Jingxuan Xu, Sheng Zhou, Jianfeng Chen, Xiaoxuan Tang, Tao Jiang, Wenhao Xu
  • Key: Reinforcement Learning, GUI agents, curriculum learning, policy optimization.
  • ExpEnv: Android Control, internal online benchmarks

• CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

  • Zeyi Sun, Yuhang Cao, Jianze Liang, Qiushi Sun, Ziyu Liu, Zhixiong Zhang, Yuhang Zang, Xiaoyi Dong, Kai Chen, Dahua Lin, Jiaqi Wang
  • Key: decoupled Reinforcement Learning, GUI agents, compositional framework, long-horizon planning.
  • ExpEnv: ScienceBoard, AgentRewardBench

• Enhance Mobile Agents Thinking Process Via Iterative Preference Learning

  • Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, Bo An
  • Key: mobile GUI agents, iterative preference learning, DPO, CoaT.
  • ExpEnv: AITZ, AMEX, AndroidControl

• UI-Venus Technical Report: Building High-performance UI Agents with RFT

  • Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, Weiqiang Wang
  • Key: reinforcement finetuning (RFT), UI agents, multimodal large language models, UI navigation.
  • ExpEnv: Screenspot-V2, Screenspot-Pro, AndroidWorld

• Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization

  • Zhitao He, Zijun Liu, Peng Li, Yi R. Fung, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
  • Key: multi-agent RL, LLMs, UI agents, credit re-assignment, environment generalization.
  • ExpEnv: AndroidWorld, MobileMiniWoB++, Mind2Web, AutoWebBench

• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

  • Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, Jiaqi Wang
  • Key: self-evolving agents, computer use agents, reinforcement learning, reward modeling.
  • ExpEnv: OS-World, AgentRewardBench.

• Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery for Foundation Model Internet Agents

  • Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, Erran Li
  • Key: autonomous skill discovery, foundation model agents, web navigation, Reinforcement Learning.
  • ExpEnv: WebVoyager, WebArena

• VSC-RL: Advancing Autonomous Vision-Language Agents with Variational Subgoal-Conditioned Reinforcement Learning

  • Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao
  • Key: framework, reinforcement learning, subgoal generation, VSC-RL, learning efficiency
  • ExpEnv: Android in the Wild

• AppVLM: A Lightweight Vision Language Model for Online App Control

  • Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, Kun Shao
  • Key: vision-language model, multi-modal, AppVLM, on-device control
  • ExpEnv: two open-source mobile control datasets

• DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents

  • Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao
  • Key: framework, reinforcement learning, distributed training, A-RIDE, on-device control
  • ExpEnv: Android in the Wild
  • code

• OpenAI operator

  • OpenAI
  • Key: A research preview of an agent that can use its own browser to perform tasks for you.
  • ExpEnv: OSWorld, WebArena, WebVoyager

• OpenAI Computer-Using Agent

  • OpenAI
  • Key: a universal interface for AI to interact with the digital world.
  • ExpEnv: OSWorld, WebArena, WebVoyager

• Claude computer use

  • anthropic
  • Key: emulating the way people interact with their own computer.
  • ExpEnv: OSWorld

• Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage

  • Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li
  • Key: Multimodal Agents, Vision-language Model, Tool usage
  • ExpEnv: GTA, GAIA benchmarks

• Lightweight Neural App Control

  • Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye HAO, Jun Wang, Kun Shao
  • Key: vision-language model, multi-modal, android control, app agent
  • ExpEnv: two open-source mobile control datasets

• Enhancing Software Agents with Monte Carlo Tree Search and Hindsight Feedback

  • Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, William Yang Wang
  • Key: agents, LLM, SWE-agents, SWE-bench, search, planning, reasoning, self-improvement, open-ended
  • ExpEnv: SWE-bench

2024

• On the Effects of Data Scale on UI Control Agents

  • Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, Oriana Riva
  • Key: Autonomous agents, UI control, AndroidControl dataset, fine-tuning, in-domain vs out-of-domain performance
  • 2024
  • code

• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

  • John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press
  • Key: Language model agents, agent-computer interface (ACI), automated software engineering, SWE-bench, HumanEvalFix
  • 2024
  • code

• Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

  • Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
  • Key: Large language models, embodied decision making, generalized interface, fine-grained metrics, subgoal decomposition, action sequencing
  • 2024
  • code

• Cradle: Empowering Foundation Agents Towards General Computer Control

  • Weihao Tan and Wentao Zhang and Xinrun Xu and Haochong Xia and et al.
  • Key: various virtual scenarios, General Computer Control
  • 2024
  • code

• Lightweight Neural App Control

  • Filippos Christianos and Georgios Papoudakis and Thomas Coste and Jianye Hao and Jun Wang and Kun Shao
  • KEY: app agents, Android apps, Action Transformer
  • 2024

• SeeAct GPT-4V(ision) is a Generalist Web Agent, if Grounded

  • Boyuan Zheng and Boyu Gou and Jihyung Kil and Huan Sun amd Yu Su
  • Key: live websites, grounding still, mage captioning, visual question answering
  • 2024
  • code

• MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot

  • Zirui Song and Yaohang Li and Meng Fang and Zhenhao Chen and Zecheng Shi and Yuan Huang and Ling Chen
  • Key: Autonomous virtual agents, Multi-Modal Agent Collaboration
  • 2024

• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

  • Kanzhi Cheng and Qiushi Sun and Yougang Chu and Fangzhi Xu and Yantao Li and Jianbing Zhang and Zhiyong Wu
  • Key: Graphical User Interface, screenshots
  • 2024 -code

• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

  • Zhiyong Wu and Zhenyu Wu and Fangzhi Xu and Yian Wang and Qiushi Sun and Chengyou Jia and Kanzhi Cheng and Zichen Ding and Liheng Chen and Paul Pu Liang and Yu Qiao
  • Key: Out-Of-Distribution, GUI grounding, language agent
  • 2024 -code

• Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

  • Yifei Zhou and Qianlan Yang and Kaixiang Lin and Min Bai and Xiong Zhou and Yu-Xiong Wang and Sergey Levine and Erran Li
  • Key: Large language models, Internet-browsing agent, autonomous task proposal
  • 2024
  • code

• Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent

  • Hanyu Lai and Xiao Liu and Iat Long Iong and Shuntian Yao and Yuxuan Chen and Pengbo Shen and Hao Yu and Hanchen Zhang and Xiaohan Zhang and Yuxiao Dong and Jie Tang
  • Key: Large language models, real-world web navigation, bilingual benchmark
  • 2024
  • code

• Dual-view visual contextualization for web navigation

  • Jihyung Kil and Chan Hee Song and Boyuan Zheng and Xiang Deng and Yu Su and Wei-Lun Chao
  • Key: Automatic web navigation, language instructions, HTML elements
  • 2024

• Agent-e: From autonomous web navigation to foundational design principles in agentic systems

  • Hanyu Lai and Xiao Liu and Iat Long Iong and Shuntian Yao and Yuxuan Chen and Pengbo Shen and Hao Yu and Hanchen Zhang and Xiaohan Zhang and Yuxiao Dong and Jie Tang
  • Key: hierarchical architecture, flexible DOM distillation, denoising method
  • 2024
  • code

• Tree search for language model agents

  • Jing Yu Koh and Stephen McAleer and Daniel Fried and Ruslan Salakhutdinov
  • Key: multi-step reasoning, planning, best-first tree search
  • 2024
  • code

• Agent S: an open agentic framework that uses computers like a human

  • Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang
  • Key: Multimodal Large Language Models, Graphical User Interface, Agent-Computer Interface
  • 2024
  • code

• Apple Intelligence Foundation Language Models

  • Apple
  • Key: Vision-Language Model, Private Cloud Compute
  • 2024

• CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

  • Xinbei Ma and Zhuosheng Zhang and Hai Zhao
  • Key: Vision-Language Model, Phone
  • 2024
  • code

• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

  • Kanzhi Cheng and Qiushi Sun and Yougang Chu and Fangzhi Xu and Yantao Li and Jianbing Zhang and Zhiyong Wu
  • Key: Vision-Language Model, PC
  • 2024
  • code

• Intention-inInteraction (IN3): Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents

  • Cheng Qian and Bingxiang He and Zhong Zhuang and Jia Deng and Yujia Qin and Xin Cong and Zhong Zhang and Jie Zhou and Yankai Lin and Zhiyuan Liu and Maosong Sun
  • Key: Language Model, User Intention
  • 2024
  • code

• LATS: Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models

  • Andy Zhou and Kai Yan and Michal Shlapentokh-Rothman and Haohan Wang and Yu-Xiong Wang
  • Key: Tree Search, Language Model
  • 2024
  • code

• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

  • Hao Bai and Yifei Zhou and Mert Cemri and Jiayi Pan and Alane Suhr and Sergey Levine and Aviral Kumar
  • Key: Vision-Language Model, Android, Reinforcement Learning
  • 2024
  • code

• ScreenAI: A Vision-Language Model for UI and Infographics Understanding

  • Gilles Baechler and Srinivas Sunkara and Maria Wang and Fedir Zubach and Hassan Mansoor and Vincent Etter and Victor Cărbune and Jason Lin and Jindong Chen and Abhanshu Sharma
  • Key: Vision-Language Model, Mobile, Infographics
  • 2024 -code

• ScreenAgent: A Vision Language Model-driven Computer Control Agent

  • Runliang Niu and Jindong Li and Shiqi Wang and Yali Fu and Xiyu Hu and Xueyuan Leng and He Kong and Yi Chang and Qi Wang
  • Key: Vision Language Model, PC
  • 2024
  • code

• Android in the Zoo: Chain-of-Action-Thought for GUI Agents

  • Jiwen Zhang and Jihao Wu and Yihua Teng and Minghui Liao and Nuo Xu and Xiao Xiao and Zhongyu Wei and Duyu Tang
  • Key: Vision-Language Model, Android, Chain-of-Action-Thought
  • 2024
  • code

• Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

  • Junyang Wang and Haiyang Xu and Haitao Jia and Xi Zhang and Ming Yan and Weizhou Shen and Ji Zhang and Fei Huang and Jitao Sang
  • Key: Vision-Language Model, Android
  • 2024
  • code

• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

  • Junyang Wang and Haiyang Xu and Jiabo Ye and Ming Yan and Weizhou Shen and Ji Zhang and Fei Huang and Jitao Sang
  • Key: Vision-Language Model, Android
  • 2024
  • code

• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

  • Hongliang He and Wenlin Yao and Kaixin Ma and Wenhao Yu and Yong Dai and Hongming Zhang and Zhenzhong Lan and Dong Yu
  • Key: Vision-Language Model, Web
  • 2024
  • code

• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

  • Zhiyong Wu and Chengcheng Han and Zichen Ding and Zhenmin Weng and Zhoumianze Liu and Shunyu Yao and Tao Yu and Lingpeng Kong
  • Key: Vision-Language Model, PC
  • 2024
  • code

• UFO: A UI-Focused Agent for Windows OS Interaction

  • Chaoyun Zhang and Liqun Li and Shilin He and Xu Zhang and Bo Qiao and Si Qin and Minghua Ma and Yu Kang and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang and Qi Zhang
  • Key: Vision-Language Model, PC, Windows OS
  • 2024
  • code

• Octopus v2: On-device language model for super agent

  • Wei Chen and Zhiyuan Li
  • Key: Vision-Language Model, Android, IOS
  • 2024

2023

• Openagents: An open platform for language agents in the wild

  • Kaixin Ma and Hongming Zhang and Hongwei Wang and Xiaoman Pan and Wenhao Yu and Dong Yu
  • wild of everyday life, Language agents, real-world evaluations
  • 2023 -code

• LASER: LLM Agent with State-Space Exploration for Web Navigation

  • Kaixin Ma and Hongming Zhang and Hongwei Wang and Xiaoman Pan and Wenhao Yu and Dong Yu
  • Large language models, web navigation, interactive task
  • 2023 -code

• AppAgent: Multimodal Agents as Smartphone Users

  • Chi Zhang and Zhao Yang and Jiaxuan Liu and Yucheng Han and Xin Chen and Zebiao Huang and Bin Fu and Gang Yu
  • Key: Vision-Language Model, Android
  • 2023
  • code

• CogAgent: A Visual Language Model for GUI Agents

  • Wenyi Hong and Weihan Wang and Qingsong Lv and Jiazheng Xu and Wenmeng Yu and Junhui Ji and Yan Wang and Zihan Wang and Yuxuan Zhang and Juanzi Li and Bin Xu and Yuxiao Dong and Ming Ding and Jie Tang
  • Key: Vision-Language Model, PC, Android, screenshots
  • 2023
  • code

• Octopus: Embodied Vision-Language Programmer from Environmental Feedback

  • Jingkang Yang and Yuhao Dong and Shuai Liu and Bo Li and Ziyue Wang and Chencheng Jiang and Haoran Tan and Jiamu Kang and Yuanhan Zhang and Kaiyang Zhou and Ziwei Liu
  • Key: Vision-Language Model, Android, IOS
  • 2023
  • code

• You Only Look at Screens: Multimodal Chain-of-Action Agents

  • Zhuosheng Zhang and Aston Zhang
  • Key: Vision-Language Model, Android, Chain-of-Action-Thought
  • 2023
  • code

• LASER: LLM Agent with State-Space Exploration for Web Navigation

  • Kaixin Ma and Hongming Zhang and Hongwei Wang and Xiaoman Pan and Wenhao Yu and Dong Yu
  • Key: Vision-Language Model, Web, State-Space Exploration
  • 2023
  • code

• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

  • Izzeddin Gur and Hiroki Furuta and Austin Huang and Mustafa Safdari and Yutaka Matsuo and Douglas Eck and Aleksandra Faust
  • Key: Vision-Language Model, Web, Planning, Program Synthesis
  • 2023

• Augmenting Autotelic Agents with Large Language Models

  • Cédric Colas and Laetitia Teodorescu and Pierre-Yves Oudeyer and Xingdi Yuan and Marc-Alexandre Côté
  • Key: Language Model
  • 2023

• Language Models can Solve Computer Tasks

  • Geunwoo Kim and Pierre Baldi and Stephen McAleer
  • Key: Language Model
  • 2023
  • code

Tools

• Opera Browser Operator: AI-based Agentic Browsing

  • Opera Software
  • Key: AI agent, agentic browsing, native client-side solution, privacy-focused
  • 2025
  • code

• OmniParser: Screen Parsing tool for Pure Vision Based GUI Agent

  • Yadong Lu and Jianwei Yang and Yelong Shen and Ahmed Awadallah
  • Key: UI parsing, vision-based agent, GPT-4V, structured elements
  • 2025
  • code

-Make Websites Accessible for Agents - Li Zhang and Shihe Wang and Xianqing Jia and Zhihan Zheng and Yunhe Yan and Longxi Gao and Yuanchun Li and Mengwei Xu - Key: websites, Agents - 2024 - code

• ToolGen: Unified Tool Retrieval and Calling via Generation

  • Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, Haonan Li
  • Key: Agent, Tool Learning, Virtual Token
  • 2024
  • code

• LEGENT: An Open Platform for Embodied Agentb Agents on Large Language Models

  • Iat Long Iong and Xiao Liu and Yuxuan Chen and Hanyu Lai and Shuntian Yao and Pengbo Shen and Hao Yu and Yuxiao Dong and Jie Tang
  • Key: Webpage, deployment
  • 2024
  • code

• LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Automation Task Evaluation

  • Li Zhang and Shihe Wang and Xianqing Jia and Zhihan Zheng and Yunhe Yan and Longxi Gao and Yuanchun Li and Mengwei Xu
  • Key: Mobile UI, Simulator
  • 2024
  • code

• WebArena: A Realistic Web Environment for Building Autonomous Agents

  • Shuyan Zhou and Frank F. Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig
  • Key: Web, Simulator
  • 2023
  • code

• Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

  • Danyang Zhang and Zhennan Shen and Rui Xie and Situo Zhang and Tianbao Xie and Zihan Zhao and Siyuan Chen and Lu Chen and Hongshen Xu and Ruisheng Cao and Kai Yu
  • Key: Android, Simulator
  • 2023
  • code

• AndroidEnv: A Reinforcement Learning Platform for Android

  • Daniel Toyama and Philippe Hamel and Anita Gergely and Gheorghe Comanici and Amelia Glaese and Zafarali Ahmed and Tyler Jackson and Shibl Mourad and Doina Precup
  • Key: Android, Reinforcement Learning, Simulator
  • 2021
  • code

• TWZRD Agent Intel - Trust scoring MCP for AI agents on Solana. Verify agent wallet identity before x402 micropayments. Free: {"mcpServers":{"twzrd-agent-intel":{"url":"https://intel.twzrd.xyz/mcp"}}}

Datasets

• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

  • Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, Kun Shao
  • Key: Two Languages, Interactive Environment, Plug-and-play Framework, 11 Agents, Diverse Metrics
  • 2025
  • code

• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks

  • Ori Yoran and Samuel Joseph Amouyal and Chaitanya Malaviya and Ben Bogin and Ofir Press and Jonathan Berant
  • Key: Web, Realistic, Time-Consuming, Benchmark
  • 2024
  • code

• WebCanvas: Benchmarking Web Agents in Online Environments

  • Yichen Pan1 and Dehan Kong and Sida Zhou and Cheng Cui and Yifei Leng and Bing Jiang and Hangyu Liu and Yanyi Shang and Shuyan Zhou and Tongshuang Wu and Zhengyang Wu
  • Key: Web, Online Environments, Benchmark
  • 2024
  • code

• MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents

  • Luyuan Wang and Yongyu Deng and Yiwei Zha and Guodong Mao and Qinmin Wang and Tianchen Min and Wei Chen and Shoufa Chen
  • Key: Mobile, Benchmark
  • 2024
  • code

• VillagerBench/VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraft

  • Yubo Dong and Xukun Zhu and Zhengzhe Pan and Linchao Zhu and Yi Yang
  • Key: Vision-Language Model, Game
  • 2024
  • code

• CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions

  • Guo, Zishan and Huang, Yufei and Xiong, Deyi
  • Key: Vision-Language Model, Phone
  • 2024
  • code

• Multi-Turn Mind2Web: On the Multi-turn Instruction Following for Conversational Web Agents

  • Yang Deng and Xuan Zhang and Wenxuan Zhang and Yifei Yuan and See-Kiong Ng and Tat-Seng Chua
  • Key: Vision-Language Model, Web Tasks
  • 2024
  • code

• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

  • Jing Yu Koh and Robert Lo and Lawrence Jang and Vikram Duvvur and Ming Chong Lim and Po-Yu Huang and Graham Neubig and Shuyan Zhou and Ruslan Salakhutdinov and Daniel Fried
  • Key: Vision-Language Model, Web Tasks
  • 2024
  • code

• Android in the Zoo: Chain-of-Action-Thought for GUI Agents

  • Jiwen Zhang and Jihao Wu and Yihua Teng and Minghui Liao and Nuo Xu and Xiao Xiao and Zhongyu Wei and Duyu Tang
  • Key: Vision-Language Model, Android, Chain-of-Action-Thought
  • 2024
  • code

• Android in the Wild: A Large-Scale Dataset for Android Device Control

  • Christopher Rawles and Alice Li and Daniel Rodriguez and Oriana Riva and Timothy Lillicrap
  • Key: Android, datasets
  • 2023
  • code

• Mind2Web: Towards a Generalist Agent for the Web

  • Xiang Deng and Yu Gu and Boyuan Zheng and Shijie Chen and Samuel Stevens and Boshi Wang and Huan Sun and Yu Su
  • Key: Web, datasets
  • 2023
  • code

• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

  • Shunyu Yao and Howard Chen and John Yang and Karthik Narasimhan
  • Key: Web, datasets
  • 2022
  • code

• Rico: A Mobile App Dataset for Building Data-Driven Design Applications

  • Deka, Biplab and Huang, Zifeng and Franzen, Chad and Hibschman, Joshua and Afergan, Daniel and Li, Yang and Nichols, Jeffrey and Kumar, Ranjitha
  • Key: mobile app, datasets
  • 2017

Related Repositories

• awesome-llm-powered-agent
• Awesome-LLM-based-Web-Agent-and-Tools
• Awesome-GUI-Agent
• computer-control-agent-knowledge-base

Contributing

Our purpose is to make this repo even better. If you are interested in contributing, please refer to HERE for instructions in contribution.

License

This repository is released under the Apache 2.0 license.