harpreetsahota204/gui_agent_research_landscape

An awesome repository that maps the current landscape of GUI/OS Agent research

Python

39 commits

updated Aug 18, 2025

README

GUI Agent Research Landscape

A comprehensive analysis of the research landscape in GUI agents, OS agents, and visual agents, spanning from 2016 to 2025. This repository contains a complete research pipeline for extracting, analyzing, and understanding trends in this rapidly evolving field.

Overview

This project provides a systematic approach to understanding the development and evolution of GUI agent research through data extraction, intelligent filtering, LLM-powered analysis, and comprehensive visualization. The analysis reveals key insights about research trends, influential papers, citation networks, and the field's trajectory.

Data Sources

Initial Dataset

The foundation of this analysis comes from the excellent work by the LLM-Brained GUI Agents Survey project. The initial dataset in the data/ directory contains four categories of papers:

benchmark.json - Papers introducing new evaluation benchmarks
dataset.json - Papers contributing new datasets
models.json - Papers presenting new models and architectures
survey.json - Survey and review papers

Each paper entry includes metadata such as title, platform, publication date, arXiv URL, key highlights, and code repository links.

Credit: We gratefully acknowledge @vyokky for curating and maintaining this valuable dataset of GUI agent research papers.

📈 Temporal Analysis of GUI Agent Research Trends

The following table presents a comprehensive analysis of research trends across different time periods, showing the evolution of the GUI agent research landscape. See this file for full details:

Period	Evolution Summary	Future Directions	Dominant Themes	Key Innovations	Emerging Trends	Research Gaps	Methodological Approaches
2016-2021	Transition from static UI analysis to dynamic interaction modeling	Advancing toward robust multimodal models with physics simulation	UI/GUI understanding, RL for navigation, document understanding	Large-scale datasets (Rico, ERICA), multimodal models (UIBert)	Metadata-free UI understanding, structured language integration	Limited annotated datasets, scalability challenges	Transformer architectures, RL with workflow guidance
2022	Integration of LLMs with embodied agents, complex benchmarks	Interactive agents with multi-round dialogue, domain-general models	Vision-language pretraining, embodied agents, LLMs for planning	LLM-Planner, WebShop benchmark, internet-scale pretraining	Interactive navigation, domain adaptation, sim-to-real transfer	Cross-domain generalization, handling ambiguity	RLHF, internet-scale pretraining, prompt engineering
2023 Q1	Tighter LLM-embodied system integration, zero-shot capabilities	Robustness in physical constraints, scalable architectures	LLM-embodied agent integration, vision-language models	Reflexion framework, DEPS planning, CLIP4MC	Zero-shot navigation, multi-modal fusion	Hardware constraints, cross-environment generalization	RL with LLM guidance, self-supervised learning
2023 Q2	Generalist multimodal agents with enhanced real-world capabilities	Real-world deployment robustness, cross-modal reasoning	Multimodal GUI navigation, LLM planning	WebGUM, AdaPlanner, ONE-PEACE models	Generalist cross-domain agents, embodied experiences	Limited real-world benchmarks, domain adaptation	RL with environment feedback, multi-task learning
2023 Q3	Autonomous agents in complex environments, multimodal integration	Efficient computation, physical reasoning, standardized benchmarks	Autonomous web agents, LLM frameworks	AutoDroid, PACE, Auto-GUI, Qwen-VL	Zero-shot physical reasoning, hierarchical frameworks	Real-time adaptation, cross-platform generalization	RL with policy gradients, chain-of-thought prompting
2023 Q4	Integrated vision-language-action systems, zero-shot learning	Real-world adaptability, edge optimization, hallucination mitigation	VLMs for automation, embodied agents, LLMs as controllers	GPT-4V, MM-Navigator, Ferret, MobileVLM	Open-world task automation, 3D vision-language-action	Dynamic environment robustness, edge deployment	RL with foundation models, prompt engineering
2024 Q1	Practical multimodal agents in real-world environments	End-to-end embodied agents, standardized benchmarks	Multimodal agents, benchmarking frameworks	ScreenAgent, WebVoyager, ScreenSpot, ShapeLLM	Ethical AI, multisensory integration, synthetic data	Evaluation standardization, long-term memory	Synthetic data generation, modular architectures
2024 Q2	Edge-optimized multimodal agents, improved benchmarking	Multilingual benchmarks, security enhancement	MLLMs for GUI/Web, on-device optimization	Octopus v4, VGA, InternLM-XComposer2-4KHD	Mobile/edge deployment, RL with LLMs	Multilingual datasets, adversarial robustness	RLHF, multi-agent collaboration, benchmark development
2024 Q3	Specialized autonomous agents, security-focused research	Secure AI-OS integration, cross-cultural benchmarks	Mobile/GUI agent capabilities, benchmarking	MobileVLM, CRAB, Mobile3M, Agent Workflow Memory	Security and privacy focus, AI-OS integration	Non-Western representation, standardized protocols	Benchmark creation, tree search planning
2024 Q4	Foundation-model-driven agents, real-world deployment	Long-term memory, safety mechanisms	GUI agents, web frameworks, safety	TAG, UGround, WebRL, AgentTrek, SPA-Bench	Vision-only GUI interaction, safety evaluation	Long-term memory, safety standardization	RL with curriculum, synthetic data generation
2025 Q1	Autonomous cross-platform agents, ethical deployment	Unified multimodal reasoning, federated learning	GUI/OS automation, environment-free training	AppVLM, VEM, Explorer, VeriSafe Agent	Autonomous evaluation, physical world integration	Benchmarking standardization, long-horizon planning	RL with pretrained value functions, federated learning
2025 Q2	Realistic evaluation, adversarial robustness, multimodal reasoning	Secure efficient agents, cross-platform generalization	Benchmarking, RL for GUI agents, security	AgentRewardBench, InfiGUI-R1, UI-E2I-Synth	Real-world deployment, process reward models	Cross-platform evaluation, ethical frameworks	RL with sub-goal planning, synthetic data generation

Research Pipeline

1. Data Extraction and Expansion (`data_extractor.py`)

The initial dataset was significantly expanded through automated extraction from arXiv:

Multi-level Citation Mining: Starting from the curated papers (Level 1), we extracted their references to identify relevant cited works (Level 2), and then extracted references from those papers (Level 3)
Content Extraction: For each paper, we extracted full text content including abstracts, introductions, methodology sections, and conclusions from arXiv HTML pages
Metadata Enrichment: Enhanced each paper with author information, publication dates, platform classifications, and research categories
Citation Network Construction: Built comprehensive forward and backward citation relationships between papers
Special Paper Integration: Included important non-arXiv papers like Rico and ERICA datasets that are frequently cited in the field

This process expanded the dataset from ~180 curated papers to over 9,000 papers with rich metadata and full content.

2. Intelligent Filtering (`filter_papers.py`)

To maintain focus on GUI/OS/Visual agent research, we implemented a sophisticated keyword-based filtering system:

Multi-tier Keyword Classification: Used high-precision keywords (GUI agent, visual grounding, screenshot analysis) and medium-precision keywords (visual agent, computer control) with different scoring weights
Context-aware Filtering: Enhanced relevance detection by considering keyword co-occurrence and context
Exclusion Patterns: Filtered out papers from unrelated domains (medical, financial, pure NLP) to maintain research focus
Level-based Strategy: Kept all Level 1 (curated) papers while applying filtering to Level 2 and 3 papers to balance comprehensiveness with relevance

This reduced the dataset to approximately 600 highly relevant papers while maintaining the core research landscape.

3. LLM-Powered Content Analysis (`paper_analyzer.py`)

Each paper was analyzed using Qwen3-8B to extract structured insights:

Key Contributions Extraction: Identified the main research themes and contributions from abstracts, introductions, and conclusions
Innovation Classification: Automatically detected whether papers introduce new models, datasets, benchmarks, or frameworks
Contribution Analysis: Analyzed how each work differs from and builds upon related work
Batch Processing: Optimized GPU utilization through efficient batch processing for scalable analysis
Quality Assurance: Implemented fallback mechanisms and error handling to ensure robust analysis

This enriched each paper with structured metadata about its contributions and innovations.

4. Temporal Trend Analysis (`research_trend_analyzer.py`)

Research trends were analyzed across different time periods using Qwen3-14B:

Time Period Segmentation: Organized papers into meaningful periods (2016-2021 Early Era, 2022 Growth Year, 2023+ Quarterly analysis)
Trend Identification: For each period, identified dominant themes, key innovations, emerging trends, and research gaps
Methodological Evolution: Tracked changes in research approaches and methodologies over time
Future Direction Prediction: Generated insights about where the field might be heading based on current trends
Cross-period Analysis: Compared different eras to understand the field's evolution

This provided a comprehensive understanding of how GUI agent research has evolved and where it's headed.

5. Timeline and Temporal Analysis (`research_timeline_analysis.py`)

Comprehensive temporal analysis with multiple visualization approaches:

Publication Timeline: Tracked paper publication rates over time with trend projections for future years
Platform Evolution: Analyzed how research focus has shifted across different platforms (Web, Mobile, Desktop, etc.)
Innovation Trends: Examined the temporal distribution of different types of contributions (models, datasets, benchmarks)
Research Acceleration: Identified periods of rapid growth and analyzed sustainability indicators
Granular Analysis: Provided quarterly-level analysis for recent years to capture fine-grained trends

Generated multiple visualizations showing the field's rapid growth, particularly accelerating after 2022.

6. Citation Network Analysis (`citation_network_analysis.py`)

Deep analysis of the citation network and research influence patterns:

Network Construction: Built a comprehensive citation graph connecting papers through their references
Influence Metrics: Calculated various influence measures including citation counts, PageRank scores, and network centrality
Foundation vs. Frontier Papers: Distinguished between foundational works that established the field and frontier papers pushing current boundaries
Citation Burst Detection: Identified papers experiencing rapid citation growth, indicating emerging influence
Research Bridge Analysis: Found papers that connect different research communities or approaches
Temporal Dynamics: Analyzed how influence patterns change over time and identified paradigm shifts

This revealed the field's key influential works, emerging stars, and the structure of research communities.

Key Findings

Rapid Growth: The field has experienced exponential growth, particularly since 2022, with 2024 seeing unprecedented research activity
Platform Diversification: Research has expanded from early web-focused work to encompass mobile, desktop, and cross-platform approaches
Innovation Patterns: Strong emphasis on new models and architectures, with increasing focus on comprehensive benchmarks and datasets
Emerging Trends: Growing interest in multimodal approaches, real-world deployment, and foundation models for GUI understanding
Research Maturation: Evolution from proof-of-concept work to production-ready systems and comprehensive evaluation frameworks

Citation Network Analysis Results

The citation_network_analysis.py script performs comprehensive analysis of the citation network among GUI agent research papers, revealing influential works, research patterns, and field evolution. The analysis generates multiple specialized tables and visualizations.

📊 Analysis Outputs

The citation network analysis produces the following key tables and insights:

Core Influence Rankings

Most Cited Papers - Top papers by raw citation count, showing the most referenced works in the field
Influential Papers Ranking - PageRank-based influence ranking that considers network effects and citation quality
Foundation Papers - Papers that established key concepts and methodologies in GUI agent research
Frontier Papers - Recent papers pushing the boundaries of current research

Research Infrastructure Analysis

Benchmark Papers - Papers introducing evaluation benchmarks and testing frameworks
Dataset Papers - Papers contributing datasets for training and evaluation
Model Papers - Papers introducing new models and architectures (GUI agent-specific)

Temporal Dynamics

Temporal Analysis - Papers with dynamic citation patterns and recent momentum
Citation Velocity - Papers with high citation rates and sustained impact
Future Impact Signals - Papers showing early indicators of potential high impact

Visualizations

Research Timeline - Visual timeline showing the evolution of influential papers over time

🔍 Key Insights from Citation Analysis

The citation network analysis reveals several important patterns:

Field Evolution

Foundation Era (2016-2021): Established by datasets like Rico and ERICA, and early GUI understanding frameworks
Growth Phase (2022-2023): Rapid expansion with new benchmarks like Mind2Web, WebArena, and WebShop
Current Wave (2024-2025): Focus on production-ready agents, multimodal approaches, and comprehensive evaluation

Most Influential Works

The analysis identifies papers with the highest network influence (combining citations and network centrality):

Rico Dataset (2017): Foundational mobile app dataset enabling data-driven GUI research
Mind2Web (2023): Breakthrough web agent benchmark that catalyzed current research wave
WebArena (2023): Comprehensive web environment for autonomous agent evaluation
Recent Multimodal Models: Growing influence of vision-language models adapted for GUI tasks

Research Patterns

Citation Bursts: Certain papers experience rapid citation growth following major conferences or breakthroughs
Cross-Platform Influence: Papers spanning multiple platforms (web, mobile, desktop) tend to have higher influence
Benchmark-Driven Growth: Introduction of new benchmarks consistently drives citation activity and follow-up research

Emerging Trends

Multimodal Integration: Increasing focus on combining vision and language for GUI understanding
Real-world Deployment: Shift from proof-of-concept to production-ready agent systems
Evaluation Sophistication: More comprehensive and realistic evaluation frameworks
Foundation Model Adaptation: Leveraging and fine-tuning large language models for GUI tasks

📈 Network Statistics

The citation network analysis processes:

Total Papers Analyzed: 600+ filtered and enriched papers
Citation Relationships: Comprehensive forward and backward citation links
Time Span: Research from 2016 to 2025
Platform Coverage: Web, Mobile, Desktop, Android, iOS, and cross-platform research

🎯 How to Use These Results

For Literature Review: Start with Foundation Papers and Most Cited Papers for comprehensive background
For Current Trends: Focus on Frontier Papers and Temporal Analysis for cutting-edge research
For Benchmarking: Reference Benchmark Papers for evaluation frameworks and comparison baselines
For Implementation: Check Model Papers and Dataset Papers for practical resources
For Future Research: Review Future Impact Signals for emerging opportunities

🔄 Analysis Methodology

The citation network analysis employs several sophisticated metrics:

PageRank Algorithm: Measures influence based on citation network structure
Temporal Pattern Detection: Identifies citation bursts and momentum changes
Cross-Platform Analysis: Evaluates papers spanning multiple research domains
Foundation vs. Frontier Classification: Distinguishes established works from cutting-edge research

All tables include arXiv badges for direct access to papers, along with key contributions and innovation summaries extracted via LLM analysis.

Repository Structure

├── data/                          # Initial curated dataset
│   ├── benchmark.json            # Benchmark papers
│   ├── dataset.json             # Dataset papers  
│   ├── models.json              # Model papers
│   └── survey.json              # Survey papers
├── src/                         # Source code directory
│   ├── data_extractor.py        # ArXiv data extraction and expansion
│   ├── filter_papers.py         # Intelligent paper filtering
│   ├── paper_analyzer.py        # LLM-powered content analysis  
│   ├── research_trend_analyzer.py # Temporal trend analysis
│   ├── research_timeline_analysis.py # Timeline visualization
│   └── citation_network_analysis.py # Citation network analysis
└── README.md                    # This file

Usage

Data Extraction: Run data_extractor.py to expand the dataset from arXiv
Filtering: Use filter_papers.py to focus on relevant GUI agent research
Content Analysis: Apply paper_analyzer.py for LLM-powered insights
Trend Analysis: Execute research_trend_analyzer.py for temporal trends
Visualization: Run timeline and network analysis scripts for comprehensive insights

Each script includes detailed command-line options and can be run independently or as part of the full pipeline.

Requirements

Python 3.8+
PyTorch with CUDA support (for LLM analysis)
Transformers library (HuggingFace)
NetworkX, Matplotlib, Seaborn (for analysis and visualization)
BeautifulSoup, Requests (for web scraping)

Citation

If you find this research landscape analysis helpful for your work, please consider citing:

@misc{sahota2025gui,
  title={GUI Agent Research Landscape: A Comprehensive Analysis of Trends, Networks, and Evolution},
  author={Sahota, Harpreet},
  year={2025},
  url={https://github.com/harpreetsahota204/gui_agent_research_landscape},
  note={Comprehensive analysis of GUI agent research from 2016-2025}
}

Acknowledgments

Original Dataset: @vyokky for the foundational curated dataset
LLM Analysis: Powered by Qwen3-8B and Qwen3-14B models from Alibaba Cloud
Research Community: All the researchers whose work is analyzed in this landscape study

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Limitations

Publication Source Bias
- Analysis primarily covers papers from arXiv
- Misses important work published exclusively in conferences, journals, or industry venues
- Only manually added exceptions (Rico and ERICA) for non-arXiv papers
Citation Network Boundaries
- Limited to 3 levels of citation depth
- May miss influential papers that are more distantly connected
- Creates an artificial boundary in understanding the full research landscape
Content Analysis Constraints
- Relies on keyword-based detection for platforms and categories
- May miss nuanced or emerging concepts not covered by predefined keywords
- Only extracts specific sections (abstract, intro, related work, methodology, conclusion)
Reference Coverage
- Primarily tracks and analyzes references with arXiv IDs
- Limited analysis of citations to non-arXiv papers
- May underrepresent industry impact or practical applications

These limitations mean the analysis provides a strong academic perspective but may not fully capture the complete research landscape, particularly for industry developments or papers primarily published through traditional academic venues.

Papers by Time Period

The following tables organize all papers in the dataset by time periods, showing their key contributions and innovations.

Total Papers: 609

Papers by Period:

2016-2021: Early Era: 28 papers
2022: Growth Year: 39 papers
2023: Q1: 20 papers
2023: Q2: 36 papers
2023: Q3: 29 papers
2023: Q4: 62 papers
2024: Q1: 71 papers
2024: Q2: 84 papers
2024: Q3: 56 papers
2024: Q4: 74 papers
2025: Q1: 71 papers
2025: Q2: 39 papers

2016-2021: Early Era

Title	Summary	Contributions
WebGPT: Browser-assisted question-answering with human feedback	The paper introduces a browser-assisted question-answering system that integrates web browsing with GPT-3, utilizing imitation learning and human feed...	This work differs from related work by explicitly combining web-browsing capabilities with large language models, leveraging human feedback for iterat...
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning	ALFWorld bridges abstract text-based policy learning and concrete visual execution, introducing the BUTLER agent that leverages pre-learned abstract k...	Unlike prior work focused on either abstract reasoning or visual execution, ALFWorld combines both through a unified simulator, enabling agents to tra...
Mapping Natural Language Instructions to Mobile UI Action Sequences	The paper introduces a novel problem of grounding natural language instructions to mobile UI actions, develops three new datasets including PIXELHELP,...	This work differs from related work by focusing specifically on mobile UI interaction, creating task-specific datasets with grounded action sequences,...
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration	The paper introduces workflow-guided exploration for reinforcement learning on web interfaces, using high-level workflows derived from expert demonstr...	This work differs from related work by using workflow-guided exploration to constrain action sequences with expert demonstrations, rather than relying...
AndroidEnv: A Reinforcement Learning Platform for Android	The paper introduces AndroidEnv, a reinforcement learning platform for Android that enables agents to interact with real-world apps via a touchscreen...	AndroidEnv differs from related work by providing a realistic Android simulation environment tailored for RL research, enabling direct interaction wit...
UIBert: Learning Generic Multimodal Representations for UI Understanding	The paper introduces UIBert, a transformer-based model for UI understanding that leverages self-aligned multimodal features (image, text, structural m...	Unlike prior work, UIBert utilizes self-alignment between UI component features and proposes five pre-training tasks to learn generic representations...
WebSRC: A Dataset for Web-Based Structural Reading Comprehension	The paper introduces WebSRC, a novel dataset for structural reading comprehension on web pages, emphasizing the need for systems to understand both se...	This work differs from related work by focusing on structural understanding of web pages, providing a comprehensive dataset with HTML, screenshots, an...
OCR-free Document Understanding Transformer	The paper introduces an OCR-free approach for document understanding using a Transformer model (Donut), addressing limitations of OCR-based methods by...	The work differs from related OCR-based VDU methods by eliminating reliance on OCR engines, using a Transformer with cross-entropy loss for end-to-end...
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning	The paper introduces robosuite, a modular simulation framework and benchmark for robot learning, leveraging the MuJoCo physics engine. Key contributio...	robosuite differs from related work by providing a modular framework with standardized benchmarks and task environments, enabling rigorous evaluation...
DocVQA: A Dataset for VQA on Document Images	The paper introduces DocVQA, a specialized dataset for Visual Question Answering on document images, emphasizing structural understanding challenges....	DocVQA differs from related work by focusing specifically on document images, requiring models to understand document structure, and providing a bench...
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embod...	The paper introduces HM3D, a large-scale 3D dataset with 1,000 building-scale environments, emphasizing its superior physical scale, visual fidelity,...	HM3D differs from related work by offering significantly larger navigable space (112.5k m²), higher visual fidelity (20-85% improvement over Replica/M...
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation	ThreeDWorld introduces a platform for interactive multi-modal physical simulation with high-fidelity rendering, customizable agents, generative enviro...	Unlike prior work, ThreeDWorld provides a unified platform with real-time physics simulation, multi-modal sensory data, and customizable agents, enabl...
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models	The paper introduces Cross-modal Prompt Tuning (CPT) to address the gap between pre-training and fine-tuning objectives in Vision-Language Pre-trained...	Unlike traditional fine-tuning approaches that require large labeled datasets, CPT leverages color-based prompt tuning to bridge the pre-training and...
Rearrangement: A Challenge for Embodied AI	The paper introduces a framework for Embodied AI research centered on the rearrangement task, emphasizing standardized metrics, scenario characterizat...	This work differs from related work by establishing a standardized rearrangement task with formal metrics and simulation testbeds, enabling reproducib...
Grounding Open-Domain Instructions to Automate Web Support Tasks	The paper introduces RUSS, a system that grounds open-domain natural language instructions into web actions using a domain-specific language (ThingTal...	This work differs from related work by introducing ThingTalk as an intermediate domain-specific language for grounding instructions, enabling more pre...
A Systematic Literature Review of Automated Techniques for Functional GUI Testin...	The paper provides a systematic review of automated GUI testing techniques for mobile applications, analyzing their effectiveness, efficiency, and pra...	This work differs from related studies by conducting a comprehensive systematic review of 25 primary studies to evaluate and compare existing automate...
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Un...	The paper introduces MarkupLM, a pre-training model that jointly learns text and markup language information for document understanding, addressing th...	This work differs from related work by using markup languages as the backbone for pre-training, rather than relying on fixed layouts or images. It exp...
Learning UI Navigation through Demonstrations composed of Macro Actions	The paper introduces a framework for UI navigation that simplifies state and action spaces using UI elements and macro actions. It proposes demo augme...	This work differs from related work by focusing on macro-action-based navigation with demo augmentation and customized DQfD, enabling efficient traini...
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household...	iGibson 2.0 introduces object-centric simulation with enhanced object states (temperature, wetness, cleanliness, toggled/sliced), predicate logic func...	Unlike prior simulation environments focused on motion and physics, iGibson 2.0 extends capabilities to handle complex object states and semantic task...
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospecti...	The paper introduces HANNA, a photo-realistic simulator for visual navigation tasks with human-like assistance. It proposes a memory-augmented neural...	This work differs from related work by integrating multimodal assistance (language + vision) with retrospective curiosity-driven imitation learning, h...
pix2code: Generating Code from a Graphical User Interface Screenshot	The paper introduces an end-to-end deep learning model that generates code from GUI screenshots with high accuracy across multiple platforms. It addre...	This work differs from related work by leveraging deep learning for direct code generation from screenshots, whereas prior methods often relied on man...
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences	This paper introduces a vision-based approach for mobile GUI understanding by abstracting visual elements into 'Pixel-Words' and structuring them into...	Unlike prior works dependent on human-created metadata (e.g., View Hierarchy), this work defines atomic visual components (Pixel-Words) and employs a...
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning	The paper introduces Screen2Words, a multi-modal approach for mobile UI summarization that integrates text, images, UI structures, and semantics. It p...	The work differs from related work by combining multi-modal data (text, images, UI structure, semantics) for UI summarization and introducing a large-...
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning...	Introduces a physically realistic benchmark for task-and-motion planning in embodied AI, emphasizing visual guidance and physics-driven interactions....	This work differs by creating a comprehensive benchmark that integrates visual task understanding, physics-based object manipulation, and realistic en...
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or...	The paper conducts a large-scale empirical study on GUI element detection methods, identifies limitations of existing approaches, and proposes a novel...	The work differs by introducing a GUI-specific old-fashioned method with a top-down coarse-to-fine strategy, integrating it with deep learning for tex...
DOM-Q-NET: Grounded RL on Structured Language	The paper addresses challenges in web navigation for RL agents by introducing DOM-Q-NET, which handles large discrete action spaces and varying action...	DOM-Q-NET differs from related work by using a structured representation of HTML via graph neural networks and separating action-specific Q-networks f...
Document Visual Question Answering Challenge 2020	The paper introduces a new problem in visual question answering (VQA) focused on document images, proposing two tasks: single-image question answering...	This work differs from related work by specifically addressing VQA on document images, which is distinct from general-purpose VQA tasks. It introduces...
VisualMRC: Machine Reading Comprehension on Document Images	Introduces VisualMRC, a dataset focused on document images with abstractive answers, and a new model integrating visual layout and content for machine...	Differently focuses on document-specific reading comprehension with abstractive answers, integrating visual layout understanding, whereas related work...

2022: Growth Year

Title	Summary	Contributions
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embod...	The paper explores using large language models (LLMs) to decompose high-level tasks into actionable steps for embodied agents without additional train...	This work differs from prior methods by leveraging pre-trained LLMs for zero-shot task decomposition, avoiding explicit step-by-step training. It focu...
LAION-5B: An open large-scale dataset for training next generation image-text mo...	The paper introduces LAION-5B, a large-scale image-text dataset enabling training of advanced language-vision models. It emphasizes the importance of...	LAION-5B differs from prior work by offering the first openly available dataset of this scale, enabling broader research on multi-modal models without...
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI	The paper introduces a GUI-based task-oriented dialogue system (GUI-TOD) that directly interacts with mobile app GUIs without relying on backend APIs,...	This work differs from related work by eliminating reliance on task-oriented dialogue-specific backend APIs, enabling direct GUI interaction for task...
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agen...	The paper introduces WebShop, a large-scale simulated e-commerce environment with real-world products and crowd-sourced instructions, addressing scala...	This work differs from related work by creating a scalable, real-world benchmark (WebShop) with extensive product and instruction data, enabling evalu...
Understanding HTML with Large Language Models	The paper explores the application of large language models (LLMs) to HTML understanding tasks, demonstrating their effectiveness in semantic classifi...	This work differs from related work by demonstrating that pre-trained LLMs, when fine-tuned, outperform task-specific models in HTML understanding tas...
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots	Introduces ScreenQA, a large-scale benchmark dataset for screen content understanding through question-answering, bridging low-level structural analys...	ScreenQA is the largest QA dataset for mobile screenshots, using full screenshots rather than cropped regions, and includes unanswerable questions and...
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language...	The paper introduces LLM-Planner, a novel method leveraging large language models (LLMs) for few-shot planning in embodied agents. It addresses high d...	LLM-Planner differs from related work by integrating physical grounding into LLM planning, enabling environment-aware few-shot task execution without...
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus	The paper introduces a vision-only approach for mobile UI understanding, emphasizing the use of screenshots and a region of interest (focus) to bypass...	Unlike prior work reliant on view hierarchies, Spotlight uses a vision-language model with screenshot and focus inputs, achieving state-of-the-art res...
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge	MineDojo introduces a framework for building generalist embodied agents by integrating an open-ended simulation environment, an internet-scale multimo...	Unlike prior work focused on isolated environments with limited tasks, MineDojo combines a diverse simulation suite, large-scale knowledge integration...
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos	The paper introduces Video PreTraining (VPT), a method for training agents in sequential decision domains by leveraging unlabeled online videos throug...	This work differs from related work by extending internet-scale pretraining to sequential decision domains using unlabeled videos, enabling zero-shot...
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding	The paper introduces Pix2Struct, a pretrained image-to-text model for visual language understanding, focusing on parsing screenshots into HTML. It add...	Pix2Struct differs from prior work by using HTML structure as a pretraining target for visual parsing, enabling OCR-free end-to-end understanding. It...
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-...	The paper introduces OFA, a unified framework that unifies cross-modal and unimodal tasks through a sequence-to-sequence learning paradigm. It emphasi...	OFA differs from related work by eliminating task-specific layers through a simple sequence-to-sequence framework, achieving competitive performance w...
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Dere...	The paper introduces MatCha, a pretraining framework that enhances visual language models by integrating math reasoning and chart derendering tasks. I...	Unlike prior work, MatCha focuses on explicit plot deconstruction and numerical reasoning, addressing limitations in existing vision-language models f...
Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied...	The paper introduces a modular approach for Embodied Instruction Following (EIF) by integrating physical constraints of robots and leveraging Large La...	Prompter differs from prior work by incorporating physical constraints into modular design and replacing trained object search models with LLM-based p...
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasib...	The paper introduces MoTIF, a novel dataset for vision-language navigation (VLN) in mobile apps that addresses unknown command feasibility. It provide...	This work differs from related work by introducing the first dataset with feasibility annotations for VLN, addressing task uncertainty through follow-...
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking	The paper introduces LayoutLMv3, a multimodal pre-trained model for Document AI that unifies text and image masking with a word-patch alignment object...	Unlike prior multimodal models that used disparate objectives for text and image modalities, LayoutLMv3 introduces a unified masking approach and a cr...
SQA3D: Situated Question Answering in 3D Scenes	Introduces SQA3D, a new task for evaluating embodied agents' scene understanding through situated question answering in 3D environments. Provides a co...	This work differs from related work by focusing on situated question answering in 3D scenes, emphasizing spatial reasoning, multi-hop logic, and commo...
VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attrib...	The paper introduces VL-CheckList, a framework to evaluate vision-language pretraining (VLP) models by analyzing their capabilities in objects, attrib...	Unlike prior work focused on downstream task performance, VL-CheckList provides an explainable, taxonomy-driven benchmark that decomposes VLP model ca...
Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI...	The paper introduces QTypist, a context-aware text input generation framework leveraging pre-trained LLMs for mobile GUI testing. It addresses the cha...	Unlike prior work, QTypist innovatively applies LLMs to GUI testing with a tailored prompt engineering approach for text generation, achieving signifi...
Improving Multimodal Interactive Agents with Reinforcement Learning from Human F...	The paper introduces a method to enhance embodied agents using reinforcement learning from human feedback (RLHF) with a novel 'Inter-temporal Bradley-...	This work differs from related work by employing IBT modeling to capture human judgment dynamics for reward learning, enabling effective RLHF in compl...
Retrospectives on the Embodied AI Workshop	The paper provides a retrospective analysis of Embodied AI research, focusing on three themes: visual navigation, rearrangement, and embodied vision-a...	This work differs from related research by offering a comprehensive retrospective analysis of challenges and progress in Embodied AI, emphasizing cros...
Dichotomy of Control: Separating What You Can Control from What You Cannot	The paper introduces the Dichotomy of Control (DoC) framework for future-conditioned supervised learning in offline reinforcement learning, addressing...	This work differs from related work by explicitly separating policy control mechanisms from environmental stochasticity through latent variable condit...
Unifying Vision, Text, and Layout for Universal Document Processing	The paper introduces UDOP, a unified model for document AI that integrates vision, text, and layout modalities. It proposes a novel Vision-Text-Layout...	UDOP differs from related work by unifying vision, text, and layout modalities into a single model with a novel architecture, enabling both document u...
MUG: Interactive Multimodal Grounding on User Interfaces	The paper introduces MUG, an interactive multimodal grounding task for UI interactions that enables iterative user-agent collaboration. It emphasizes...	Unlike prior works that focused on single-round UI grounding, this work introduces iterative interaction frameworks, a dedicated dataset for multi-rou...
Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Un...	The paper explores the use of pre-trained BERT models for multi-step tasks involving GUI interaction, specifically through web browsers. It introduces...	This work differs from related work by explicitly integrating pre-trained BERTs with GUI interaction for multi-step tasks, whereas prior research prim...
Objaverse: A Universe of Annotated 3D Objects	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...
Enabling Conversational Interaction with Mobile UI using Large Language Models	The paper explores using pre-trained large language models (LLMs) with prompting techniques to enable versatile conversational interactions with mobil...	This work differs from related work by avoiding the need for task-specific datasets or models, leveraging general-purpose LLMs with carefully designed...
CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation	The paper explores zero-shot Vision-and-Language Navigation (VLN) using CLIP, demonstrating its capability to navigate environments via natural langua...	This work differs from related work by applying CLIP for zero-shot VLN using natural language referring expressions (rather than object class template...
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends	The paper provides a comprehensive survey of vision-language pre-training (VLP) methods, categorizing them into image-text, core computer vision, and...	This work differs from related work by offering a structured survey of VLP methods across diverse modalities, emphasizing challenges and advanced rese...
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-conn...	The paper introduces mPLUG, a vision-language foundation model addressing computational inefficiency and information asymmetry in cross-modal alignmen...	mPLUG differs from prior work by introducing cross-modal skip-connections to reduce computational costs of long visual sequences and mitigate informat...
A data-driven approach for learning to control computers	The paper introduces a data-driven approach combining reinforcement learning with behavioral priors derived from human-computer interactions to achiev...	This work differs from related work by focusing on scalable reinforcement learning with behavioral priors from real human interactions, avoiding hand-...
Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Wid...	The paper introduces a psychologically-inspired, unsupervised method for perceptual grouping of GUI widgets using Gestalt principles, addressing the c...	This work differs from related work by leveraging Gestalt theory for unsupervised perceptual grouping without supervision or runtime data, outperformi...
Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcemen...	The paper introduces PIRLTest, a platform-independent GUI testing framework combining image embedding and reinforcement learning. It addresses limitat...	Unlike prior work relying on static models or random exploration, PIRLTest innovates by embedding GUI images as states for reinforcement learning, usi...
Masader Plus: A New Interface for Exploring +500 Arabic NLP Datasets	The paper introduces Masader Plus, a web interface designed to enhance the exploration of Arabic NLP datasets by addressing UX challenges, enabling da...	This work differs from related work by providing a user-centric web interface for dataset exploration, emphasizing usability improvements and interact...
Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language...	The paper introduces a method combining language models and reinforcement learning for voxel building tasks in Minecraft-like environments, emphasizin...	This work differs by integrating language model-generated sub-goals with reinforcement learning for task execution, improving feasibility verification...
IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents	The paper introduces IGLU Gridworld, a reinforcement learning environment focused on embodied dialog agents with visual embodiment, language-condition...	Unlike prior work, IGLU Gridworld provides a unified benchmark combining visual agent embodiment, language conditioning, and complex task spaces, enab...
Collecting Interactive Multi-modal Datasets for Grounded Language Understanding	The paper formalizes a collaborative embodied agent using natural language tasks, develops a scalable data collection tool, and introduces the first d...	This work differs from related work by providing the first interactive grounded language understanding dataset, along with a formal framework for embo...
Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phras...	The paper advances 3D scene understanding by introducing a fine-grained task (3DPAG) that explicitly handles phrase-level object relationships in visu...	This work differs from related work by focusing on fine-grained phrase-aware grounding rather than coarse-grained object localization, introducing a l...
AccoMontage2: A Complete Harmonization and Accompaniment Arrangement System	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q1

Title	Summary	Contributions
GPT-4 Technical Report	GPT-4 is a large-scale multimodal model with human-level performance on professional and academic benchmarks, improved post-training alignment for fac...	Differs from prior work by achieving human-level performance on complex benchmarks (e.g., simulated bar exams), demonstrating multimodal capabilities...
A Survey of Large Language Models	The paper provides a comprehensive survey of large language models (LLMs), focusing on their evolution from statistical to neural models, the impact o...	This work differs from related work by offering a structured review of LLMs' advancements, emphasizing scaling laws, emergent capabilities, and practi...
Reflexion: Language Agents with Verbal Reinforcement Learning	Reflexion introduces a novel framework for language agents that uses verbal reinforcement learning through self-reflective feedback. It leverages epis...	Reflexion differentiates from related work by using verbal feedback as a 'semantic gradient' instead of traditional reinforcement learning, enabling p...
Describe, Explain, Plan and Select: Interactive Planning with Large Language Mod...	The paper introduces DEPS, an interactive planning framework leveraging Large Language Models (LLMs) to address long-term reasoning and sub-task effic...	DEPS differs from prior work by integrating self-explanation and feedback loops for error correction, introducing a trainable goal selector to optimiz...
Language Models can Solve Computer Tasks	The paper introduces a novel prompting method (Recursively Criticizes and Improves, RCI) enabling pre-trained LLMs to execute computer tasks via natur...	This work differs by using RCI prompting with minimal demonstrations and no task-specific reward functions, achieving state-of-the-art results on Mini...
Where are we in the search for an Artificial Visual Cortex for Embodied Intellig...	The paper presents CortexBench, a comprehensive benchmark for evaluating pre-trained visual representations (PVRs) across 17 EAI tasks. It evaluates e...	This work unifies fragmented EAI research by introducing CortexBench, a broad benchmark for evaluating PVRs across diverse tasks. It refutes prior ass...
Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Age...	The paper addresses the challenge of integrating semantic knowledge from large language models (LLMs) with grounded environmental understanding for em...	This work differs from related work by combining the semantic capabilities of LLMs with grounded models of the environment through a novel decoding fr...
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Lang...	The paper introduces a novel approach combining large language models (LLMs) with reinforcement learning (RL) for embodied agents, focusing on languag...	This work differs from related work by integrating LLMs to hypothesize abstract world models (AWMs) for planning, followed by environment-driven verif...
Collaborating with language models for embodied reasoning	This work explores the integration of large language models (LLMs) with embodied agents to enhance reasoning in complex environments. It introduces a...	This work differs from related work by proposing a novel architecture that synergizes pre-trained language models with embodied agents through a colla...
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills	ManiSkill2 introduces a comprehensive benchmark addressing limitations in existing manipulation benchmarks by incorporating diverse object-level varia...	Unlike prior benchmarks, ManiSkill2 offers a unified framework with extensive object diversity, dynamic simulations, and scalable infrastructure, enab...
Lexi: Self-Supervised Learning of the UI Language	The paper introduces Lexi, a self-supervised model for learning visio-linguistic representations of UIs without relying on metadata like UI trees or a...	This work differs from related work by avoiding reliance on UI metadata, leveraging self-supervised learning with a novel dataset, and focusing on UI-...
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigati...	The paper introduces ESC, a zero-shot object navigation framework that leverages pre-trained vision-language models and commonsense reasoning from lar...	ESC differs from prior work by using pre-trained commonsense knowledge from LLMs for zero-shot object navigation without requiring environment-specifi...
Reinforcement Learning Friendly Vision-Language Model for Minecraft	The paper introduces CLIP4MC, a cross-modal contrastive learning framework for training RL-friendly vision-language models (VLMs) in open-ended tasks....	Unlike prior work, CLIP4MC explicitly incorporates task completion degree into the VLM training objective to provide more instructive reward signals f...
Distilling Internet-Scale Vision-Language Models into Embodied Agents	The paper introduces a method to distill internet-scale vision-language models (VLMs) into embodied agents for language grounding. It combines model d...	This work differs from related work by repurposing pre-trained VLMs for offline supervision of embodied agents through prompt-based distillation and H...
Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zer...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Inte...	The paper introduces NOPA, a method for socially intelligent home assistants that addresses adaptivity and scalability in online goal inference. Key c...	NOPA differs from prior work by integrating neural and probabilistic methods for adaptive goal inference and uncertainty-aware helping strategies, alo...
Alexa Arena: A User-Centric Interactive Platform for Embodied AI	The paper introduces Alexa Arena, a user-centric simulation platform for Embodied AI (EAI) research, enabling human-robot interaction (HRI) missions t...	Alexa Arena differs from related work by offering a user-centric, gamified platform that bridges simulation and real-world HRI, along with a dialog-en...
ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and P...	ViewRefer addresses 3D visual grounding by integrating multi-view knowledge from both text and 3D modalities. It leverages large-scale language models...	Unlike prior works focusing solely on 3D modality or manual alignment, ViewRefer innovates by extracting view knowledge from both text and 3D inputs....
The Universal NFT Vector Database: A Scaleable Vector Database for NFT Similarit...	The paper introduces a scalable vector database for NFT similarity matching, a modular cloud-centered NFT processing system compliant with ERC-721 sta...	This work differs from related work by proposing an off-chain, hardware-agnostic architecture for NFT data aggregation based on vector similarity, com...
CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Emb...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q2

Title	Summary	Contributions
Voyager: An Open-Ended Embodied Agent with Large Language Models	Voyager introduces an LLM-powered embodied lifelong learning agent for open-ended environments like Minecraft, featuring an automatic curiosity-driven...	Unlike prior works that rely on fixed curricula or require human interaction, Voyager enables open-ended exploration through a bottom-up curiosity-dri...
Mind2Web: Towards a Generalist Agent for the Web	The paper introduces Mind2Web, the first dataset for generalist web agents, emphasizing real-world websites, diverse domains/tasks, and user interacti...	Differs from prior work by using real-world websites instead of simulations, providing diverse tasks across 31 domains, and integrating LLMs with a fi...
Multimodal Web Navigation with Instruction-Finetuned Foundation Models	This work introduces WebGUM, a multimodal agent for web navigation that combines vision-language foundation models with instruction-finetuning. It add...	Unlike prior works reliant on domain-specific architectures and online RL, this work leverages instruction-finetuned vision-language foundation models...
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought	The paper introduces EmbodiedGPT, a multi-modal foundation model for embodied AI that integrates vision-language pre-training with embodied planning a...	This work differs from related work by explicitly combining Chain-of-Thought planning with vision-language pre-training, enabling more effective task...
AdaPlanner: Adaptive Planning from Feedback with Language Models	AdaPlanner introduces adaptive planning for LLM agents through feedback, combining in-plan and out-of-plan refinement strategies. It mitigates halluci...	Unlike prior methods that use static plans or greedy actions, AdaPlanner introduces closed-loop adaptive planning with feedback, reducing hallucinatio...
Mindstorms in Natural Language-Based Societies of Mind	The paper introduces the concept of Natural Language-Based Societies of Mind (NLSOMs), where diverse neural networks communicate via natural language...	This work differs from related work by focusing on the societal organization of agents rather than individual models, leveraging natural language as a...
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User In...	This paper introduces a pixel-based approach for GUI agents, demonstrating that agents can outperform humans on instruction-following tasks using only...	The work differs from prior methods by using purely pixel-based inputs and generic action spaces instead of structured representations (e.g., DOM tree...
Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction	The paper introduces Mobile-Env, a comprehensive toolkit for creating qualified GUI interaction benchmarks in Android environments. It addresses limit...	Mobile-Env differs from existing benchmarks by offering a controllable and isolated environment with support for intermediate rewards and instructions...
Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing	This paper introduces GPTDroid, a novel approach leveraging large language models (LLMs) like GPT-3 for zero-shot human-like mobile GUI testing. It fo...	GPTDroid differs from related work by combining LLM-based natural language understanding with GUI testing, using dynamic context iteration and neural...
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension a...	UniChart introduces a pretrained vision-language model specialized for chart comprehension and reasoning, incorporating chart-specific pretraining tas...	UniChart differs from related work by explicitly modeling chart structures (e.g., visual elements, data relationships) through task-specific pretraini...
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Age...	The paper introduces the PET framework, which leverages pre-trained LLMs to simplify control tasks for embodied agents by decomposing tasks into sub-t...	This work differs by using LLMs as teachers to simplify control problems rather than directly solving them, employing modular components (Plan, Elimin...
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitie...	The paper introduces a scalable general representation model (ONE-PEACE) capable of handling unlimited modalities through modality adapters, shared se...	Unlike prior work that often relies on pre-trained models for specific modalities, ONE-PEACE introduces a unified architecture with modality-agnostic...
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control	Synapse introduces state abstraction, trajectory-as-exemplar prompting, and exemplar memory to enhance computer control agents. It addresses limitatio...	Unlike prior work relying on task-specific exemplars or incomplete prompts, Synapse uses abstracted state trajectories and memory-based retrieval to e...
Language Models Meet World Models: Embodied Experiences Enhance Language Models	The paper introduces a framework that enhances language models (LMs) with embodied experiences from world models to improve reasoning and planning in...	This work differs from related work by explicitly combining world models with LMs to bridge the gap in embodied knowledge, while preserving the LMs' g...
WebCPM: Interactive Web Search for Chinese Long-form Question Answering	The paper introduces WebCPM, the first Chinese long-form question answering (LFQA) dataset that leverages interactive web search for information retri...	This work differs from related work by introducing WebCPM, a novel dataset collected via interactive web search, and by emphasizing the synthesis of h...
LLM as A Robotic Brain: Unifying Egocentric Memory and Control	The paper introduces LLM-Brain, a framework that unifies egocentric memory and control in embodied AI using large language models (LLMs). It emphasize...	This work differs from related work by unifying memory and control through an embodied LLM, leveraging zero-shot learning for robotic tasks, and integ...
Prompting Is All You Need: Automated Android Bug Replay with Large Language Mode...	This paper introduces AdbGPT, a lightweight LLM-based approach for automated Android bug replay through prompt engineering. It leverages few-shot lear...	Unlike prior work relying on S2R extraction and predefined patterns, AdbGPT uses LLMs with prompt engineering to infer bug reproduction steps without...
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language...	The paper introduces NavGPT, a purely LLM-based navigation agent that demonstrates explicit reasoning capabilities in vision-and-language navigation (...	This work differs from related work by leveraging large language models (LLMs) for zero-shot VLN without additional training, emphasizing explicit rea...
Towards Generalist Robots: A Promising Paradigm via Generative Simulation	The paper proposes a generative simulation paradigm for robotics research, leveraging large-scale foundation models to automate task and scene generat...	This work differs from related work by advocating a fully automated generative pipeline (generative simulation) that uses foundation models to create...
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human P...	WebGLM introduces a web-enhanced question-answering system that integrates a retriever, bootstrapped generator, and human preference-aware scorer to i...	WebGLM differs by combining retrieval augmentation with human preference learning, creating a novel architecture for efficient web QA. It introduces a...
Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement L...	This paper addresses efficient interaction between agents and large language models (LLMs) using reinforcement learning. It introduces When2Ask, an RL...	Unlike prior work that relies on hard-coded rules or frequent LLM querying, this paper proposes a learned interaction policy (When2Ask) that dynamical...
Multimodal Procedural Planning via Dual Text-Image Prompting	The paper introduces the Multimodal Procedural Planning (MPP) task, which generates paired text-image steps for task execution. It proposes Text-Image...	This work differs from related work by explicitly addressing cross-modal informativeness, temporal coherence, and accuracy in procedural planning thro...
Fine-Grained Visual Prompting	The paper introduces Fine-Grained Visual Prompting (FGVP) to enhance zero-shot instance-level tasks for Vision-Language Models (VLMs). It proposes the...	This work differs from related work by introducing fine-grained visual prompting using segmentation masks (e.g., from SAM) instead of coarse cues like...
Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving	This paper introduces a novel multi-modal visual grounding task (LiDAR Grounding) tailored for autonomous driving scenarios, proposing the Multi-modal...	This work differs from related work by focusing on 3D outdoor autonomous driving scenarios rather than 2D or indoor 3D domains. It introduces a novel...
DUBLIN -- Document Understanding By Language-Image Network	The paper introduces DUBLIN, a model pretrained on web pages with novel objectives for visual document understanding. It achieves state-of-the-art res...	DUBLIN differs from related work by employing three novel training objectives (Masked Document Text Generation, Bounding Box Task, Rendered Question A...
GUILGET: GUI Layout GEneration with Transformer	The paper introduces GUILGET, a transformer-based model for generating GUI layouts from positional constraints (GUI-AGs), emphasizing constraint adher...	GUILGET differs from related work by explicitly addressing GUI design constraints through a transformer architecture that captures semantic relationsh...
A Generalist Dynamics Model for Control	This paper introduces transformer sequence models as dynamics models (TDMs) for control, demonstrating their strong generalization capabilities in bot...	The work differs from related work by focusing on using transformers as dynamics models rather than policies, emphasizing generalization across unseen...
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learnin...	The paper advances deep reinforcement learning (Deep RL) for bipedal robots, demonstrating agile soccer skills through zero-shot sim-to-real transfer....	This work differs by focusing on full-body control of bipedal robots for long-horizon, multi-agent competitive tasks, whereas prior research primarily...
Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solu...	The paper introduces a large-scale, diverse dataset (POIE) for Visual Information Extraction (VIE) in real-world scenarios, along with an end-to-end f...	The work differs by proposing a more challenging and diverse dataset (POIE) compared to existing benchmarks like SROIE and EPHOIE, and introduces a no...
Document Understanding Dataset and Evaluation (DUDE)	The paper introduces a novel dataset (DUDE) for visually-rich documents, emphasizing multi-industry, multi-domain, and multi-page layouts. It proposes...	This work differs from related work by focusing on practical, real-world applicable benchmarks through diverse datasets and multi-task evaluations, ad...
Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysi...	The paper introduces the STARS framework, which enhances LLM-based task learning for embodied agents by addressing limitations of prompt engineering....	STARS differs from related work by combining online task learning, multi-source knowledge exploitation, and proactive LLM response evaluation without...
Navigating to Objects Specified by Images	The paper presents a modular system for embodied agents to navigate to objects specified by images, combining semantic visual reasoning with explorati...	This work differs from related work by using a modular approach with off-the-shelf components for sub-tasks (exploration, re-identification, localizat...
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explana...	The paper introduces Musketeer, a vision-language model trained jointly across multiple heterogeneous tasks with fully shared parameters. It proposes...	Unlike prior work with separate task-specific heads or adapters, Musketeer employs a unified encoder-decoder architecture with fully shared parameters...
ENTL: Embodied Navigation Trajectory Learner	ENTL introduces a unified approach for embodied navigation by integrating world modeling, localization, and imitation learning into a single sequence...	ENTL differs from related work by unifying world modeling and localization into a sequence prediction framework without explicit rewards, using a tran...
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabil...	The paper focuses on developing embodied agents capable of human-like collaboration through grounded language instructions. Key contributions include...	This work differs from related research by providing a comprehensive framework for collecting and utilizing grounded language instructions, offering a...
OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answe...	The paper introduces the first large-scale open-domain Vietnamese VQA dataset (OpenViVQA) with 11,000+ images and 37,000+ question-answer pairs, along...	This work differs from related work by focusing on open-ended answer generation in Vietnamese, providing a new benchmark for low-resource languages, a...

2023: Q3

Title	Summary	Contributions
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text...	The paper introduces Qwen-VL, a series of vision-language models with enhanced visual perception capabilities through a visual receptor, 3-stage train...	The work differs from related work by integrating a meticulously designed 3-stage training pipeline, visual receptor, and input-output interface, alon...
WebArena: A Realistic Web Environment for Building Autonomous Agents	The paper introduces WebArena, a highly realistic and reproducible web environment for autonomous agents, featuring functional websites from four doma...	WebArena differs from prior work by providing a realistic web environment with dynamic, functional websites and a benchmark focused on functional corr...
The Rise and Potential of Large Language Model Based Agents: A Survey	The paper provides a comprehensive survey of LLM-based agents, tracing their conceptual origins, presenting a general framework (brain, perception, ac...	This work differs by offering a holistic framework for LLM-based agents, systematically categorizing applications, and analyzing societal implications...
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control	The paper introduces RT-2, a vision-language-action (VLA) model that integrates large-scale web data with robotic control through co-fine-tuning. It e...	This work differs by co-fine-tuning vision-language models on both robotic trajectory data and internet-scale vision-language tasks, treating actions...
A Real-World WebAgent with Planning, Long Context Understanding, and Program Syn...	The paper introduces WebAgent, an LLM-driven autonomous agent for real-world web automation that addresses open-domainness, long-context HTML understa...	Unlike prior works relying on simulated environments or single LLMs, WebAgent combines HTML-T5 (specialized for HTML with novel attention mechanisms)...
You Only Look at Screens: Multimodal Chain-of-Action Agents	The paper introduces Auto-GUI, a multimodal GUI agent that directly interacts with interfaces without environment parsing or application-specific APIs...	Auto-GUI differs from prior work by eliminating reliance on external tools (e.g., OCR) and application APIs, directly interacting with GUIs through mu...
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis	The paper introduces SDXL, a latent diffusion model for text-to-image synthesis with enhanced performance through a larger UNet backbone, novel condit...	SDXL improves upon existing latent diffusion models by scaling the UNet architecture, introducing novel conditioning techniques, and incorporating a r...
Android in the Wild: A Large-Scale Dataset for Android Device Control	The paper introduces a large-scale dataset (AITW) for device-control research, emphasizing visual and language interaction, multi-step tasks, and robu...	This work differs from related work by providing an order-of-magnitude larger dataset with diverse Android versions, device types, and complex multi-s...
AutoDroid: LLM-powered Task Automation in Android	AutoDroid integrates large language models (LLMs) with domain-specific app knowledge through dynamic analysis to enable scalable, hands-free mobile ta...	AutoDroid differs from related work by combining LLM commonsense knowledge with app-specific domain knowledge through automated dynamic analysis, elim...
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs	BuboGPT enhances multi-modal LLMs with fine-grained visual grounding, enabling precise cross-modal interactions between vision, audio, and language. I...	Unlike prior works that rely on coarse-grained mappings, BuboGPT achieves fine-grained visual grounding by integrating a SAM-based module and a novel...
Building Cooperative Embodied Agents Modularly with Large Language Models	The paper introduces a modular framework integrating Large Language Models (LLMs) into embodied agents for decentralized multi-agent cooperation. It p...	This work differs from related research by employing a decentralized, modular cognitive-inspired framework that leverages LLMs for reasoning, language...
KOSMOS-2.5: A Multimodal Literate Model	KOSMOS-2.5 introduces a multimodal literate model for document-level text recognition and structured output generation, combining spatial-aware text b...	Unlike prior work focused on line-level OCR or domain-specific structured parsing, KOSMOS-2.5 unifies spatial and structural document understanding th...
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizati...	Introduces Retroformer, a framework for reinforcing large language agents using policy gradient optimization with a retrospective model that automatic...	Proposes a policy gradient approach for language agents, differentiating from prior work by leveraging environment rewards directly rather than relyin...
Embodied Task Planning with Large Language Models	This paper introduces TaPA, an embodied task planning agent that integrates large language models (LLMs) with visual perception for grounded planning....	The work differs by combining LLM-generated plans with scene-grounded object detection for executable actions, using a custom multimodal dataset for t...
Physically Grounded Vision-Language Models for Robotic Manipulation	The paper introduces PhysObjects, a large-scale dataset annotating physical concepts of household objects, and demonstrates how fine-tuning vision-lan...	This work differs from prior work by focusing on physical concept annotations (e.g., material, fragility) rather than visual attributes, and by integr...
An In-depth Survey of Large Language Model-based Artificial Intelligence Agents	The paper provides a comprehensive comparison between LLM-based AI agents and traditional agents, analyzing core components (planning, memory, tool us...	This work differs from related work by offering a systematic survey of LLM-based agents, emphasizing their unique strengths in natural language proces...
When Do Program-of-Thoughts Work for Reasoning?	The paper investigates the effectiveness of program-of-thought prompting for reasoning, introducing the Complexity-Impacted Reasoning Score (CIRS) to...	This work differs from related studies by introducing CIRS, a metric that evaluates code complexity through both structural (AST) and logical (difficu...
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as a...	LLM-Grounder introduces a zero-shot, open-vocabulary 3D visual grounding approach using LLMs as agents to decompose queries, reason about spatial rela...	Unlike prior CLIP-based methods that suffer from 'bag-of-words' limitations, LLM-Grounder leverages LLMs for compositional semantic decomposition and...
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution...	The paper introduces DoReMi, a framework for grounding language models in robotic tasks by detecting and recovering from plan-execution misalignment....	Unlike prior work that assumes perfect execution of low-level skills or relies on manual feedback, DoReMi introduces a hierarchical framework with aut...
Learning to Model the World with Language	The paper introduces Dynalang, a multimodal agent that unifies language understanding with future prediction as a self-supervised learning objective....	Unlike prior work that relies on task-specific language instructions or supervised learning, Dynalang uses a world model to predict future text/image...
InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent	The paper explores integrating ChatGPT into embodied agent systems through role-based prompt engineering, demonstrating high success rates in simulate...	Unlike prior work, InterAct leverages ChatGPT's versatility through role assignment (e.g., checker, sorter) and prompt engineering to enhance cooperat...
REX: Rapid Exploration and eXploitation for AI Agents	The paper introduces REX, a framework enhancing AI agents' exploration and exploitation by integrating UCB-inspired rewards and offline behavior utili...	REX differs from related work by incorporating UCB-based reward structures and try-and-fail procedures akin to RL, enabling efficient offline behavior...
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Em...	The paper introduces the SimBot Challenge as a new competition for developing embodied AI agents, emphasizing the integration of vision, language, and...	This work differs from related work by establishing the SimBot Challenge as a benchmark for embodied AI, focusing on task completion in simulated phys...
Planning with Logical Graph-based Language Model for Instruction Generation	The paper introduces Logical-GLM, a graph-based language model that integrates logical Bayesian graphs with language models to enhance instruction gen...	Unlike prior work that relies solely on pre-trained LLMs or static knowledge graphs, Logical-GLM combines AI planning with dynamic logical graph struc...
ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation	The paper introduces two-stage sampling and dynamic sampling approaches to enhance the efficiency of reinforcement learning (RL) in sequence generatio...	This work differs from related work by focusing on optimizing sampling efficiency in RL for sequence generation, rather than proposing new architectur...
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model	The paper introduces PACE, a novel prompt editing framework leveraging actor-critic reinforcement learning to enhance LLM performance. It addresses pr...	PACE differs from related work by using actor-critic within LLMs themselves for prompt optimization, avoiding external reward models or training. It e...
Pair then Relation: Pair-Net for Panoptic Scene Graph Generation	This paper addresses the challenge of Panoptic Scene Graph (PSG) generation by introducing Pair-Net, a novel framework that improves inter-object pair...	The work introduces Pair-Net, a novel architecture that directly learns sparse pairwise relationships via a Matrix Learner within the PPN, improving u...
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation...	VELMA introduces a novel approach to urban Vision and Language Navigation (VLN) by leveraging large language models (LLMs) with verbalization of traje...	VELMA differs from prior work by employing a verbalization-based embodiment strategy, where navigation instructions, trajectories, and visual observat...
WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q4

Title	Summary	Contributions
MiniGPT-v2: large language model as a unified interface for vision-language mult...	The paper introduces MiniGPT-v2, a unified interface leveraging large language models for vision-language tasks. Key contributions include task-specif...	Unlike prior work, MiniGPT-v2 introduces task-specific identifiers during training to enhance task distinction and learning efficiency, combined with...
CogAgent: A Visual Language Model for GUI Agents	CogAgent introduces a specialized visual language model (VLM) for GUI agents, achieving state-of-the-art performance on VQA benchmarks and GUI navigat...	CogAgent differs by directly processing GUI screenshots (not HTML/OCR) with a high-resolution VLM architecture, enabling human-level GUI understanding...
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V	The paper introduces Set-of-Mark (SoM), a novel visual prompting method that enhances visual grounding capabilities of large multimodal models like GP...	This work differs from related work by focusing on prompt engineering rather than model architecture or training methods, enabling zero-shot visual gr...
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navig...	The paper introduces MM-Navigator, a GPT-4V-based agent for zero-shot smartphone GUI navigation, demonstrating high accuracy in action description and...	This work differs from related work by leveraging GPT-4V's advanced screen interpretation and action reasoning capabilities for zero-shot GUI navigati...
Ferret: Refer and Ground Anything Anywhere at Any Granularity	Ferret introduces a unified framework for referring and grounding in images, utilizing a hybrid region representation and spatial-aware visual sampler...	Ferret differs from related work by unifying referring and grounding in a single framework, supporting diverse region inputs (points, boxes, shapes),...
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-mod...	The paper introduces Monkey, a method to enhance Large Multimodal Models (LMMs) by addressing high-resolution image processing and detailed scene unde...	Monkey differs from related work by introducing a patch-based processing module with sliding window and LoRA adjustments for efficient high-resolution...
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices	MobileVLM introduces mobile-optimized multimodal vision language models with efficient cross-modality interaction via a lightweight projector, achievi...	MobileVLM differs from prior work by emphasizing mobile-specific optimizations (e.g., efficient projector, compact language models) and achieving comp...
A Long Way to Go: Investigating Length Correlations in RLHF	The paper highlights that optimizing response length is a critical, underappreciated factor in RLHF, demonstrating that length-based rewards can repli...	This work differs from related work by systematically analyzing length as a core feature in RLHF, challenging the assumption that length increases are...
UReader: Universal OCR-free Visually-situated Language Understanding with Multim...	URender introduces an OCR-free approach for visually-situated language understanding using a Multimodal Large Language Model (MLLM). Key contributions...	This work differs from related work by eliminating the need for domain-specific pretraining and fine-tuning, leveraging a minimal parameter fine-tunin...
Learning Interactive Real-World Simulators	The paper introduces UniSim, a universal simulator that integrates diverse datasets (image, video, robotics, navigation) to simulate real-world intera...	Unlike prior work focused on domain-specific simulations or limited data, UniSim unifies heterogeneous datasets in a conditional video generation fram...
GLaMM: Pixel Grounding Large Multimodal Model	GLaMM introduces the first model capable of generating natural language responses with object segmentation masks, enabling visually grounded conversat...	GLaMM unifies region understanding, pixel-level grounding, and conversational abilities through end-to-end training, differing from prior works that e...
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation	This paper introduces AssistGUI, a novel benchmark for evaluating GUI automation on Windows desktop environments, along with an Actor-Critic Embodied...	Unlike prior work on mobile/web or terminal automation, this paper focuses on desktop GUI automation with a comprehensive benchmark (AssistGUI) and a...
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs	The paper introduces V*, an LLM-guided visual search mechanism to enhance multimodal LLMs (MLLMs) with efficient visual querying, leading to the SEAL...	The work differs from related work by leveraging rich common sense knowledge from LLMs for dynamic visual search, incorporating both top-down feature...
NExT-Chat: An LMM for Chat, Detection and Segmentation	The paper introduces the pix2emb paradigm for region-level visual understanding, enabling multiple location formats (bounding boxes, masks) through em...	Unlike prior pix2seq methods limited to discrete coordinates, pix2emb uses embeddings decoded into diverse formats via specialized decoders. This arch...
Octopus: Embodied Vision-Language Programmer from Environmental Feedback	Introduces Octopus, an embodied vision-language programmer that bridges high-level planning and real-world manipulation via executable code generation...	Octopus differs by integrating code generation as a medium between planning and manipulation, addressing gaps in prior works that either output low-le...
SteP: Stacked LLM Policies for Web Actions	SteP introduces a dynamic policy composition framework for web tasks using stacked LLM policies, addressing challenges in handling combinatorial web i...	SteP differs from related work by enabling dynamic policy invocation and stacking, allowing adaptive control over task complexity. Unlike static hiera...
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language P...	The paper introduces ViLa, a novel robotic planning framework that integrates vision-language models (VLMs) for grounded task planning. It emphasizes...	Unlike prior work relying on external affordance models or LLMs with limited environmental grounding, ViLa unifies vision and language processing with...
A Zero-Shot Language Agent for Computer Control with Structured Reflection	The paper introduces a zero-shot language agent for computer control that leverages structured reflection to autonomously learn from mistakes without...	This work differs from related work by eliminating the reliance on expert traces or additional screen information, instead using structured self-refle...
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones	Introduces TinyGPT-V, an efficient multimodal large language model (MLLM) with a compact architecture, enabling low-resource training (24GB) and infer...	Differently from prior MLLMs relying on large backbones (e.g., LLaMA2-7B, Vicuna-13B), TinyGPT-V uses a smaller Phi-2 backbone with optimized architec...
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API	The paper introduces a multimodal model for grounding natural language instructions in UI screenshots, leveraging a visual encoder and language decode...	This work differs by integrating reinforcement learning with visual and language components for spatial decoding, proposing a metadata-free grounding...
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models	The paper introduces LLaVA-Grounding, a model that combines visual chat and grounding capabilities. It creates a new dataset (GVC) for grounded visual...	This work differs from related work by integrating chat and grounding tasks into a single model, creating a dedicated dataset and benchmark for ground...
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots	Habitat 3.0 introduces a simulation platform for collaborative human-robot tasks in home environments, focusing on accurate humanoid simulation, human...	Habitat 3.0 differs from related work by integrating humanoid and robot simulation in the same environment, supporting HITL control via VR/mouse/keybo...
WebWISE: Web Interface Control and Sequential Exploration with Large Language Mo...	The paper introduces a method for web interface control using LLMs with in-context learning, leveraging filtered DOM elements as observations and sequ...	WebWISE differs from related work by using DOM observations and in-context learning with minimal examples (single manual or auto-generated example) in...
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversat...	The paper addresses UI task challenges in Vision-Language Models (VLMs) by generating a large-scale conversational dataset without human annotations....	Unlike prior work requiring human annotations, this study generates UI training data automatically using machine conversations, enabling VLMs to handl...
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem	The paper introduces the AIOS-Agent ecosystem, positioning Large Language Models (LLMs) as the core of an intelligent operating system (AIOS) and agen...	This work differs from related work by proposing a conceptual framework where LLMs function as the system-level OS (LLMOS), enabling agents as applica...
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception	The paper introduces MP5, a multi-modal embodied system for open-ended tasks in Minecraft, emphasizing active perception, modular architecture, and co...	MP5 differs from prior work by introducing a modular architecture with active perception and context-aware planning, addressing limitations of previou...
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE	The paper introduces Octavius, a framework addressing task interference in Multimodal Large Language Models (MLLMs) by integrating Mixture-of-Experts...	The work combines MoE with LoRA to create a novel decoder (LoRA-MoE) for MLLMs, addressing task interference through specialized learning paths. This...
An Embodied Generalist Agent in 3D World	The paper introduces LEO, an embodied multi-modal generalist agent designed for 3D world interaction, addressing limitations in 3D perception, groundi...	LEO extends beyond existing 2D-focused generalist models by introducing a unified 3D vision-language-action architecture, a novel 3D dataset with obje...
See and Think: Embodied Agent in Virtual Environment	The paper introduces STEVE, an embodied agent combining vision perception, language instruction, and code action for Minecraft. It proposes a multi-mo...	STEVE differs from prior work by integrating vision perception with LLM-based reasoning and code action generation, enabling more precise and autonomo...
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Wo...	The paper introduces Steve-Eye, a multimodal model integrating LLMs with visual perception for open-world embodied agents. It addresses limitations in...	Unlike prior work focusing on text-based interactions, Steve-Eye integrates visual perception with LLMs, introduces a large-scale open-world instructi...
GROOT: Learning to Follow Instructions by Watching Gameplay Videos	This work introduces GROOT, a novel agent that learns to follow open-ended instructions by watching gameplay videos, addressing limitations in existin...	GROOT differs from prior work by using reference gameplay videos as goal specifications instead of text or static images, enabling self-imitation lear...
UINav: A Practical Approach to Train On-Device Automation Agents	UINav introduces a demonstration-based approach for training on-device UI automation agents with high accuracy using minimal human demonstrations. It...	UINav differs from prior work by combining demonstration-based training with a referee model for real-time feedback and data augmentation, achieving h...
Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing	The paper introduces DroidAgent, an autonomous GUI testing agent for Android that leverages Large Language Models (LLMs) and memory mechanisms for int...	This work differs from related work by integrating LLMs with memory systems to enable high-level semantic testing, moving beyond traditional code cove...
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM...	The paper introduces VisionTasker, a two-stage framework for mobile task automation combining vision-based UI understanding and LLM-driven step-by-ste...	VisionTasker differs from related work by replacing view hierarchies with vision-based UI interpretation and integrating LLMs for dynamic, step-by-ste...
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Alig...	ReMax introduces a reinforcement learning method tailored for aligning large language models (LLMs) by leveraging RLHF properties (fast simulation, de...	ReMax differs from prior work by exploiting RLHF-specific properties and using a greedy baseline in REINFORCE for variance reduction, eliminating the...
An In-depth Look at Gemini's Language Abilities	The paper provides a third-party, reproducible comparison of Google's Gemini and OpenAI's GPT models across 10 language tasks, identifies performance...	This work differs from related work by offering an independent, transparent evaluation of Gemini's language capabilities through reproducible experime...
Creative Agents: Empowering Agents with Imagination for Creative Tasks	The paper introduces creative agents enhanced with an 'imaginator' component to generate task outcomes from language instructions, enabling novel solu...	This work differs from related work by integrating an imaginator for generating task-specific imaginations, enabling creativity in agents. It also int...
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Mod...	The paper addresses underspecification in vision-language tasks by introducing RepARe, a framework that rephrases and augments questions with visually...	Unlike prior work focusing on image captioning or CoT reasoning, RepARe explicitly leverages visual grounding in question rephrasing, utilizing LVLMs...
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on...	The paper introduces a new benchmark (CompWoB) for evaluating compositional web automation tasks, highlights the limitations of language model agents...	This work differs from related work by focusing on compositional task compositionality rather than isolated tasks, introducing a new benchmark (CompWo...
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model...	The paper introduces PCA-EVAL, a benchmark for evaluating embodied decision-making in perception, cognition, and action. It proposes HOLMES, a multi-a...	This work differs by introducing PCA-EVAL as a comprehensive benchmark and HOLMES as a multi-agent framework that directly integrates MLLMs for end-to...
From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models	The paper investigates the effectiveness of different visual encoders in MLLMs, highlighting the advantages of CLIP's shallow layers for fine-grained...	This work differs from related work by analyzing the role of visual encoder depth and introducing COMM, a multi-level feature merging architecture tha...
"What's important here?": Opportunities and Challenges of Using LLMs in Retrievi...	The paper investigates LLMs' ability to retrieve important UI elements from web pages based on user queries, focusing on prompting strategies (example...	Unlike prior works focused on autonomous web navigation, this study decomposes the problem into atomic operations, analyzing how prompting strategies...
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Ma...	The paper introduces SPOC, a transformer-based architecture that imitates shortest-path planners in simulation to enable real-world navigation and man...	This work differs by using shortest-path expert trajectories instead of human demonstrations or dense rewards, combining a transformer architecture wi...
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models	PG-Video-LLaVA introduces the first video-based Large Multimodal Model (LMM) with pixel-level grounding capabilities, integrating audio cues via trans...	This work differs from related work by introducing pixel-level visual grounding in videos, integrating audio context through transcription, proposing...
InfMLLM: A Unified Framework for Visual-Language Tasks	The paper introduces InfMLLM, a unified framework for visual-language tasks, employing a three-stage training scheme (lightweight alignment pretrainin...	InfMLLM differs from prior MLLMs by introducing a progressive training strategy with stage-specific objectives and a novel pool-adapter architecture t...
Vision-Based Mobile App GUI Testing: A Survey	The paper provides a comprehensive survey of vision-based mobile app GUI testing techniques, highlighting their advantages over traditional code/layou...	This work differs from related work by offering a holistic survey of 271 papers (92 vision-based), systematically categorizing GUI testing topics, and...
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld	The paper introduces EMMA, an Embodied Multi-Modal Agent trained via cross-modality imitation learning between a text-world LLM and a visual-world VLM...	This work differs from related work by combining cross-modal distillation from an LLM expert with a novel DAgger-DPO algorithm, enabling embodied VLM...
Large Language Models as Generalizable Policies for Embodied Tasks	The paper introduces LLaRP, a method that adapts large language models (LLMs) as generalizable policies for embodied visual tasks through reinforcemen...	LLaRP differs from prior work by using reinforcement learning directly on pre-trained LLMs for embodied tasks without requiring task-specific training...
CoPAL: Corrective Planning of Robot Actions with Large Language Models	The paper introduces CoPAL, a hierarchical architecture for robot task planning that integrates multi-level feedback loops with Large Language Models...	This work differs from related work by proposing a closed-loop task planning architecture (CoPAL) that systematically integrates geometric constraints...
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasonin...	LL3DA introduces a novel approach for 3D scene understanding, reasoning, and planning by directly processing point cloud inputs, enabling interaction...	LL3DA differs from prior work by directly encoding 3D point clouds instead of using multi-view projections, employing an attention mechanism to integr...
Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraint...	Ctrl-Room introduces a two-stage framework for text-driven 3D room generation with layout constraints, enabling structurally plausible layouts, high-f...	Unlike prior methods that struggle with global layout consistency or lack editability, Ctrl-Room decouples layout and appearance generation, employs s...
Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation	The paper introduces Zero-shot Interactive Personalized Object Navigation (ZIPON), a task that combines zero-shot object navigation with natural langu...	This work differs from related work by introducing ORION, a framework that leverages LLMs for sequential decision-making in interactive navigation, in...
Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Langu...	The paper introduces MERLIM, a multi-modal benchmark for evaluating Instruction Tuning Large Vision and Language Models (IT-LVLMs) on fundamental comp...	This work differs from related work by introducing MERLIM, a standardized benchmark specifically designed to evaluate IT-LVLMs for fundamental vision...
Improving web element localization by using a large language model	The paper introduces VON Similo LLM, an enhanced web element localization approach leveraging Large Language Models (LLMs) to improve accuracy by inco...	Unlike prior post-repair approaches (e.g., WATER, GPT-2-based methods) that rely on attribute comparisons or heuristic repairs, this work integrates L...
Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Langu...	The paper introduces Co-NavGPT, a framework that integrates Vision Language Models (VLMs) as global planners for multi-robot cooperative visual target...	Co-NavGPT differs from related work by combining VLMs with multi-robot systems for semantic reasoning and global planning, enabling efficient collabor...
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts	The paper introduces M3DBench, a comprehensive multi-modal 3D instruction-following dataset that supports diverse 3D tasks, interleaved visual prompts...	Unlike prior 3D datasets focused on specific tasks (e.g., object detection, navigation), M3DBench unifies region and scene-level 3D tasks with multi-m...
Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficientl...	The paper introduces RLFP framework with FAC algorithm, leveraging foundation models for policy, value, and reward priors to enhance sample efficiency...	This work differs from related work by integrating foundation priors (policy, value, success-reward) into RL, enabling automatic reward functions and...
Large Language Model as a Policy Teacher for Training Reinforcement Learning Age...	This paper introduces a framework that combines Large Language Models (LLMs) with reinforcement learning (RL) to train specialized student agents. The...	This work differs from related work by proposing a knowledge distillation framework where an LLM acts as a teacher to guide a specialized RL student a...
InstructDET: Diversifying Referring Object Detection with Generalized Instructio...	The paper introduces InstructDET, a data-centric approach to referring object detection (ROD) that generates diverse instructions for object localizat...	This work differs from related work by (1) generating diverse, human-like instructions via foundation models rather than manual annotation, (2) creati...
GSVA: Generalized Segmentation via Multimodal Large Language Models	The paper introduces GSVA, a novel approach to Generalized Referring Expression Segmentation (GRES) that addresses multi-target and empty-target scena...	GSVA differs from prior work by introducing the [REJ] token for explicit rejection of empty targets and enabling simultaneous segmentation of multiple...
Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI	The paper introduces a scene-driven multimodal knowledge graph (Scene-MMKG) construction method for embodied AI, combining knowledge engineering with...	The work differs from related work by proposing a unified scene knowledge injection framework that integrates conventional knowledge bases with large...
WebVLN: Vision-and-Language Navigation on Websites	Introduces the WebVLN task for navigating websites using natural language instructions, incorporates HTML content alongside visual and textual data, a...	Differs from related work by explicitly incorporating HTML content (non-visual web-specific data) and introducing a new task and benchmark focused on...

2024: Q1

Title	Summary	Contributions
GPT-4V(ision) is a Generalist Web Agent, if Grounded	This work introduces SEEACT, a generalist web agent leveraging large multimodal models (LMMs) like GPT-4V for integrated visual understanding and web...	This work differs from related work by proposing a novel grounding strategy that integrates HTML structure and visuals for web agents, rather than rel...
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents	The paper introduces SeeClick, a visual GUI agent that automates tasks using screenshots instead of structured data, addresses the GUI grounding chall...	SeeClick differs from prior work by eliminating reliance on structured text (e.g., HTML) and GUI metadata, leveraging LVLMs for direct screenshot-base...
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks	Introduces VisualWebArena, a benchmark for evaluating multimodal agents on visually grounded web tasks, emphasizing integration of visual and textual...	VisualWebArena fills the gap in evaluating multimodal agents on visually grounded tasks, offering a comprehensive benchmark with real-world tasks and...
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception	The paper introduces Mobile-Agent, a multi-modal agent that uses visual perception to operate mobile apps without relying on XML metadata. It presents...	Mobile-Agent differs from prior work by employing a vision-centric approach without requiring XML or system metadata, introducing Mobile-Eval as a new...
DeepSeek-VL: Towards Real-World Vision-Language Understanding	DeepSeek-VL introduces a hybrid vision encoder for efficient high-resolution image processing, a comprehensive real-world dataset with diverse modalit...	DeepSeek-VL differentiates from prior work by combining vision and language pretraining with a hybrid vision encoder, addressing limitations of projec...
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models	This paper introduces WebVoyager, a multimodal web agent leveraging large multimodal models (LMMs) to interact with real-world websites end-to-end. It...	WebVoyager differs from prior work by enabling real-world web navigation through multimodal inputs (screenshots and text), utilizing a novel benchmark...
UFO: A UI-Focused Agent for Windows OS Interaction	UFO introduces a dual-agent framework for Windows OS interaction, leveraging GPT-Vision to analyze GUIs and controls, enabling automated task executio...	Unlike prior LLM agents focused on general task planning or code-centric approaches, UFO specifically targets Windows OS UI automation with a dual-age...
ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models	The paper introduces a synthetic dataset (ALLaVA) generated via GPT4V to enhance lite vision-language models (VLMs), achieving performance comparable...	This work differs by focusing on high-quality synthetic data generation (ALLaVA) to bridge performance gaps between lite and large VLMs, rather than r...
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement	The paper introduces OS-Copilot, a framework for building generalist computer agents capable of interacting with diverse OS elements. It presents FRID...	This work differs from related work by focusing on generalist agents for OS interactions, introducing self-improvement mechanisms, and demonstrating s...
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue	The paper introduces WEBLINX, a large-scale benchmark for conversational web navigation, and proposes a retrieval-inspired model to address the challe...	This work differs from related work by introducing a novel benchmark (WEBLINX) and a retrieval-inspired architecture tailored for web navigation tasks...
Android in the Zoo: Chain-of-Action-Thought for GUI Agents	This work introduces Chain-of-Action-Thought (CoAT) for GUI agents, emphasizing semantic reasoning through screen context, action thinking, targets, a...	Unlike prior works focusing solely on coordinate-based actions or separating element recognition from action inference, CoAT integrates semantic reaso...
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language...	SPHINX-X introduces a scalable family of multi-modal large language models (MLLMs) with architectural optimizations, a comprehensive multimodal datase...	This work differs by scaling data and parameters, modifying the SPHINX architecture with skip tokens and one-stage training, and curating a diverse da...
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous A...	Introduces OmniACT, the first dataset and benchmark for evaluating autonomous agents' ability to generate executable scripts for both desktop and web...	OmniACT differs from prior work by combining desktop and web tasks, requiring executable script generation rather than just action prediction, and emp...
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs	Identifies systematic visual shortcomings in multimodal LLMs (MLLMs) stemming from CLIP-based visual encoders, introduces the MMVP benchmark using CLI...	Systematically exposes the limitations of CLIP-based visual encoders in MLLMs, highlights the disconnect between CLIP's embedding space and vision-onl...
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document	TextMonkey introduces an OCR-Free large multimodal model for document understanding, enhancing text-centric tasks through Shifted Window Attention wit...	Unlike OCR-Model-Driven methods, TextMonkey eliminates reliance on external OCR systems by integrating text detection, layout understanding, and visua...
Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Se...	The paper provides a comprehensive survey of Personal LLM Agents, focusing on their architecture, capabilities, efficiency, and security. It highlight...	This work differs from related work by offering a systematic survey and analysis of Personal LLM Agents, emphasizing their integration with personal d...
ScreenAI: A Vision-Language Model for UI and Infographics Understanding	ScreenAI introduces a vision-language model specialized in understanding UIs and infographics, combining PaLI architecture with Pix2Struct's flexible...	ScreenAI differs from related work by unifying UI and infographic understanding through a novel visual language model architecture, leveraging a hybri...
Cradle: Empowering Foundation Agents Towards General Computer Control	The paper introduces the General Computer Control (GCC) setting to standardize interaction with software via screenshots and low-level actions, propos...	This work differs from related work by introducing a standardized interface (screenshots + keyboard/mouse actions), a modular framework (Cradle) that...
Common 7B Language Models Already Possess Strong Math Capabilities	The paper demonstrates that LLaMA-2 7B models possess strong mathematical capabilities without specialized pre-training, achieving high accuracy on GS...	This work differs from related work by focusing on enhancing small language models' mathematical abilities through efficient SFT and synthetic data sc...
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?	The paper introduces WorkArena, a benchmark for evaluating web agents on enterprise software tasks, and BrowserGym, a unified environment for agent ev...	WorkArena focuses on enterprise software workflows (ServiceNow) unlike prior benchmarks (e.g., WebArena, MiniWoB), and introduces BrowserGym as a flex...
Understanding the planning of LLM agents: A survey	The paper provides the first systematic taxonomy of LLM-based agent planning, categorizing existing works into Task Decomposition, Plan Selection, Ext...	This work differs from related work by offering the first comprehensive survey and structured taxonomy of LLM-Agent planning, synthesizing existing re...
Large Language Model based Multi-Agents: A Survey of Progress and Challenges	The paper provides a comprehensive survey of LLM-based multi-agent systems, focusing on their applications in complex problem-solving and world simula...	This work differentiates itself by offering a holistic survey of LLM-based multi-agent systems, emphasizing collective intelligence, inter-agent inter...
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding	The paper introduces Unified Structure Learning for OCR-free Visual Document Understanding, emphasizing structure-aware parsing tasks and multi-graine...	Unlike prior OCR-dependent or limited-domain approaches, this work unifies structure learning across diverse text-rich images (documents, webpages, ta...
ScreenAgent: A Vision Language Model-driven Computer Control Agent	The paper introduces ScreenAgent, a Vision Language Model (VLM)-driven computer control agent that interacts with real computer screens via GUI manipu...	Unlike prior work, ScreenAgent combines VLM with a task-specific control pipeline and dataset for real-world GUI interaction, achieving superior UI po...
LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model	The paper introduces LLaVA-Phi, a compact multi-modal assistant leveraging the Phi-2 small language model for efficient visual-dialogue tasks. It demo...	LLaVA-Phi differs from related work by combining a small language model (Phi-2) with LLaVA-1.5 training methodology, achieving efficiency without sacr...
AgentStudio: A Toolkit for Building General Virtual Agents	AgentStudio introduces a comprehensive toolkit for building general virtual agents, featuring a lightweight interactive environment with generic obser...	Unlike prior domain-specific simulators (e.g., WebShop, AndroidEnv) or narrow benchmarks, AgentStudio provides a generic, real-world compatible enviro...
TravelPlanner: A Benchmark for Real-World Planning with Language Agents	Introduces TravelPlanner, a benchmark for evaluating language agents in complex real-world planning tasks. Highlights challenges in multi-constraint p...	TravelPlanner differs from prior work by focusing on real-world, multi-constraint travel planning scenarios with a rich sandbox environment, diverse t...
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Eng...	The paper introduces Design2Code, the first real-world benchmark for multimodal code generation in front-end engineering. It evaluates current MLLMs o...	This work differs from related studies by creating the first benchmark specifically for visual-to-code generation, providing detailed evaluation metri...
Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dat...	The paper introduces WebSight, a large-scale synthetic dataset of 2 million HTML-code-to-screenshot pairs, and Sightseer, a vision-language model with...	Unlike prior work that relied on small datasets or heuristic-based approaches, this work introduces WebSight, a synthetic dataset with 2.5x larger siz...
Large Multimodal Agents: A Survey	The paper provides a systematic review of Large Multimodal Agents (LMAs), categorizing research into four types, compiling evaluation methodologies, p...	This work differs from related work by offering a comprehensive survey of LMAs, unifying diverse research categories, standardizing evaluation framewo...
Agent AI: Surveying the Horizons of Multimodal Interaction	The paper defines 'Agent AI' as a framework for embodied multimodal systems that integrate visual, linguistic, and environmental data to enable contex...	This work differs from related research by systematically framing Agent AI as a class of systems that prioritize environmental grounding and multimoda...
MobileAgent: enhancing mobile control via human-machine interaction and SOP inte...	The paper introduces MobileAgent, an LLM-based agent that enhances mobile device control through human-machine interaction and integration of Standard...	This work differs from related work by integrating SOP information into in-context learning for better task comprehension and introducing interactive...
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation	The paper introduces CoCo-Agent, a multimodal large language model (MLLM) agent for smartphone GUI automation, focusing on comprehensive environment p...	Unlike prior work relying on strong pre-trained models or black-box APIs, CoCo-Agent introduces a trainable architecture with CEP and CAP to systemati...
3D-VLA: A 3D Vision-Language-Action Generative World Model	The paper introduces 3D-VLA, a generative world model that integrates 3D perception, reasoning, and action for embodied agents. It addresses limitatio...	3D-VLA differs from prior work by explicitly modeling 3D dynamics and using 3D features for action generation, introducing a novel dataset for 3D embo...
ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Ch...	The paper introduces ChartX, a comprehensive multi-modal benchmark for evaluating chart understanding capabilities, and ChartVLM, a new model designed...	This work differs from related work by introducing ChartX as a comprehensive benchmark with diverse chart types and tasks, and ChartVLM with a cascade...
Understanding the Weakness of Large Language Model Agents within a Complex Andro...	The paper identifies critical weaknesses of LLM agents in complex Android environments, introduces AndroidArena as a benchmark for evaluating LLM agen...	This work differs from related research by introducing AndroidArena, the first benchmark specifically designed to evaluate LLM agents in complex OS en...
Exploring Large Language Model based Intelligent Agents: Definitions, Methods, a...	The paper provides a comprehensive survey of LLM-based intelligent agents, covering definitions, research frameworks, foundational components (plannin...	This work differs from related research by offering an in-depth synthesis of LLM-based agents' methodologies, multi-agent system integration strategie...
SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning...	Introduces SkyEyeGPT, a unified multi-modal large language model for remote sensing (RS) vision-language tasks. Develops a large-scale RS instruction-...	Unlike prior RS models like RSGPT that handle single tasks with separate models, SkyEyeGPT unifies diverse RS tasks through instruction tuning and a s...
Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models	The paper addresses limitations in Multimodal Large Language Models (MLLMs) by introducing P2G, a framework for plug-and-play grounding of visual reas...	The work differs from related work by leveraging external agents for on-the-fly grounding instead of relying on training data or existing modules, and...
CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning	Introduces Chain of Manipulations (CoM) for step-by-step visual reasoning in VLMs, proposes the CogCoM model with 17B parameters, annotates 6K graphic...	Differ from prior work by emphasizing step-by-step visual manipulation reasoning (CoM) instead of direct alignment, introducing a multi-turn multi-ima...
On the Multi-turn Instruction Following for Conversational Web Agents	This work introduces the Conversational Web Navigation task, a novel framework (Self-MAP) for handling multi-turn interactions with users and environm...	Unlike prior work focusing on single-turn web navigation tasks, this paper emphasizes multi-turn conversational interactions, introducing a dataset (M...
Tur[k]ingBench: A Challenge Benchmark for Web Agents	Introduces TurkingBench, a benchmark for web-based agents using real HTML pages from crowdsourcing platforms, evaluates multi-modal models on complex...	TurkingBench differs from prior work by using naturally occurring crowdsourcing HTML pages rather than synthesized environments, offering more realist...
DocLLM: A layout-aware generative language model for multimodal document underst...	DocLLM introduces a layout-aware generative language model for multimodal document understanding, focusing on spatial-textual cross-alignment through...	DocLLM differs from related work by avoiding expensive image encoders, focusing exclusively on bounding box-based spatial layouts, and introducing dis...
Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation	The paper introduces a hierarchical auto-organizing system (HAS) for multi-agent navigation in dynamic environments like Minecraft. Key contributions...	Unlike prior work focusing on single-agent reinforcement learning or pre-trained LLMs for task planning, HAS introduces a novel hierarchical architect...
Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models vi...	This paper addresses object hallucination in Large Vision-Language Models (LVLMs) by introducing CLIP-Guided Decoding (CGD), a training-free method th...	Unlike prior work relying on internal model information (e.g., token likelihoods, hidden states) or complex external tools, this work proposes a train...
Dual-View Visual Contextualization for Web Navigation	The paper introduces a method to enhance HTML element representations for web navigation by leveraging dual views (textual and visual context from scr...	Unlike prior work relying solely on HTML documents or simplified representations, this work integrates visual context from webpage screenshots, contex...
BAGEL: Bootstrapping Agents by Guiding Exploration with Language	BAGEL introduces a method to bootstrap language model agents for digital environment interaction without human supervision. It uses iterative round-tr...	BAGEL differs from prior work by eliminating the need for human demonstrations or reward functions, instead leveraging synthetic demonstrations genera...
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simula...	Introduces MineDreamer, an embodied agent in Minecraft that uses Chain-of-Imagination (CoI) to translate natural language instructions into precise vi...	Proposes CoI mechanism to address sequential instruction-following challenges by breaking tasks into stages and generating state-aware visual prompts,...
Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning	Scene-LLM integrates 3D visual understanding with LLMs for interactive indoor environments, combining scene-level and ego-centric 3D information. It i...	Scene-LLM differs from prior work by jointly leveraging scene-level and ego-centric 3D information, employing a hybrid feature representation for dyna...
Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipane...	The paper introduces MultipanelVQA, a novel benchmark for evaluating Multimodal Large Language Models (MLLMs) on multipanel image understanding. It hi...	This work differs from related studies by focusing specifically on multipanel image understanding, introducing a synthetic benchmark to isolate layout...
VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysi...	The paper introduces VHM, a vision language model tailored for remote sensing image analysis, emphasizing versatility and honesty. It proposes two nov...	Unlike prior work focused on factual questions and sparse captions, VHM introduces comprehensive captions (VersaD) and deceptive question datasets (Hn...
ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grai...	The paper addresses visual grounding issues in LVLMs by introducing ViGoR, a framework that leverages fine-grained reward modeling with human evaluati...	ViGoR differs from related work by focusing on reward modeling with fine-grained human feedback to enhance visual grounding, rather than relying solel...
LLM Multi-Agent Systems: Challenges and Open Problems	The paper identifies key challenges in multi-agent systems (MAS) involving LLMs, such as optimizing task allocation, fostering collaborative reasoning...	This work differs from related studies by focusing on systemic challenges in multi-agent collaboration (e.g., context management, memory, and layered...
The Revolution of Multimodal Large Language Models: A Survey	The paper provides a comprehensive survey of Multimodal Large Language Models (MLLMs), focusing on architectural designs, multimodal alignment strateg...	This work differentiates from related studies by offering an exhaustive review of visual-based MLLMs, emphasizing their integration with visual modali...
DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exe...	DoraemonGPT introduces a novel LLM-driven video agent for dynamic scene understanding, addressing spatial-temporal reasoning, large planning spaces, a...	Unlike prior multimodal systems, DoraemonGPT specifically targets dynamic video tasks with a modular architecture combining symbolic memory, sub-task...
AI Assistance for UX: A Literature Review Through Human-Centered AI	The paper provides a systematic literature review of 359 papers on AI-enabled UX tools, highlighting gaps in empathy-building, multi-screen user exper...	This work differs from related studies by focusing on UX practitioners' unmet needs through a Human-Centered AI lens, mapping findings onto the Double...
Embodied LLM Agents Learn to Cooperate in Organized Teams	The paper introduces a framework for organizing LLM agents using prompt-based structures to enhance cooperation in multi-agent systems. It emphasizes...	This work differs from related work by proposing a novel multi-LLM-agent architecture with hierarchical organization structures and a Criticize-Reflec...
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend Wha...	The paper introduces the Draw-and-Understand framework for integrating visual prompting into MLLMs, supports multiple visual prompt types (points, box...	Unlike prior ROI-based methods that rely on segmentation models or ground truth masks, this work proposes a general architecture adaptable to diverse...
MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments	The paper introduces MEIA, a multimodal embodied agent that integrates visual and linguistic memory for task planning in unknown environments. It prop...	MEIA differs from prior work by combining visual-language memory (MEM) for multimodal scene understanding, enabling real-time environmental grounding...
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction	This paper introduces ShapeLLM, the first 3D Multimodal Large Language Model (LLM) for embodied interaction, combining 3D geometry understanding with...	Unlike prior work focusing on 3D representation learning or vision-language models, ShapeLLM unifies 3D geometry understanding (via ReCon++) with lang...
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worl...	The paper introduces MultiPLY, a multisensory embodied large language model that actively interacts with 3D environments by integrating visual, audio,...	This work differs from related work by enabling active 3D environment interaction through multisensory data integration, introducing object-centric re...
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alig...	The paper introduces GOMA, a framework for proactive verbal communication in embodied agents to align mental states toward shared goals. It addresses...	GOMA differs from prior work by explicitly modeling mental state alignment through goal-oriented planning, enabling proactive and contextually relevan...
Will GPT-4 Run DOOM?	Demonstrates GPT-4's ability to play Doom through reasoning and planning without reinforcement learning, evaluates LLM capabilities in complex environ...	This work differs from related work by demonstrating LLM-based planning in a video game without reinforcement learning or fine-tuning, using Doom as a...
ChatterBox: Multi-round Multimodal Referring and Grounding	The paper introduces the MRG task for instance-level multimodal dialogues, a new benchmark (CB-300K) with multi-round dialogue and complex spatial cha...	This work differs by addressing multi-round dialogues and complex spatial relationships in MRG, which existing benchmarks lack. It introduces CB-300K...
GroundingGPT:Language Enhanced Multi-modal Grounding Model	GroundingGPT introduces a language-enhanced multi-modal grounding model focused on fine-grained understanding of local information across modalities (...	GroundingGPT differs from prior work by explicitly targeting fine-grained multi-modal grounding through modality-specific adapters, coordinate-based s...
LCV2: An Efficient Pretraining-Free Framework for Grounded Visual Question Answe...	The paper introduces LCV2, a modular, pretraining-free framework for grounded visual question answering (VQA) that leverages a frozen large language m...	LCV2 differs from related work by eliminating the need for pre-training, using a modular architecture with a frozen LLM as a mediator, and enabling ef...
Enhancing Vision-Language Pre-training with Rich Supervisions	The paper introduces S4, a pre-training paradigm for Vision-Language Models (VLMs) using web screenshots and rich supervisions. It leverages HTML elem...	Unlike prior work focusing on image-text pairs or object detection tasks, S4 innovates by utilizing web screenshots with structured HTML annotations a...
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Vis...	Griffon v2 enhances multimodal perception by addressing image resolution limitations in LVLMs, enabling high-resolution scaling (up to 1K) and visual-...	Unlike prior methods that rely on division-based resolution enhancement or low-resolution image encoding, Griffon v2 employs a lightweight projector t...
Language-guided Skill Learning with Temporal Variational Inference	The paper introduces a framework for language-guided skill discovery using hierarchical variational inference and an auxiliary objective based on Mini...	The work differs from prior methods by integrating LLM-generated semantic segmentation with variational inference, avoiding language-only skill repres...
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modul...	CREMA introduces a modular, parameter-efficient multimodal fusion framework for video-language reasoning, enabling seamless integration of diverse mod...	Unlike prior work, CREMA achieves scalability and efficiency by decoupling modality processing from the LLM, using parameter-efficient modules and a n...
Object-Centric Instruction Augmentation for Robotic Manipulation	The paper introduces the Object-Centric Instruction Augmentation (OCI) framework to enhance robotic manipulation by integrating object positions into...	Unlike prior work focusing on task planning or vision-language model integration, this work emphasizes augmenting language instructions with explicit...

2024: Q2

Title	Summary	Contributions
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with O...	The paper introduces three key improvements: (1) a strong vision encoder with continuous learning for better visual understanding, (2) dynamic high-re...	InternVL 1.5 addresses gaps in parameter scale, image resolution, and multilingual capability compared to proprietary models by integrating a continuo...
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer En...	The paper introduces OSWorld, a real computer environment for evaluating multimodal agents in open-ended tasks across multiple operating systems. It p...	OSWorld differs from prior work by providing a scalable, real-world interactive environment and benchmark that captures the diversity and complexity o...
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents	The paper introduces AndroidWorld, a dynamic benchmarking environment for autonomous agents on Android, featuring 116 programmatic tasks across 20 rea...	AndroidWorld differs from related work by providing the first comprehensive mobile benchmark with dynamically generated, parameterized tasks across di...
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs	The paper introduces Cambrian-1, a vision-centric multimodal LLM family that evaluates diverse visual representations through visual instruction tunin...	Unlike prior work, Cambrian-1 focuses on vision-centric design and integrates spatial awareness via SVA to enhance visual grounding. It introduces CV-...
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs	Ferret-UI introduces a specialized multimodal large language model (MLLM) for mobile UI understanding, addressing limitations in existing models throu...	This work differs from related work by explicitly addressing UI-specific challenges (e.g., elongated aspect ratios, small objects) through a dual-subi...
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Reso...	The paper introduces InternLM-XComposer2-4KHD, a large vision-language model capable of handling resolutions from 336 pixels to 4K HD. It addresses re...	Unlike prior work that uses fixed resolutions or separate HR/LR encoders, this work proposes dynamic resolution scaling with automatic patch configura...
What matters when building vision-language models?	The paper systematically evaluates critical design choices in vision-language models (VLMs), including architecture, data, and training methods. It id...	This work differs from related research by rigorously ablation-studying key design decisions in VLMs through controlled experiments, providing empiric...
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement...	The paper introduces DigiRL, an autonomous reinforcement learning framework for training device-control agents in real-world GUI environments. Key con...	DigiRL differs from prior work by combining autonomous offline-to-online RL with pre-trained VLMs, addressing real-world stochasticity and non-station...
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via...	The paper introduces a multi-agent architecture (planning, decision, reflection agents) for mobile device operation tasks, addressing navigation chall...	This work differs from related work by proposing a specialized multi-agent collaboration framework tailored for mobile device GUI operations, incorpor...
AutoWebGLM: A Large Language Model-based Web Navigating Agent	The paper introduces AutoWebGLM, a large language model-based web navigating agent designed to overcome challenges in real-world web navigation. Key c...	This work differs from related work by addressing HTML complexity and open-domain task challenges through HTML simplification, curriculum learning, an...
On the Effects of Data Scale on UI Control Agents	The paper investigates the scalability of fine-tuning LLMs for UI control agents, introduces the AndroidControl dataset with high/low-level instructio...	The work introduces AndroidControl, the most diverse UI control dataset with both high/low-level instructions, and provides systematic analysis of how...
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devi...	This work introduces GUI Odyssey, a comprehensive dataset for cross-app GUI navigation, and develops OdysseyAgent, a multimodal navigation agent. It a...	This work differs from related work by focusing on cross-app navigation tasks, which previous datasets and models primarily ignored. It introduces GUI...
GUICourse: From General Vision Language Models to Versatile GUI Agents	The paper introduces GUICourse, a comprehensive suite of datasets (GUIEnv, GUIAct, GUIChat) to enhance Vision Language Models (VLMs) for GUI agent tas...	This work differs from related work by explicitly addressing VLM limitations in OCR, grounding, and GUI-specific knowledge through purpose-built datas...
MANTIS: Interleaved Multi-Image Instruction Tuning	The paper introduces Mantis, a family of large multimodal models trained via instruction tuning on a curated dataset (Mantis-Instruct) to excel in mul...	Unlike prior work that relies on pre-training on large noisy interleaved data, Mantis achieves strong multi-image capabilities through instruction tun...
Autonomous Evaluation and Refinement of Digital Agents	The paper introduces domain-general automatic evaluators for digital agents, demonstrating their effectiveness in improving web navigation and device...	This work differs from related work by proposing automated evaluators that operate without expert demonstrations or supervision, leveraging vision-lan...
From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function	The paper establishes DPO as a token-level Q-learning algorithm within the RLHF framework, bridging the gap between contextual bandit-based DPO and cl...	The work theoretically frames DPO as an inverse Q-learning algorithm in token-level MDPs, resolving discrepancies with classical RLHF by satisfying Be...
OpenVLA: An Open-Source Vision-Language-Action Model	This paper introduces OpenVLA, an open-source vision-language-action (VLA) model trained on 970k real-world robot demonstrations. It combines Llama 2...	OpenVLA differs from prior work by integrating Internet-scale vision-language foundation models with robot demonstrations, enabling generalization to...
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding a...	The paper introduces VisualWebBench, a comprehensive benchmark for evaluating multimodal large language models (MLLMs) in web page understanding and g...	VisualWebBench differs from related work by providing a holistic evaluation framework tailored to web contexts, encompassing perception, comprehension...
A Survey on the Memory Mechanism of Large Language Model based Agents	The paper provides a comprehensive survey on memory mechanisms in LLM-based agents, systematically reviewing existing designs, evaluating their effect...	This work differs from related studies by offering the first holistic review of memory mechanisms in LLM-based agents, abstracting design patterns, an...
Octopus v2: On-device language model for super agent	The paper introduces Octopus v2, an on-device language model with 2B parameters that significantly improves accuracy and reduces latency compared to G...	The work differs by introducing functional tokens for efficient function calling, optimizing latency and accuracy for on-device use, and achieving cos...
WebCanvas: Benchmarking Web Agents in Online Environments	The paper introduces WebCanvas, an online evaluation framework for web agents that addresses the dynamic nature of web environments. Key contributions...	Unlike static benchmarks, WebCanvas focuses on dynamic web environments with real-time task evaluation, introduces a live dataset with intermediate st...
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language...	Ferret-v2 enhances referring and grounding capabilities in MLLMs through high-resolution image processing, multi-granularity visual encoding with DINO...	Unlike prior MLLMs reliant on low-resolution encoders (e.g., CLIP), Ferret-v2 introduces DINOv2 for multi-granularity visual encoding and a three-stag...
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation	LlamaTouch introduces a novel testbed for evaluating mobile UI task automation agents by focusing on faithful and scalable task execution evaluation....	Unlike prior work relying on static datasets or human validation, LlamaTouch introduces a dynamic benchmark that evaluates agents based on traversing...
GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents	Introduces GUI-World, a comprehensive dataset for GUI-oriented multimodal LLM agents, emphasizing dynamic and sequential tasks across diverse GUI scen...	GUI-World addresses gaps in prior work by covering dynamic GUI content, multi-window interactions, and diverse operating environments. It provides a b...
Large Language Models Can Self-Improve At Web Agent Tasks	The paper explores LLM self-improvement for web agent tasks using synthetic training data, achieves a 31% task completion rate improvement on WebArena...	This work differs by focusing on self-improvement through fine-tuning on model-generated data for long-horizon web agent tasks, and by introducing com...
MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents	The paper introduces MobileAgentBench, a benchmark for evaluating mobile LLM agents on Android devices. It addresses the lack of standardized testing...	This work differs from related work by focusing on benchmarking rather than model development. While prior studies explored LLM/MLLM-based agents and...
MMInA: Benchmarking Multihop Multimodal Internet Agents	The paper introduces MMInA, a benchmark for evaluating embodied agents in multihop, multimodal web tasks. Key contributions include evolving real-worl...	MMInA addresses gaps in existing benchmarks by focusing on real-world evolving websites, multihop tasks, and multimodal reasoning. It introduces a hol...
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding	Introduces MuirBench, a comprehensive benchmark for evaluating multi-image understanding in multimodal LLMs, highlighting challenges in spatial-tempor...	MuirBench differs from prior work by providing a broader, more comprehensive assessment of multi-image reasoning capabilities, including 12 diverse ta...
Benchmarking Mobile Device Control Agents across Diverse Configurations	The paper introduces B-MoCA, a novel benchmark for evaluating mobile device control agents across diverse device configurations. It emphasizes tasks m...	Unlike prior work focused on specific agent types or limited task sets, B-MoCA provides a unified benchmark with diverse device configurations and 131...
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundi...	This work introduces the Screen Point-and-Read (ScreenPR) task, a novel GUI-referring task involving screen reading based on user-indicated points. It...	This work differs from related work by focusing on the ScreenPR task, which is underexplored in GUI understanding. It introduces the ToL grounding mec...
MoVA: Adapting Mixture of Vision Experts to Multimodal Context	The paper addresses the limitations of single vision encoders in MLLMs by introducing MoVA, a novel framework that adaptively routes and fuses task-sp...	MoVA differs from prior work by introducing a context-aware expert routing strategy guided by LLMs and a fine-grained MoV-Adapter for task-specific kn...
Octopus: On-device language model for function calling of software APIs	The paper introduces Octopus, an on-device LLM fine-tuned for software API function calling, with techniques like conditional masking and a novel benc...	This work differs by focusing on on-device LLMs for API interactions, introducing conditional masking for format alignment, and creating a specialized...
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Mod...	Groma introduces localized visual tokenization for grounded multimodal understanding, enabling region-level tasks like captioning and grounding. It in...	Groma differs by embedding localization into image tokenization rather than relying on external modules or LLM-based coordinate outputs, enabling unif...
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy	LLaRA introduces a framework for adapting pretrained Vision-Language Models (VLMs) into Vision-Language-Action (VLA) models for robotic control. It ge...	LLaRA differs from related work by focusing on visuomotor instruction tuning for robotics, generating task-specific conversation-style datasets from e...
Efficient Multimodal Large Language Models: A Survey	The paper provides a comprehensive survey of efficient Multimodal Large Language Models (MLLMs), summarizing their development timelines, efficient st...	This work differs from related work by offering a systematic review of efficient MLLMs, highlighting scalability challenges, and proposing future dire...
Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single M...	The paper introduces STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied agents, addressing limitations in multi-LLM syst...	STEVE-2 differs from prior work by distilling complex multi-agent systems into a single model with hierarchical architecture, enabling dynamic adaptat...
Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vi...	The paper identifies 'blind tokens' in LVLMs that cause hallucinations due to excessive attention on irrelevant image regions. It introduces AvisC, a...	Unlike prior output-level methods that contrast whole-image representations or use external models, AvisC focuses on internal attention patterns to id...
VideoGUI: A Benchmark for GUI Automation from Instructional Videos	Introduces VideoGUI, a multi-modal benchmark for advanced GUI automation tasks derived from instructional videos, emphasizing visual-centric tasks, hi...	Differently from prior benchmarks limited to simple text-instruction tasks, VideoGUI targets complex visual-centric GUI tasks via instructional videos...
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents	Wilbur introduces adaptive in-context learning for web agents, combining a differentiable ranking model, instruction synthesis, and an intelligent bac...	Wilbur differs from prior work by introducing backtracking to recover from mistakes, synthesizing both positive and negative task demonstrations, and...
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Langua...	The paper presents TextHawk, a Multimodal Large Language Model (MLLM) specifically designed for document-oriented tasks, featuring four novel componen...	TextHawk differs from prior work by introducing a novel architecture with ReSA, SPEs, QPN, and MLCA for efficient fine-grained document perception, al...
Search Beyond Queries: Training Smaller Language Models for Web Interactions via...	The paper introduces GLAINTEL, a novel agent for intelligent web navigation that leverages reinforcement learning and smaller language models to addre...	GLAINTEL differs from related work by focusing on smaller language models trained via reinforcement learning and unsupervised methods, demonstrating s...
Training a Vision Language Model as Smartphone Assistant	This research introduces a Vision Language Model (VLM) designed to control mobile devices through UI interactions, leveraging sequences of past screen...	The work differs from prior methods by directly interacting with the UI through visual inputs and action sequences, rather than relying on API calls o...
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Mul...	The paper introduces Web2Code, a large-scale dataset and evaluation framework for webpage-to-code generation tasks, addressing the gap in MLLMs' abili...	Unlike existing datasets that lack instruction information or focus on general visual tasks, Web2Code integrates instruction tuning, diverse QA pairs,...
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasonin...	GuardAgent introduces a novel guardrail framework for LLM agents, enabling dynamic safety checks through knowledge-enabled reasoning and code executio...	Unlike prior guardrails focused on input/output moderation, GuardAgent introduces a task-planning and code-execution framework for safeguarding LLM ag...
Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM...	The paper introduces GovSim, a novel simulation platform to study cooperative decision-making in LLM agents, highlighting challenges in achieving sust...	This work differs by focusing on multi-agent cooperation in dynamic, realistic scenarios (GovSim) rather than single-agent benchmarks. It introduces a...
WebSuite: Systematically Evaluating Why Web Agents Fail	The paper introduces WebSuite, a diagnostic benchmark for web agents that systematically evaluates failure patterns by categorizing web actions into a...	Unlike existing benchmarks that only measure task success or failure, WebSuite introduces a taxonomy of web actions and an extensible benchmark suite...
Latent State Estimation Helps UI Agents to Reason	The paper explores how LLMs can estimate latent states in UI agents through zero-shot prompting, demonstrating significant improvements in task comple...	This work differs from related work by formalizing the use of pre-trained LLMs to estimate latent states in a textual space without task-specific fine...
Mitigating Object Hallucinations in Large Vision-Language Models with Assembly o...	The paper addresses object hallucinations in Large Vision-Language Models (LVLMs) by identifying attention deficiency as a root cause. It introduces A...	This work differs from related work by focusing on attention mechanism design rather than instruction tuning or post-hoc revisers. It proposes a dual-...
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LL...	The paper introduces a new learning paradigm called 'list items one by one' to enhance Multimodal Large Language Models' (MLLMs) ability to understand...	This work differs from related studies by focusing on a novel training paradigm ('list items one by one') and a curated dataset to enable SoM promptin...
Octopus v4: Graph of language models	The paper introduces Octopus v4, a system that integrates multiple open-source language models using functional tokens and a graph structure to optimi...	This work differs from related work by introducing a graph-based coordination framework with functional tokens to dynamically route and reformat queri...
Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent	The paper introduces a compact, on-device multimodal AI agent (Octopus v3) optimized for edge devices with <1B parameters, featuring functional tokens...	Octopus v3 differs by introducing functional tokens for AI agent-specific reasoning, optimizing for sub-1B parameters for edge deployment, and demonst...
Octo-planner: On-device Language Model for Planner-Action Agents	The paper introduces an on-device Planner-Action framework for AI agents, separating planning and action execution. It utilizes Phi-3 Mini (a 3.8B par...	The work differs from related work by focusing on on-device execution with fine-tuned LLMs (Phi-3 Mini) and multi-LoRA for multi-domain adaptability,...
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal...	Introduces V-Zen, a novel Multimodal Large Language Model (MLLM) specialized for GUI understanding and grounding, along with the GUIDE dataset. The wo...	V-Zen improves GUI grounding precision with dual-resolution image encoders and a dedicated grounding module, addressing limitations in existing MLLMs....
GUI Action Narrator: Where and When Did That Action Take Place?	The paper introduces a GUI video captioning benchmark (Act2Cap) with 4,189 samples, addressing GUI-specific challenges like dense information and rapi...	This work differs from related work by introducing a specialized GUI video captioning benchmark and framework that explicitly addresses GUI-specific c...
LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions	The paper surveys existing LLM-based single-agent and multi-agent reinforcement learning frameworks, emphasizing communication strategies, coordinatio...	This work differs from related research by focusing on the integration of LLMs into MARL frameworks, emphasizing language-conditioned communication, c...
GUIDE: Graphical User Interface Data for Execution	The paper introduces GUIDE, a novel dataset for MLLMs in RPA, emphasizing multi-platform adaptability and diverse website coverage. It presents V-Zen,...	GUIDE differs from related work by providing annotated data with action grounding, multi-platform adaptability, and diverse website coverage, while V-...
VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Prog...	The paper introduces ICAL, a method that refines suboptimal trajectories into high-quality multimodal programs of thought using VLM self-refinement an...	Unlike prior text-based methods that lack visual cues or introspection, ICAL leverages VLMs to generate causal reasoning and subgoals from noisy demon...
Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based App...	The paper introduces DCGen, a divide-and-conquer approach for generating UI code from screenshots, addressing challenges like element omission, distor...	DCGen differs from related work by introducing a segment-aware divide-and-conquer framework specifically tailored for MLLMs, improving upon CNN-based...
Scaling Instructable Agents Across Many Simulated Worlds	The paper introduces the SIMA project, which develops agents capable of following arbitrary language instructions across diverse 3D environments. Key...	Unlike prior work focused on specific games or environments, SIMA emphasizes training agents across a wide range of 3D worlds (including commercial ga...
DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Te...	DocGenome introduces a large-scale, multi-modal scientific document benchmark addressing limitations in multi-page document extraction, understanding...	DocGenome differs from related work by offering the first comprehensive dataset with structured multi-modal data (including LaTeX, layout attributes,...
Adversarial Robustness for Visual Grounding of Multimodal Large Language Models	This paper addresses the adversarial robustness of visual grounding in Multi-modal Large Language Models (MLLMs) by proposing three novel adversarial...	This work differs from related studies by focusing specifically on adversarial robustness in visual grounding tasks, whereas prior research primarily...
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business...	The paper introduces WONDERBREAD, a benchmark for evaluating multimodal foundation models (FMs) on business process management (BPM) tasks beyond auto...	Unlike existing benchmarks focused on workflow automation, WONDERBREAD addresses underexplored BPM tasks like documentation and knowledge transfer. It...
Large Language Models for Mobile GUI Text Input Generation: An Empirical Study	The paper evaluates the effectiveness of nine state-of-the-art LLMs in generating text inputs for Android GUI testing, demonstrating varying page-pass...	This work differs from related studies by conducting the first large-scale empirical evaluation of LLMs for Android text-input generation in GUI testi...
CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front...	This paper introduces a novel LLM-based GUI agent that operates solely on screenshot inputs, eliminating reliance on HTML/DOM data. It proposes Contex...	Unlike prior work relying on DOM/HTML inputs or integrated architectures, this work uses modular design with LLM-driven action planning via CAAP promp...
Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts	Prompt2Task introduces a multi-agent system for UI task automation on smartphones, translating textual prompts into operation sequences without requir...	Unlike prior UI automation systems that rely on predefined flows or detailed step-by-step instructions, Prompt2Task employs a text-prompt-driven appro...
ChatShop: Interactive Information Seeking with Language Agents	The paper introduces ChatShop, a novel task framework for evaluating language agents' strategic information-seeking capabilities in interactive scenar...	Unlike prior work focused on static information retrieval or limited reference games, ChatShop introduces a realistic, interactive benchmark that eval...
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration	The paper introduces ReAd, a framework for efficient LLM grounding in multi-agent collaboration through Reinforced Advantage feedback. It addresses in...	Unlike prior work relying on physical verification or self-reflection, ReAd introduces a principled advantage-based feedback mechanism grounded in mul...
A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipula...	The paper introduces SC-VLA, a self-correcting vision-language-action model combining fast and slow systems for robust robotic manipulation. It addres...	SC-VLA differs from prior work by explicitly addressing two key limitations: direct correction of low-level SE(3) poses and learning from correction f...
LEGENT: Open Platform for Embodied Agents	Introduces LEGENT, an open platform for embodied agents integrating LLMs and LMMs, offering a 3D environment, data generation pipeline, and demonstrat...	LEGENT differs from related work by providing an open, scalable platform that combines LMMs with embodied training, addressing limitations of existing...
AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language M...	The paper introduces AlignMMBench, a comprehensive Chinese multimodal alignment benchmark addressing gaps in existing benchmarks by focusing on nuance...	AlignMMBench differs from related work by being the first benchmark specifically designed for Chinese visual contexts, incorporating thirteen tasks ac...
Imp: Highly Capable Large Multimodal Models for Mobile Devices	Systematic exploration of lightweight LMMs through model architecture, training strategy, and training data; development of Imp models (2B-4B paramete...	Combines architectural innovations, optimized training strategies, and curated data to achieve high performance in lightweight LMMs, enabling efficien...
Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset f...	The paper introduces BRMData, a comprehensive bimanual-mobile robot manipulation dataset for household tasks, addressing limitations in existing datas...	BRMData differs from related work by providing the first dataset combining bimanual manipulation with mobile mobility, diverse task difficulty levels,...
VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fin...	The paper introduces VGA, a fine-tuned model for GUI comprehension that addresses hallucinations through image-centric fine-tuning. Key contributions...	Unlike prior work that relies on traditional GUI comprehension methods or LVLMs with limited visual alignment, VGA introduces image-centric fine-tunin...
E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion	The paper introduces E-ANT, the first Chinese GUI navigation dataset with real human behavior and high-quality annotations, aiming to improve MLLM dec...	E-ANT differs from prior work by providing a large-scale, Chinese-focused dataset with human-traced interactions and high-quality screenshots, enablin...
On AI-Inspired UI-Design	The paper explores three AI approaches to enhance UI design creativity: using LLMs for direct UI genera

Truncated — view the full README on GitHub.

Contributors

harpreetsahota204

34 commits

Burhan-Q

5 commits

Similar repos

ZJU-REAL/Awesome-GUI-Agents

A curated collection of resources, tools, and frameworks for developing GUI Agents.

460

showlab/Awesome-GUI-Agent

💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.

1,216

OS-Agent-Survey/OS-Agent-Survey

This is the repo for the paper "OS Agents: A Survey on MLLM-based Agents for Computer, Phone and…

486

trycua/acu

A curated list of resources about AI agents for Computer Use, including research papers, projects,…

1,770

Steve2457/Awesome-RL-GUI-Agents

A curated list of awesome RL in GUI Agent papers

WILLOSCAR/Awesome-HCI-LLM

Awesome-HCI （Ubiquitous, LLM, MLLM, Agent, RAG, Embodied-AI, RLHF)

ai-agents-2030/awesome-deep-research-agent

638

Bingo-W/Awesome-Agent-Search

[Survey] Agent and Tool Search: Foundations, Techniques, and Open Challenges

Languages

Python

100.0%

harpreetsahota204/gui_agent_research_landscape

An awesome repository that maps the current landscape of GUI/OS Agent research

Python

39 commits

updated Aug 18, 2025

See the code

README

GUI Agent Research Landscape

Overview

Data Sources

Initial Dataset

The foundation of this analysis comes from the excellent work by the LLM-Brained GUI Agents Survey project. The initial dataset in the data/ directory contains four categories of papers:

benchmark.json - Papers introducing new evaluation benchmarks
dataset.json - Papers contributing new datasets
models.json - Papers presenting new models and architectures
survey.json - Survey and review papers

Each paper entry includes metadata such as title, platform, publication date, arXiv URL, key highlights, and code repository links.

Credit: We gratefully acknowledge @vyokky for curating and maintaining this valuable dataset of GUI agent research papers.

📈 Temporal Analysis of GUI Agent Research Trends

The following table presents a comprehensive analysis of research trends across different time periods, showing the evolution of the GUI agent research landscape. See this file for full details:

Period	Evolution Summary	Future Directions	Dominant Themes	Key Innovations	Emerging Trends	Research Gaps	Methodological Approaches
2016-2021	Transition from static UI analysis to dynamic interaction modeling	Advancing toward robust multimodal models with physics simulation	UI/GUI understanding, RL for navigation, document understanding	Large-scale datasets (Rico, ERICA), multimodal models (UIBert)	Metadata-free UI understanding, structured language integration	Limited annotated datasets, scalability challenges	Transformer architectures, RL with workflow guidance
2022	Integration of LLMs with embodied agents, complex benchmarks	Interactive agents with multi-round dialogue, domain-general models	Vision-language pretraining, embodied agents, LLMs for planning	LLM-Planner, WebShop benchmark, internet-scale pretraining	Interactive navigation, domain adaptation, sim-to-real transfer	Cross-domain generalization, handling ambiguity	RLHF, internet-scale pretraining, prompt engineering
2023 Q1	Tighter LLM-embodied system integration, zero-shot capabilities	Robustness in physical constraints, scalable architectures	LLM-embodied agent integration, vision-language models	Reflexion framework, DEPS planning, CLIP4MC	Zero-shot navigation, multi-modal fusion	Hardware constraints, cross-environment generalization	RL with LLM guidance, self-supervised learning
2023 Q2	Generalist multimodal agents with enhanced real-world capabilities	Real-world deployment robustness, cross-modal reasoning	Multimodal GUI navigation, LLM planning	WebGUM, AdaPlanner, ONE-PEACE models	Generalist cross-domain agents, embodied experiences	Limited real-world benchmarks, domain adaptation	RL with environment feedback, multi-task learning
2023 Q3	Autonomous agents in complex environments, multimodal integration	Efficient computation, physical reasoning, standardized benchmarks	Autonomous web agents, LLM frameworks	AutoDroid, PACE, Auto-GUI, Qwen-VL	Zero-shot physical reasoning, hierarchical frameworks	Real-time adaptation, cross-platform generalization	RL with policy gradients, chain-of-thought prompting
2023 Q4	Integrated vision-language-action systems, zero-shot learning	Real-world adaptability, edge optimization, hallucination mitigation	VLMs for automation, embodied agents, LLMs as controllers	GPT-4V, MM-Navigator, Ferret, MobileVLM	Open-world task automation, 3D vision-language-action	Dynamic environment robustness, edge deployment	RL with foundation models, prompt engineering
2024 Q1	Practical multimodal agents in real-world environments	End-to-end embodied agents, standardized benchmarks	Multimodal agents, benchmarking frameworks	ScreenAgent, WebVoyager, ScreenSpot, ShapeLLM	Ethical AI, multisensory integration, synthetic data	Evaluation standardization, long-term memory	Synthetic data generation, modular architectures
2024 Q2	Edge-optimized multimodal agents, improved benchmarking	Multilingual benchmarks, security enhancement	MLLMs for GUI/Web, on-device optimization	Octopus v4, VGA, InternLM-XComposer2-4KHD	Mobile/edge deployment, RL with LLMs	Multilingual datasets, adversarial robustness	RLHF, multi-agent collaboration, benchmark development
2024 Q3	Specialized autonomous agents, security-focused research	Secure AI-OS integration, cross-cultural benchmarks	Mobile/GUI agent capabilities, benchmarking	MobileVLM, CRAB, Mobile3M, Agent Workflow Memory	Security and privacy focus, AI-OS integration	Non-Western representation, standardized protocols	Benchmark creation, tree search planning
2024 Q4	Foundation-model-driven agents, real-world deployment	Long-term memory, safety mechanisms	GUI agents, web frameworks, safety	TAG, UGround, WebRL, AgentTrek, SPA-Bench	Vision-only GUI interaction, safety evaluation	Long-term memory, safety standardization	RL with curriculum, synthetic data generation
2025 Q1	Autonomous cross-platform agents, ethical deployment	Unified multimodal reasoning, federated learning	GUI/OS automation, environment-free training	AppVLM, VEM, Explorer, VeriSafe Agent	Autonomous evaluation, physical world integration	Benchmarking standardization, long-horizon planning	RL with pretrained value functions, federated learning
2025 Q2	Realistic evaluation, adversarial robustness, multimodal reasoning	Secure efficient agents, cross-platform generalization	Benchmarking, RL for GUI agents, security	AgentRewardBench, InfiGUI-R1, UI-E2I-Synth	Real-world deployment, process reward models	Cross-platform evaluation, ethical frameworks	RL with sub-goal planning, synthetic data generation

Research Pipeline

1. Data Extraction and Expansion (`data_extractor.py`)

The initial dataset was significantly expanded through automated extraction from arXiv:

Multi-level Citation Mining: Starting from the curated papers (Level 1), we extracted their references to identify relevant cited works (Level 2), and then extracted references from those papers (Level 3)
Content Extraction: For each paper, we extracted full text content including abstracts, introductions, methodology sections, and conclusions from arXiv HTML pages
Metadata Enrichment: Enhanced each paper with author information, publication dates, platform classifications, and research categories
Citation Network Construction: Built comprehensive forward and backward citation relationships between papers
Special Paper Integration: Included important non-arXiv papers like Rico and ERICA datasets that are frequently cited in the field

This process expanded the dataset from ~180 curated papers to over 9,000 papers with rich metadata and full content.

2. Intelligent Filtering (`filter_papers.py`)

To maintain focus on GUI/OS/Visual agent research, we implemented a sophisticated keyword-based filtering system:

Multi-tier Keyword Classification: Used high-precision keywords (GUI agent, visual grounding, screenshot analysis) and medium-precision keywords (visual agent, computer control) with different scoring weights
Context-aware Filtering: Enhanced relevance detection by considering keyword co-occurrence and context
Exclusion Patterns: Filtered out papers from unrelated domains (medical, financial, pure NLP) to maintain research focus
Level-based Strategy: Kept all Level 1 (curated) papers while applying filtering to Level 2 and 3 papers to balance comprehensiveness with relevance

This reduced the dataset to approximately 600 highly relevant papers while maintaining the core research landscape.

3. LLM-Powered Content Analysis (`paper_analyzer.py`)

Each paper was analyzed using Qwen3-8B to extract structured insights:

Key Contributions Extraction: Identified the main research themes and contributions from abstracts, introductions, and conclusions
Innovation Classification: Automatically detected whether papers introduce new models, datasets, benchmarks, or frameworks
Contribution Analysis: Analyzed how each work differs from and builds upon related work
Batch Processing: Optimized GPU utilization through efficient batch processing for scalable analysis
Quality Assurance: Implemented fallback mechanisms and error handling to ensure robust analysis

This enriched each paper with structured metadata about its contributions and innovations.

4. Temporal Trend Analysis (`research_trend_analyzer.py`)

Research trends were analyzed across different time periods using Qwen3-14B:

Time Period Segmentation: Organized papers into meaningful periods (2016-2021 Early Era, 2022 Growth Year, 2023+ Quarterly analysis)
Trend Identification: For each period, identified dominant themes, key innovations, emerging trends, and research gaps
Methodological Evolution: Tracked changes in research approaches and methodologies over time
Future Direction Prediction: Generated insights about where the field might be heading based on current trends
Cross-period Analysis: Compared different eras to understand the field's evolution

This provided a comprehensive understanding of how GUI agent research has evolved and where it's headed.

5. Timeline and Temporal Analysis (`research_timeline_analysis.py`)

Comprehensive temporal analysis with multiple visualization approaches:

Publication Timeline: Tracked paper publication rates over time with trend projections for future years
Platform Evolution: Analyzed how research focus has shifted across different platforms (Web, Mobile, Desktop, etc.)
Innovation Trends: Examined the temporal distribution of different types of contributions (models, datasets, benchmarks)
Research Acceleration: Identified periods of rapid growth and analyzed sustainability indicators
Granular Analysis: Provided quarterly-level analysis for recent years to capture fine-grained trends

Generated multiple visualizations showing the field's rapid growth, particularly accelerating after 2022.

6. Citation Network Analysis (`citation_network_analysis.py`)

Deep analysis of the citation network and research influence patterns:

Network Construction: Built a comprehensive citation graph connecting papers through their references
Influence Metrics: Calculated various influence measures including citation counts, PageRank scores, and network centrality
Foundation vs. Frontier Papers: Distinguished between foundational works that established the field and frontier papers pushing current boundaries
Citation Burst Detection: Identified papers experiencing rapid citation growth, indicating emerging influence
Research Bridge Analysis: Found papers that connect different research communities or approaches
Temporal Dynamics: Analyzed how influence patterns change over time and identified paradigm shifts

This revealed the field's key influential works, emerging stars, and the structure of research communities.

Key Findings

Rapid Growth: The field has experienced exponential growth, particularly since 2022, with 2024 seeing unprecedented research activity
Platform Diversification: Research has expanded from early web-focused work to encompass mobile, desktop, and cross-platform approaches
Innovation Patterns: Strong emphasis on new models and architectures, with increasing focus on comprehensive benchmarks and datasets
Emerging Trends: Growing interest in multimodal approaches, real-world deployment, and foundation models for GUI understanding
Research Maturation: Evolution from proof-of-concept work to production-ready systems and comprehensive evaluation frameworks

Citation Network Analysis Results

📊 Analysis Outputs

The citation network analysis produces the following key tables and insights:

Core Influence Rankings

Most Cited Papers - Top papers by raw citation count, showing the most referenced works in the field
Influential Papers Ranking - PageRank-based influence ranking that considers network effects and citation quality
Foundation Papers - Papers that established key concepts and methodologies in GUI agent research
Frontier Papers - Recent papers pushing the boundaries of current research

Research Infrastructure Analysis

Benchmark Papers - Papers introducing evaluation benchmarks and testing frameworks
Dataset Papers - Papers contributing datasets for training and evaluation
Model Papers - Papers introducing new models and architectures (GUI agent-specific)

Temporal Dynamics

Temporal Analysis - Papers with dynamic citation patterns and recent momentum
Citation Velocity - Papers with high citation rates and sustained impact
Future Impact Signals - Papers showing early indicators of potential high impact

Visualizations

Research Timeline - Visual timeline showing the evolution of influential papers over time

🔍 Key Insights from Citation Analysis

The citation network analysis reveals several important patterns:

Field Evolution

Foundation Era (2016-2021): Established by datasets like Rico and ERICA, and early GUI understanding frameworks
Growth Phase (2022-2023): Rapid expansion with new benchmarks like Mind2Web, WebArena, and WebShop
Current Wave (2024-2025): Focus on production-ready agents, multimodal approaches, and comprehensive evaluation

Most Influential Works

The analysis identifies papers with the highest network influence (combining citations and network centrality):

Rico Dataset (2017): Foundational mobile app dataset enabling data-driven GUI research
Mind2Web (2023): Breakthrough web agent benchmark that catalyzed current research wave
WebArena (2023): Comprehensive web environment for autonomous agent evaluation
Recent Multimodal Models: Growing influence of vision-language models adapted for GUI tasks

Research Patterns

Citation Bursts: Certain papers experience rapid citation growth following major conferences or breakthroughs
Cross-Platform Influence: Papers spanning multiple platforms (web, mobile, desktop) tend to have higher influence
Benchmark-Driven Growth: Introduction of new benchmarks consistently drives citation activity and follow-up research

Emerging Trends

Multimodal Integration: Increasing focus on combining vision and language for GUI understanding
Real-world Deployment: Shift from proof-of-concept to production-ready agent systems
Evaluation Sophistication: More comprehensive and realistic evaluation frameworks
Foundation Model Adaptation: Leveraging and fine-tuning large language models for GUI tasks

📈 Network Statistics

The citation network analysis processes:

Total Papers Analyzed: 600+ filtered and enriched papers
Citation Relationships: Comprehensive forward and backward citation links
Time Span: Research from 2016 to 2025
Platform Coverage: Web, Mobile, Desktop, Android, iOS, and cross-platform research

🎯 How to Use These Results

For Literature Review: Start with Foundation Papers and Most Cited Papers for comprehensive background
For Current Trends: Focus on Frontier Papers and Temporal Analysis for cutting-edge research
For Benchmarking: Reference Benchmark Papers for evaluation frameworks and comparison baselines
For Implementation: Check Model Papers and Dataset Papers for practical resources
For Future Research: Review Future Impact Signals for emerging opportunities

🔄 Analysis Methodology

The citation network analysis employs several sophisticated metrics:

PageRank Algorithm: Measures influence based on citation network structure
Temporal Pattern Detection: Identifies citation bursts and momentum changes
Cross-Platform Analysis: Evaluates papers spanning multiple research domains
Foundation vs. Frontier Classification: Distinguishes established works from cutting-edge research

All tables include arXiv badges for direct access to papers, along with key contributions and innovation summaries extracted via LLM analysis.

Repository Structure

├── data/                          # Initial curated dataset
│   ├── benchmark.json            # Benchmark papers
│   ├── dataset.json             # Dataset papers  
│   ├── models.json              # Model papers
│   └── survey.json              # Survey papers
├── src/                         # Source code directory
│   ├── data_extractor.py        # ArXiv data extraction and expansion
│   ├── filter_papers.py         # Intelligent paper filtering
│   ├── paper_analyzer.py        # LLM-powered content analysis  
│   ├── research_trend_analyzer.py # Temporal trend analysis
│   ├── research_timeline_analysis.py # Timeline visualization
│   └── citation_network_analysis.py # Citation network analysis
└── README.md                    # This file

Usage

Data Extraction: Run data_extractor.py to expand the dataset from arXiv
Filtering: Use filter_papers.py to focus on relevant GUI agent research
Content Analysis: Apply paper_analyzer.py for LLM-powered insights
Trend Analysis: Execute research_trend_analyzer.py for temporal trends
Visualization: Run timeline and network analysis scripts for comprehensive insights

Each script includes detailed command-line options and can be run independently or as part of the full pipeline.

Requirements

Python 3.8+
PyTorch with CUDA support (for LLM analysis)
Transformers library (HuggingFace)
NetworkX, Matplotlib, Seaborn (for analysis and visualization)
BeautifulSoup, Requests (for web scraping)

Citation

If you find this research landscape analysis helpful for your work, please consider citing:

@misc{sahota2025gui,
  title={GUI Agent Research Landscape: A Comprehensive Analysis of Trends, Networks, and Evolution},
  author={Sahota, Harpreet},
  year={2025},
  url={https://github.com/harpreetsahota204/gui_agent_research_landscape},
  note={Comprehensive analysis of GUI agent research from 2016-2025}
}

Acknowledgments

Original Dataset: @vyokky for the foundational curated dataset
LLM Analysis: Powered by Qwen3-8B and Qwen3-14B models from Alibaba Cloud
Research Community: All the researchers whose work is analyzed in this landscape study

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Limitations

Publication Source Bias
- Analysis primarily covers papers from arXiv
- Misses important work published exclusively in conferences, journals, or industry venues
- Only manually added exceptions (Rico and ERICA) for non-arXiv papers
Citation Network Boundaries
- Limited to 3 levels of citation depth
- May miss influential papers that are more distantly connected
- Creates an artificial boundary in understanding the full research landscape
Content Analysis Constraints
- Relies on keyword-based detection for platforms and categories
- May miss nuanced or emerging concepts not covered by predefined keywords
- Only extracts specific sections (abstract, intro, related work, methodology, conclusion)
Reference Coverage
- Primarily tracks and analyzes references with arXiv IDs
- Limited analysis of citations to non-arXiv papers
- May underrepresent industry impact or practical applications

Papers by Time Period

The following tables organize all papers in the dataset by time periods, showing their key contributions and innovations.

Total Papers: 609

Papers by Period:

2016-2021: Early Era: 28 papers
2022: Growth Year: 39 papers
2023: Q1: 20 papers
2023: Q2: 36 papers
2023: Q3: 29 papers
2023: Q4: 62 papers
2024: Q1: 71 papers
2024: Q2: 84 papers
2024: Q3: 56 papers
2024: Q4: 74 papers
2025: Q1: 71 papers
2025: Q2: 39 papers

2016-2021: Early Era

Title	Summary	Contributions
WebGPT: Browser-assisted question-answering with human feedback	The paper introduces a browser-assisted question-answering system that integrates web browsing with GPT-3, utilizing imitation learning and human feed...	This work differs from related work by explicitly combining web-browsing capabilities with large language models, leveraging human feedback for iterat...
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning	ALFWorld bridges abstract text-based policy learning and concrete visual execution, introducing the BUTLER agent that leverages pre-learned abstract k...	Unlike prior work focused on either abstract reasoning or visual execution, ALFWorld combines both through a unified simulator, enabling agents to tra...
Mapping Natural Language Instructions to Mobile UI Action Sequences	The paper introduces a novel problem of grounding natural language instructions to mobile UI actions, develops three new datasets including PIXELHELP,...	This work differs from related work by focusing specifically on mobile UI interaction, creating task-specific datasets with grounded action sequences,...
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration	The paper introduces workflow-guided exploration for reinforcement learning on web interfaces, using high-level workflows derived from expert demonstr...	This work differs from related work by using workflow-guided exploration to constrain action sequences with expert demonstrations, rather than relying...
AndroidEnv: A Reinforcement Learning Platform for Android	The paper introduces AndroidEnv, a reinforcement learning platform for Android that enables agents to interact with real-world apps via a touchscreen...	AndroidEnv differs from related work by providing a realistic Android simulation environment tailored for RL research, enabling direct interaction wit...
UIBert: Learning Generic Multimodal Representations for UI Understanding	The paper introduces UIBert, a transformer-based model for UI understanding that leverages self-aligned multimodal features (image, text, structural m...	Unlike prior work, UIBert utilizes self-alignment between UI component features and proposes five pre-training tasks to learn generic representations...
WebSRC: A Dataset for Web-Based Structural Reading Comprehension	The paper introduces WebSRC, a novel dataset for structural reading comprehension on web pages, emphasizing the need for systems to understand both se...	This work differs from related work by focusing on structural understanding of web pages, providing a comprehensive dataset with HTML, screenshots, an...
OCR-free Document Understanding Transformer	The paper introduces an OCR-free approach for document understanding using a Transformer model (Donut), addressing limitations of OCR-based methods by...	The work differs from related OCR-based VDU methods by eliminating reliance on OCR engines, using a Transformer with cross-entropy loss for end-to-end...
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning	The paper introduces robosuite, a modular simulation framework and benchmark for robot learning, leveraging the MuJoCo physics engine. Key contributio...	robosuite differs from related work by providing a modular framework with standardized benchmarks and task environments, enabling rigorous evaluation...
DocVQA: A Dataset for VQA on Document Images	The paper introduces DocVQA, a specialized dataset for Visual Question Answering on document images, emphasizing structural understanding challenges....	DocVQA differs from related work by focusing specifically on document images, requiring models to understand document structure, and providing a bench...
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embod...	The paper introduces HM3D, a large-scale 3D dataset with 1,000 building-scale environments, emphasizing its superior physical scale, visual fidelity,...	HM3D differs from related work by offering significantly larger navigable space (112.5k m²), higher visual fidelity (20-85% improvement over Replica/M...
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation	ThreeDWorld introduces a platform for interactive multi-modal physical simulation with high-fidelity rendering, customizable agents, generative enviro...	Unlike prior work, ThreeDWorld provides a unified platform with real-time physics simulation, multi-modal sensory data, and customizable agents, enabl...
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models	The paper introduces Cross-modal Prompt Tuning (CPT) to address the gap between pre-training and fine-tuning objectives in Vision-Language Pre-trained...	Unlike traditional fine-tuning approaches that require large labeled datasets, CPT leverages color-based prompt tuning to bridge the pre-training and...
Rearrangement: A Challenge for Embodied AI	The paper introduces a framework for Embodied AI research centered on the rearrangement task, emphasizing standardized metrics, scenario characterizat...	This work differs from related work by establishing a standardized rearrangement task with formal metrics and simulation testbeds, enabling reproducib...
Grounding Open-Domain Instructions to Automate Web Support Tasks	The paper introduces RUSS, a system that grounds open-domain natural language instructions into web actions using a domain-specific language (ThingTal...	This work differs from related work by introducing ThingTalk as an intermediate domain-specific language for grounding instructions, enabling more pre...
A Systematic Literature Review of Automated Techniques for Functional GUI Testin...	The paper provides a systematic review of automated GUI testing techniques for mobile applications, analyzing their effectiveness, efficiency, and pra...	This work differs from related studies by conducting a comprehensive systematic review of 25 primary studies to evaluate and compare existing automate...
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Un...	The paper introduces MarkupLM, a pre-training model that jointly learns text and markup language information for document understanding, addressing th...	This work differs from related work by using markup languages as the backbone for pre-training, rather than relying on fixed layouts or images. It exp...
Learning UI Navigation through Demonstrations composed of Macro Actions	The paper introduces a framework for UI navigation that simplifies state and action spaces using UI elements and macro actions. It proposes demo augme...	This work differs from related work by focusing on macro-action-based navigation with demo augmentation and customized DQfD, enabling efficient traini...
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household...	iGibson 2.0 introduces object-centric simulation with enhanced object states (temperature, wetness, cleanliness, toggled/sliced), predicate logic func...	Unlike prior simulation environments focused on motion and physics, iGibson 2.0 extends capabilities to handle complex object states and semantic task...
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospecti...	The paper introduces HANNA, a photo-realistic simulator for visual navigation tasks with human-like assistance. It proposes a memory-augmented neural...	This work differs from related work by integrating multimodal assistance (language + vision) with retrospective curiosity-driven imitation learning, h...
pix2code: Generating Code from a Graphical User Interface Screenshot	The paper introduces an end-to-end deep learning model that generates code from GUI screenshots with high accuracy across multiple platforms. It addre...	This work differs from related work by leveraging deep learning for direct code generation from screenshots, whereas prior methods often relied on man...
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences	This paper introduces a vision-based approach for mobile GUI understanding by abstracting visual elements into 'Pixel-Words' and structuring them into...	Unlike prior works dependent on human-created metadata (e.g., View Hierarchy), this work defines atomic visual components (Pixel-Words) and employs a...
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning	The paper introduces Screen2Words, a multi-modal approach for mobile UI summarization that integrates text, images, UI structures, and semantics. It p...	The work differs from related work by combining multi-modal data (text, images, UI structure, semantics) for UI summarization and introducing a large-...
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning...	Introduces a physically realistic benchmark for task-and-motion planning in embodied AI, emphasizing visual guidance and physics-driven interactions....	This work differs by creating a comprehensive benchmark that integrates visual task understanding, physics-based object manipulation, and realistic en...
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or...	The paper conducts a large-scale empirical study on GUI element detection methods, identifies limitations of existing approaches, and proposes a novel...	The work differs by introducing a GUI-specific old-fashioned method with a top-down coarse-to-fine strategy, integrating it with deep learning for tex...
DOM-Q-NET: Grounded RL on Structured Language	The paper addresses challenges in web navigation for RL agents by introducing DOM-Q-NET, which handles large discrete action spaces and varying action...	DOM-Q-NET differs from related work by using a structured representation of HTML via graph neural networks and separating action-specific Q-networks f...
Document Visual Question Answering Challenge 2020	The paper introduces a new problem in visual question answering (VQA) focused on document images, proposing two tasks: single-image question answering...	This work differs from related work by specifically addressing VQA on document images, which is distinct from general-purpose VQA tasks. It introduces...
VisualMRC: Machine Reading Comprehension on Document Images	Introduces VisualMRC, a dataset focused on document images with abstractive answers, and a new model integrating visual layout and content for machine...	Differently focuses on document-specific reading comprehension with abstractive answers, integrating visual layout understanding, whereas related work...

2022: Growth Year

Title	Summary	Contributions
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embod...	The paper explores using large language models (LLMs) to decompose high-level tasks into actionable steps for embodied agents without additional train...	This work differs from prior methods by leveraging pre-trained LLMs for zero-shot task decomposition, avoiding explicit step-by-step training. It focu...
LAION-5B: An open large-scale dataset for training next generation image-text mo...	The paper introduces LAION-5B, a large-scale image-text dataset enabling training of advanced language-vision models. It emphasizes the importance of...	LAION-5B differs from prior work by offering the first openly available dataset of this scale, enabling broader research on multi-modal models without...
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI	The paper introduces a GUI-based task-oriented dialogue system (GUI-TOD) that directly interacts with mobile app GUIs without relying on backend APIs,...	This work differs from related work by eliminating reliance on task-oriented dialogue-specific backend APIs, enabling direct GUI interaction for task...
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agen...	The paper introduces WebShop, a large-scale simulated e-commerce environment with real-world products and crowd-sourced instructions, addressing scala...	This work differs from related work by creating a scalable, real-world benchmark (WebShop) with extensive product and instruction data, enabling evalu...
Understanding HTML with Large Language Models	The paper explores the application of large language models (LLMs) to HTML understanding tasks, demonstrating their effectiveness in semantic classifi...	This work differs from related work by demonstrating that pre-trained LLMs, when fine-tuned, outperform task-specific models in HTML understanding tas...
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots	Introduces ScreenQA, a large-scale benchmark dataset for screen content understanding through question-answering, bridging low-level structural analys...	ScreenQA is the largest QA dataset for mobile screenshots, using full screenshots rather than cropped regions, and includes unanswerable questions and...
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language...	The paper introduces LLM-Planner, a novel method leveraging large language models (LLMs) for few-shot planning in embodied agents. It addresses high d...	LLM-Planner differs from related work by integrating physical grounding into LLM planning, enabling environment-aware few-shot task execution without...
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus	The paper introduces a vision-only approach for mobile UI understanding, emphasizing the use of screenshots and a region of interest (focus) to bypass...	Unlike prior work reliant on view hierarchies, Spotlight uses a vision-language model with screenshot and focus inputs, achieving state-of-the-art res...
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge	MineDojo introduces a framework for building generalist embodied agents by integrating an open-ended simulation environment, an internet-scale multimo...	Unlike prior work focused on isolated environments with limited tasks, MineDojo combines a diverse simulation suite, large-scale knowledge integration...
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos	The paper introduces Video PreTraining (VPT), a method for training agents in sequential decision domains by leveraging unlabeled online videos throug...	This work differs from related work by extending internet-scale pretraining to sequential decision domains using unlabeled videos, enabling zero-shot...
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding	The paper introduces Pix2Struct, a pretrained image-to-text model for visual language understanding, focusing on parsing screenshots into HTML. It add...	Pix2Struct differs from prior work by using HTML structure as a pretraining target for visual parsing, enabling OCR-free end-to-end understanding. It...
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-...	The paper introduces OFA, a unified framework that unifies cross-modal and unimodal tasks through a sequence-to-sequence learning paradigm. It emphasi...	OFA differs from related work by eliminating task-specific layers through a simple sequence-to-sequence framework, achieving competitive performance w...
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Dere...	The paper introduces MatCha, a pretraining framework that enhances visual language models by integrating math reasoning and chart derendering tasks. I...	Unlike prior work, MatCha focuses on explicit plot deconstruction and numerical reasoning, addressing limitations in existing vision-language models f...
Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied...	The paper introduces a modular approach for Embodied Instruction Following (EIF) by integrating physical constraints of robots and leveraging Large La...	Prompter differs from prior work by incorporating physical constraints into modular design and replacing trained object search models with LLM-based p...
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasib...	The paper introduces MoTIF, a novel dataset for vision-language navigation (VLN) in mobile apps that addresses unknown command feasibility. It provide...	This work differs from related work by introducing the first dataset with feasibility annotations for VLN, addressing task uncertainty through follow-...
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking	The paper introduces LayoutLMv3, a multimodal pre-trained model for Document AI that unifies text and image masking with a word-patch alignment object...	Unlike prior multimodal models that used disparate objectives for text and image modalities, LayoutLMv3 introduces a unified masking approach and a cr...
SQA3D: Situated Question Answering in 3D Scenes	Introduces SQA3D, a new task for evaluating embodied agents' scene understanding through situated question answering in 3D environments. Provides a co...	This work differs from related work by focusing on situated question answering in 3D scenes, emphasizing spatial reasoning, multi-hop logic, and commo...
VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attrib...	The paper introduces VL-CheckList, a framework to evaluate vision-language pretraining (VLP) models by analyzing their capabilities in objects, attrib...	Unlike prior work focused on downstream task performance, VL-CheckList provides an explainable, taxonomy-driven benchmark that decomposes VLP model ca...
Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI...	The paper introduces QTypist, a context-aware text input generation framework leveraging pre-trained LLMs for mobile GUI testing. It addresses the cha...	Unlike prior work, QTypist innovatively applies LLMs to GUI testing with a tailored prompt engineering approach for text generation, achieving signifi...
Improving Multimodal Interactive Agents with Reinforcement Learning from Human F...	The paper introduces a method to enhance embodied agents using reinforcement learning from human feedback (RLHF) with a novel 'Inter-temporal Bradley-...	This work differs from related work by employing IBT modeling to capture human judgment dynamics for reward learning, enabling effective RLHF in compl...
Retrospectives on the Embodied AI Workshop	The paper provides a retrospective analysis of Embodied AI research, focusing on three themes: visual navigation, rearrangement, and embodied vision-a...	This work differs from related research by offering a comprehensive retrospective analysis of challenges and progress in Embodied AI, emphasizing cros...
Dichotomy of Control: Separating What You Can Control from What You Cannot	The paper introduces the Dichotomy of Control (DoC) framework for future-conditioned supervised learning in offline reinforcement learning, addressing...	This work differs from related work by explicitly separating policy control mechanisms from environmental stochasticity through latent variable condit...
Unifying Vision, Text, and Layout for Universal Document Processing	The paper introduces UDOP, a unified model for document AI that integrates vision, text, and layout modalities. It proposes a novel Vision-Text-Layout...	UDOP differs from related work by unifying vision, text, and layout modalities into a single model with a novel architecture, enabling both document u...
MUG: Interactive Multimodal Grounding on User Interfaces	The paper introduces MUG, an interactive multimodal grounding task for UI interactions that enables iterative user-agent collaboration. It emphasizes...	Unlike prior works that focused on single-round UI grounding, this work introduces iterative interaction frameworks, a dedicated dataset for multi-rou...
Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Un...	The paper explores the use of pre-trained BERT models for multi-step tasks involving GUI interaction, specifically through web browsers. It introduces...	This work differs from related work by explicitly integrating pre-trained BERTs with GUI interaction for multi-step tasks, whereas prior research prim...
Objaverse: A Universe of Annotated 3D Objects	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...
Enabling Conversational Interaction with Mobile UI using Large Language Models	The paper explores using pre-trained large language models (LLMs) with prompting techniques to enable versatile conversational interactions with mobil...	This work differs from related work by avoiding the need for task-specific datasets or models, leveraging general-purpose LLMs with carefully designed...
CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation	The paper explores zero-shot Vision-and-Language Navigation (VLN) using CLIP, demonstrating its capability to navigate environments via natural langua...	This work differs from related work by applying CLIP for zero-shot VLN using natural language referring expressions (rather than object class template...
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends	The paper provides a comprehensive survey of vision-language pre-training (VLP) methods, categorizing them into image-text, core computer vision, and...	This work differs from related work by offering a structured survey of VLP methods across diverse modalities, emphasizing challenges and advanced rese...
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-conn...	The paper introduces mPLUG, a vision-language foundation model addressing computational inefficiency and information asymmetry in cross-modal alignmen...	mPLUG differs from prior work by introducing cross-modal skip-connections to reduce computational costs of long visual sequences and mitigate informat...
A data-driven approach for learning to control computers	The paper introduces a data-driven approach combining reinforcement learning with behavioral priors derived from human-computer interactions to achiev...	This work differs from related work by focusing on scalable reinforcement learning with behavioral priors from real human interactions, avoiding hand-...
Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Wid...	The paper introduces a psychologically-inspired, unsupervised method for perceptual grouping of GUI widgets using Gestalt principles, addressing the c...	This work differs from related work by leveraging Gestalt theory for unsupervised perceptual grouping without supervision or runtime data, outperformi...
Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcemen...	The paper introduces PIRLTest, a platform-independent GUI testing framework combining image embedding and reinforcement learning. It addresses limitat...	Unlike prior work relying on static models or random exploration, PIRLTest innovates by embedding GUI images as states for reinforcement learning, usi...
Masader Plus: A New Interface for Exploring +500 Arabic NLP Datasets	The paper introduces Masader Plus, a web interface designed to enhance the exploration of Arabic NLP datasets by addressing UX challenges, enabling da...	This work differs from related work by providing a user-centric web interface for dataset exploration, emphasizing usability improvements and interact...
Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language...	The paper introduces a method combining language models and reinforcement learning for voxel building tasks in Minecraft-like environments, emphasizin...	This work differs by integrating language model-generated sub-goals with reinforcement learning for task execution, improving feasibility verification...
IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents	The paper introduces IGLU Gridworld, a reinforcement learning environment focused on embodied dialog agents with visual embodiment, language-condition...	Unlike prior work, IGLU Gridworld provides a unified benchmark combining visual agent embodiment, language conditioning, and complex task spaces, enab...
Collecting Interactive Multi-modal Datasets for Grounded Language Understanding	The paper formalizes a collaborative embodied agent using natural language tasks, develops a scalable data collection tool, and introduces the first d...	This work differs from related work by providing the first interactive grounded language understanding dataset, along with a formal framework for embo...
Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phras...	The paper advances 3D scene understanding by introducing a fine-grained task (3DPAG) that explicitly handles phrase-level object relationships in visu...	This work differs from related work by focusing on fine-grained phrase-aware grounding rather than coarse-grained object localization, introducing a l...
AccoMontage2: A Complete Harmonization and Accompaniment Arrangement System	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q1

Title	Summary	Contributions
GPT-4 Technical Report	GPT-4 is a large-scale multimodal model with human-level performance on professional and academic benchmarks, improved post-training alignment for fac...	Differs from prior work by achieving human-level performance on complex benchmarks (e.g., simulated bar exams), demonstrating multimodal capabilities...
A Survey of Large Language Models	The paper provides a comprehensive survey of large language models (LLMs), focusing on their evolution from statistical to neural models, the impact o...	This work differs from related work by offering a structured review of LLMs' advancements, emphasizing scaling laws, emergent capabilities, and practi...
Reflexion: Language Agents with Verbal Reinforcement Learning	Reflexion introduces a novel framework for language agents that uses verbal reinforcement learning through self-reflective feedback. It leverages epis...	Reflexion differentiates from related work by using verbal feedback as a 'semantic gradient' instead of traditional reinforcement learning, enabling p...
Describe, Explain, Plan and Select: Interactive Planning with Large Language Mod...	The paper introduces DEPS, an interactive planning framework leveraging Large Language Models (LLMs) to address long-term reasoning and sub-task effic...	DEPS differs from prior work by integrating self-explanation and feedback loops for error correction, introducing a trainable goal selector to optimiz...
Language Models can Solve Computer Tasks	The paper introduces a novel prompting method (Recursively Criticizes and Improves, RCI) enabling pre-trained LLMs to execute computer tasks via natur...	This work differs by using RCI prompting with minimal demonstrations and no task-specific reward functions, achieving state-of-the-art results on Mini...
Where are we in the search for an Artificial Visual Cortex for Embodied Intellig...	The paper presents CortexBench, a comprehensive benchmark for evaluating pre-trained visual representations (PVRs) across 17 EAI tasks. It evaluates e...	This work unifies fragmented EAI research by introducing CortexBench, a broad benchmark for evaluating PVRs across diverse tasks. It refutes prior ass...
Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Age...	The paper addresses the challenge of integrating semantic knowledge from large language models (LLMs) with grounded environmental understanding for em...	This work differs from related work by combining the semantic capabilities of LLMs with grounded models of the environment through a novel decoding fr...
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Lang...	The paper introduces a novel approach combining large language models (LLMs) with reinforcement learning (RL) for embodied agents, focusing on languag...	This work differs from related work by integrating LLMs to hypothesize abstract world models (AWMs) for planning, followed by environment-driven verif...
Collaborating with language models for embodied reasoning	This work explores the integration of large language models (LLMs) with embodied agents to enhance reasoning in complex environments. It introduces a...	This work differs from related work by proposing a novel architecture that synergizes pre-trained language models with embodied agents through a colla...
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills	ManiSkill2 introduces a comprehensive benchmark addressing limitations in existing manipulation benchmarks by incorporating diverse object-level varia...	Unlike prior benchmarks, ManiSkill2 offers a unified framework with extensive object diversity, dynamic simulations, and scalable infrastructure, enab...
Lexi: Self-Supervised Learning of the UI Language	The paper introduces Lexi, a self-supervised model for learning visio-linguistic representations of UIs without relying on metadata like UI trees or a...	This work differs from related work by avoiding reliance on UI metadata, leveraging self-supervised learning with a novel dataset, and focusing on UI-...
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigati...	The paper introduces ESC, a zero-shot object navigation framework that leverages pre-trained vision-language models and commonsense reasoning from lar...	ESC differs from prior work by using pre-trained commonsense knowledge from LLMs for zero-shot object navigation without requiring environment-specifi...
Reinforcement Learning Friendly Vision-Language Model for Minecraft	The paper introduces CLIP4MC, a cross-modal contrastive learning framework for training RL-friendly vision-language models (VLMs) in open-ended tasks....	Unlike prior work, CLIP4MC explicitly incorporates task completion degree into the VLM training objective to provide more instructive reward signals f...
Distilling Internet-Scale Vision-Language Models into Embodied Agents	The paper introduces a method to distill internet-scale vision-language models (VLMs) into embodied agents for language grounding. It combines model d...	This work differs from related work by repurposing pre-trained VLMs for offline supervision of embodied agents through prompt-based distillation and H...
Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zer...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Inte...	The paper introduces NOPA, a method for socially intelligent home assistants that addresses adaptivity and scalability in online goal inference. Key c...	NOPA differs from prior work by integrating neural and probabilistic methods for adaptive goal inference and uncertainty-aware helping strategies, alo...
Alexa Arena: A User-Centric Interactive Platform for Embodied AI	The paper introduces Alexa Arena, a user-centric simulation platform for Embodied AI (EAI) research, enabling human-robot interaction (HRI) missions t...	Alexa Arena differs from related work by offering a user-centric, gamified platform that bridges simulation and real-world HRI, along with a dialog-en...
ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and P...	ViewRefer addresses 3D visual grounding by integrating multi-view knowledge from both text and 3D modalities. It leverages large-scale language models...	Unlike prior works focusing solely on 3D modality or manual alignment, ViewRefer innovates by extracting view knowledge from both text and 3D inputs....
The Universal NFT Vector Database: A Scaleable Vector Database for NFT Similarit...	The paper introduces a scalable vector database for NFT similarity matching, a modular cloud-centered NFT processing system compliant with ERC-721 sta...	This work differs from related work by proposing an off-chain, hardware-agnostic architecture for NFT data aggregation based on vector similarity, com...
CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Emb...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q2

Title	Summary	Contributions
Voyager: An Open-Ended Embodied Agent with Large Language Models	Voyager introduces an LLM-powered embodied lifelong learning agent for open-ended environments like Minecraft, featuring an automatic curiosity-driven...	Unlike prior works that rely on fixed curricula or require human interaction, Voyager enables open-ended exploration through a bottom-up curiosity-dri...
Mind2Web: Towards a Generalist Agent for the Web	The paper introduces Mind2Web, the first dataset for generalist web agents, emphasizing real-world websites, diverse domains/tasks, and user interacti...	Differs from prior work by using real-world websites instead of simulations, providing diverse tasks across 31 domains, and integrating LLMs with a fi...
Multimodal Web Navigation with Instruction-Finetuned Foundation Models	This work introduces WebGUM, a multimodal agent for web navigation that combines vision-language foundation models with instruction-finetuning. It add...	Unlike prior works reliant on domain-specific architectures and online RL, this work leverages instruction-finetuned vision-language foundation models...
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought	The paper introduces EmbodiedGPT, a multi-modal foundation model for embodied AI that integrates vision-language pre-training with embodied planning a...	This work differs from related work by explicitly combining Chain-of-Thought planning with vision-language pre-training, enabling more effective task...
AdaPlanner: Adaptive Planning from Feedback with Language Models	AdaPlanner introduces adaptive planning for LLM agents through feedback, combining in-plan and out-of-plan refinement strategies. It mitigates halluci...	Unlike prior methods that use static plans or greedy actions, AdaPlanner introduces closed-loop adaptive planning with feedback, reducing hallucinatio...
Mindstorms in Natural Language-Based Societies of Mind	The paper introduces the concept of Natural Language-Based Societies of Mind (NLSOMs), where diverse neural networks communicate via natural language...	This work differs from related work by focusing on the societal organization of agents rather than individual models, leveraging natural language as a...
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User In...	This paper introduces a pixel-based approach for GUI agents, demonstrating that agents can outperform humans on instruction-following tasks using only...	The work differs from prior methods by using purely pixel-based inputs and generic action spaces instead of structured representations (e.g., DOM tree...
Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction	The paper introduces Mobile-Env, a comprehensive toolkit for creating qualified GUI interaction benchmarks in Android environments. It addresses limit...	Mobile-Env differs from existing benchmarks by offering a controllable and isolated environment with support for intermediate rewards and instructions...
Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing	This paper introduces GPTDroid, a novel approach leveraging large language models (LLMs) like GPT-3 for zero-shot human-like mobile GUI testing. It fo...	GPTDroid differs from related work by combining LLM-based natural language understanding with GUI testing, using dynamic context iteration and neural...
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension a...	UniChart introduces a pretrained vision-language model specialized for chart comprehension and reasoning, incorporating chart-specific pretraining tas...	UniChart differs from related work by explicitly modeling chart structures (e.g., visual elements, data relationships) through task-specific pretraini...
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Age...	The paper introduces the PET framework, which leverages pre-trained LLMs to simplify control tasks for embodied agents by decomposing tasks into sub-t...	This work differs by using LLMs as teachers to simplify control problems rather than directly solving them, employing modular components (Plan, Elimin...
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitie...	The paper introduces a scalable general representation model (ONE-PEACE) capable of handling unlimited modalities through modality adapters, shared se...	Unlike prior work that often relies on pre-trained models for specific modalities, ONE-PEACE introduces a unified architecture with modality-agnostic...
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control	Synapse introduces state abstraction, trajectory-as-exemplar prompting, and exemplar memory to enhance computer control agents. It addresses limitatio...	Unlike prior work relying on task-specific exemplars or incomplete prompts, Synapse uses abstracted state trajectories and memory-based retrieval to e...
Language Models Meet World Models: Embodied Experiences Enhance Language Models	The paper introduces a framework that enhances language models (LMs) with embodied experiences from world models to improve reasoning and planning in...	This work differs from related work by explicitly combining world models with LMs to bridge the gap in embodied knowledge, while preserving the LMs' g...
WebCPM: Interactive Web Search for Chinese Long-form Question Answering	The paper introduces WebCPM, the first Chinese long-form question answering (LFQA) dataset that leverages interactive web search for information retri...	This work differs from related work by introducing WebCPM, a novel dataset collected via interactive web search, and by emphasizing the synthesis of h...
LLM as A Robotic Brain: Unifying Egocentric Memory and Control	The paper introduces LLM-Brain, a framework that unifies egocentric memory and control in embodied AI using large language models (LLMs). It emphasize...	This work differs from related work by unifying memory and control through an embodied LLM, leveraging zero-shot learning for robotic tasks, and integ...
Prompting Is All You Need: Automated Android Bug Replay with Large Language Mode...	This paper introduces AdbGPT, a lightweight LLM-based approach for automated Android bug replay through prompt engineering. It leverages few-shot lear...	Unlike prior work relying on S2R extraction and predefined patterns, AdbGPT uses LLMs with prompt engineering to infer bug reproduction steps without...
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language...	The paper introduces NavGPT, a purely LLM-based navigation agent that demonstrates explicit reasoning capabilities in vision-and-language navigation (...	This work differs from related work by leveraging large language models (LLMs) for zero-shot VLN without additional training, emphasizing explicit rea...
Towards Generalist Robots: A Promising Paradigm via Generative Simulation	The paper proposes a generative simulation paradigm for robotics research, leveraging large-scale foundation models to automate task and scene generat...	This work differs from related work by advocating a fully automated generative pipeline (generative simulation) that uses foundation models to create...
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human P...	WebGLM introduces a web-enhanced question-answering system that integrates a retriever, bootstrapped generator, and human preference-aware scorer to i...	WebGLM differs by combining retrieval augmentation with human preference learning, creating a novel architecture for efficient web QA. It introduces a...
Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement L...	This paper addresses efficient interaction between agents and large language models (LLMs) using reinforcement learning. It introduces When2Ask, an RL...	Unlike prior work that relies on hard-coded rules or frequent LLM querying, this paper proposes a learned interaction policy (When2Ask) that dynamical...
Multimodal Procedural Planning via Dual Text-Image Prompting	The paper introduces the Multimodal Procedural Planning (MPP) task, which generates paired text-image steps for task execution. It proposes Text-Image...	This work differs from related work by explicitly addressing cross-modal informativeness, temporal coherence, and accuracy in procedural planning thro...
Fine-Grained Visual Prompting	The paper introduces Fine-Grained Visual Prompting (FGVP) to enhance zero-shot instance-level tasks for Vision-Language Models (VLMs). It proposes the...	This work differs from related work by introducing fine-grained visual prompting using segmentation masks (e.g., from SAM) instead of coarse cues like...
Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving	This paper introduces a novel multi-modal visual grounding task (LiDAR Grounding) tailored for autonomous driving scenarios, proposing the Multi-modal...	This work differs from related work by focusing on 3D outdoor autonomous driving scenarios rather than 2D or indoor 3D domains. It introduces a novel...
DUBLIN -- Document Understanding By Language-Image Network	The paper introduces DUBLIN, a model pretrained on web pages with novel objectives for visual document understanding. It achieves state-of-the-art res...	DUBLIN differs from related work by employing three novel training objectives (Masked Document Text Generation, Bounding Box Task, Rendered Question A...
GUILGET: GUI Layout GEneration with Transformer	The paper introduces GUILGET, a transformer-based model for generating GUI layouts from positional constraints (GUI-AGs), emphasizing constraint adher...	GUILGET differs from related work by explicitly addressing GUI design constraints through a transformer architecture that captures semantic relationsh...
A Generalist Dynamics Model for Control	This paper introduces transformer sequence models as dynamics models (TDMs) for control, demonstrating their strong generalization capabilities in bot...	The work differs from related work by focusing on using transformers as dynamics models rather than policies, emphasizing generalization across unseen...
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learnin...	The paper advances deep reinforcement learning (Deep RL) for bipedal robots, demonstrating agile soccer skills through zero-shot sim-to-real transfer....	This work differs by focusing on full-body control of bipedal robots for long-horizon, multi-agent competitive tasks, whereas prior research primarily...
Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solu...	The paper introduces a large-scale, diverse dataset (POIE) for Visual Information Extraction (VIE) in real-world scenarios, along with an end-to-end f...	The work differs by proposing a more challenging and diverse dataset (POIE) compared to existing benchmarks like SROIE and EPHOIE, and introduces a no...
Document Understanding Dataset and Evaluation (DUDE)	The paper introduces a novel dataset (DUDE) for visually-rich documents, emphasizing multi-industry, multi-domain, and multi-page layouts. It proposes...	This work differs from related work by focusing on practical, real-world applicable benchmarks through diverse datasets and multi-task evaluations, ad...
Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysi...	The paper introduces the STARS framework, which enhances LLM-based task learning for embodied agents by addressing limitations of prompt engineering....	STARS differs from related work by combining online task learning, multi-source knowledge exploitation, and proactive LLM response evaluation without...
Navigating to Objects Specified by Images	The paper presents a modular system for embodied agents to navigate to objects specified by images, combining semantic visual reasoning with explorati...	This work differs from related work by using a modular approach with off-the-shelf components for sub-tasks (exploration, re-identification, localizat...
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explana...	The paper introduces Musketeer, a vision-language model trained jointly across multiple heterogeneous tasks with fully shared parameters. It proposes...	Unlike prior work with separate task-specific heads or adapters, Musketeer employs a unified encoder-decoder architecture with fully shared parameters...
ENTL: Embodied Navigation Trajectory Learner	ENTL introduces a unified approach for embodied navigation by integrating world modeling, localization, and imitation learning into a single sequence...	ENTL differs from related work by unifying world modeling and localization into a sequence prediction framework without explicit rewards, using a tran...
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabil...	The paper focuses on developing embodied agents capable of human-like collaboration through grounded language instructions. Key contributions include...	This work differs from related research by providing a comprehensive framework for collecting and utilizing grounded language instructions, offering a...
OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answe...	The paper introduces the first large-scale open-domain Vietnamese VQA dataset (OpenViVQA) with 11,000+ images and 37,000+ question-answer pairs, along...	This work differs from related work by focusing on open-ended answer generation in Vietnamese, providing a new benchmark for low-resource languages, a...

2023: Q3

Title	Summary	Contributions
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text...	The paper introduces Qwen-VL, a series of vision-language models with enhanced visual perception capabilities through a visual receptor, 3-stage train...	The work differs from related work by integrating a meticulously designed 3-stage training pipeline, visual receptor, and input-output interface, alon...
WebArena: A Realistic Web Environment for Building Autonomous Agents	The paper introduces WebArena, a highly realistic and reproducible web environment for autonomous agents, featuring functional websites from four doma...	WebArena differs from prior work by providing a realistic web environment with dynamic, functional websites and a benchmark focused on functional corr...
The Rise and Potential of Large Language Model Based Agents: A Survey	The paper provides a comprehensive survey of LLM-based agents, tracing their conceptual origins, presenting a general framework (brain, perception, ac...	This work differs by offering a holistic framework for LLM-based agents, systematically categorizing applications, and analyzing societal implications...
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control	The paper introduces RT-2, a vision-language-action (VLA) model that integrates large-scale web data with robotic control through co-fine-tuning. It e...	This work differs by co-fine-tuning vision-language models on both robotic trajectory data and internet-scale vision-language tasks, treating actions...
A Real-World WebAgent with Planning, Long Context Understanding, and Program Syn...	The paper introduces WebAgent, an LLM-driven autonomous agent for real-world web automation that addresses open-domainness, long-context HTML understa...	Unlike prior works relying on simulated environments or single LLMs, WebAgent combines HTML-T5 (specialized for HTML with novel attention mechanisms)...
You Only Look at Screens: Multimodal Chain-of-Action Agents	The paper introduces Auto-GUI, a multimodal GUI agent that directly interacts with interfaces without environment parsing or application-specific APIs...	Auto-GUI differs from prior work by eliminating reliance on external tools (e.g., OCR) and application APIs, directly interacting with GUIs through mu...
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis	The paper introduces SDXL, a latent diffusion model for text-to-image synthesis with enhanced performance through a larger UNet backbone, novel condit...	SDXL improves upon existing latent diffusion models by scaling the UNet architecture, introducing novel conditioning techniques, and incorporating a r...
Android in the Wild: A Large-Scale Dataset for Android Device Control	The paper introduces a large-scale dataset (AITW) for device-control research, emphasizing visual and language interaction, multi-step tasks, and robu...	This work differs from related work by providing an order-of-magnitude larger dataset with diverse Android versions, device types, and complex multi-s...
AutoDroid: LLM-powered Task Automation in Android	AutoDroid integrates large language models (LLMs) with domain-specific app knowledge through dynamic analysis to enable scalable, hands-free mobile ta...	AutoDroid differs from related work by combining LLM commonsense knowledge with app-specific domain knowledge through automated dynamic analysis, elim...
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs	BuboGPT enhances multi-modal LLMs with fine-grained visual grounding, enabling precise cross-modal interactions between vision, audio, and language. I...	Unlike prior works that rely on coarse-grained mappings, BuboGPT achieves fine-grained visual grounding by integrating a SAM-based module and a novel...
Building Cooperative Embodied Agents Modularly with Large Language Models	The paper introduces a modular framework integrating Large Language Models (LLMs) into embodied agents for decentralized multi-agent cooperation. It p...	This work differs from related research by employing a decentralized, modular cognitive-inspired framework that leverages LLMs for reasoning, language...
KOSMOS-2.5: A Multimodal Literate Model	KOSMOS-2.5 introduces a multimodal literate model for document-level text recognition and structured output generation, combining spatial-aware text b...	Unlike prior work focused on line-level OCR or domain-specific structured parsing, KOSMOS-2.5 unifies spatial and structural document understanding th...
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizati...	Introduces Retroformer, a framework for reinforcing large language agents using policy gradient optimization with a retrospective model that automatic...	Proposes a policy gradient approach for language agents, differentiating from prior work by leveraging environment rewards directly rather than relyin...
Embodied Task Planning with Large Language Models	This paper introduces TaPA, an embodied task planning agent that integrates large language models (LLMs) with visual perception for grounded planning....	The work differs by combining LLM-generated plans with scene-grounded object detection for executable actions, using a custom multimodal dataset for t...
Physically Grounded Vision-Language Models for Robotic Manipulation	The paper introduces PhysObjects, a large-scale dataset annotating physical concepts of household objects, and demonstrates how fine-tuning vision-lan...	This work differs from prior work by focusing on physical concept annotations (e.g., material, fragility) rather than visual attributes, and by integr...
An In-depth Survey of Large Language Model-based Artificial Intelligence Agents	The paper provides a comprehensive comparison between LLM-based AI agents and traditional agents, analyzing core components (planning, memory, tool us...	This work differs from related work by offering a systematic survey of LLM-based agents, emphasizing their unique strengths in natural language proces...
When Do Program-of-Thoughts Work for Reasoning?	The paper investigates the effectiveness of program-of-thought prompting for reasoning, introducing the Complexity-Impacted Reasoning Score (CIRS) to...	This work differs from related studies by introducing CIRS, a metric that evaluates code complexity through both structural (AST) and logical (difficu...
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as a...	LLM-Grounder introduces a zero-shot, open-vocabulary 3D visual grounding approach using LLMs as agents to decompose queries, reason about spatial rela...	Unlike prior CLIP-based methods that suffer from 'bag-of-words' limitations, LLM-Grounder leverages LLMs for compositional semantic decomposition and...
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution...	The paper introduces DoReMi, a framework for grounding language models in robotic tasks by detecting and recovering from plan-execution misalignment....	Unlike prior work that assumes perfect execution of low-level skills or relies on manual feedback, DoReMi introduces a hierarchical framework with aut...
Learning to Model the World with Language	The paper introduces Dynalang, a multimodal agent that unifies language understanding with future prediction as a self-supervised learning objective....	Unlike prior work that relies on task-specific language instructions or supervised learning, Dynalang uses a world model to predict future text/image...
InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent	The paper explores integrating ChatGPT into embodied agent systems through role-based prompt engineering, demonstrating high success rates in simulate...	Unlike prior work, InterAct leverages ChatGPT's versatility through role assignment (e.g., checker, sorter) and prompt engineering to enhance cooperat...
REX: Rapid Exploration and eXploitation for AI Agents	The paper introduces REX, a framework enhancing AI agents' exploration and exploitation by integrating UCB-inspired rewards and offline behavior utili...	REX differs from related work by incorporating UCB-based reward structures and try-and-fail procedures akin to RL, enabling efficient offline behavior...
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Em...	The paper introduces the SimBot Challenge as a new competition for developing embodied AI agents, emphasizing the integration of vision, language, and...	This work differs from related work by establishing the SimBot Challenge as a benchmark for embodied AI, focusing on task completion in simulated phys...
Planning with Logical Graph-based Language Model for Instruction Generation	The paper introduces Logical-GLM, a graph-based language model that integrates logical Bayesian graphs with language models to enhance instruction gen...	Unlike prior work that relies solely on pre-trained LLMs or static knowledge graphs, Logical-GLM combines AI planning with dynamic logical graph struc...
ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation	The paper introduces two-stage sampling and dynamic sampling approaches to enhance the efficiency of reinforcement learning (RL) in sequence generatio...	This work differs from related work by focusing on optimizing sampling efficiency in RL for sequence generation, rather than proposing new architectur...
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model	The paper introduces PACE, a novel prompt editing framework leveraging actor-critic reinforcement learning to enhance LLM performance. It addresses pr...	PACE differs from related work by using actor-critic within LLMs themselves for prompt optimization, avoiding external reward models or training. It e...
Pair then Relation: Pair-Net for Panoptic Scene Graph Generation	This paper addresses the challenge of Panoptic Scene Graph (PSG) generation by introducing Pair-Net, a novel framework that improves inter-object pair...	The work introduces Pair-Net, a novel architecture that directly learns sparse pairwise relationships via a Matrix Learner within the PPN, improving u...
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation...	VELMA introduces a novel approach to urban Vision and Language Navigation (VLN) by leveraging large language models (LLMs) with verbalization of traje...	VELMA differs from prior work by employing a verbalization-based embodiment strategy, where navigation instructions, trajectories, and visual observat...
WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...	Analysis failed: Batch generation error: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 896.00 KiB i...

2023: Q4

Title	Summary	Contributions
MiniGPT-v2: large language model as a unified interface for vision-language mult...	The paper introduces MiniGPT-v2, a unified interface leveraging large language models for vision-language tasks. Key contributions include task-specif...	Unlike prior work, MiniGPT-v2 introduces task-specific identifiers during training to enhance task distinction and learning efficiency, combined with...
CogAgent: A Visual Language Model for GUI Agents	CogAgent introduces a specialized visual language model (VLM) for GUI agents, achieving state-of-the-art performance on VQA benchmarks and GUI navigat...	CogAgent differs by directly processing GUI screenshots (not HTML/OCR) with a high-resolution VLM architecture, enabling human-level GUI understanding...
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V	The paper introduces Set-of-Mark (SoM), a novel visual prompting method that enhances visual grounding capabilities of large multimodal models like GP...	This work differs from related work by focusing on prompt engineering rather than model architecture or training methods, enabling zero-shot visual gr...
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navig...	The paper introduces MM-Navigator, a GPT-4V-based agent for zero-shot smartphone GUI navigation, demonstrating high accuracy in action description and...	This work differs from related work by leveraging GPT-4V's advanced screen interpretation and action reasoning capabilities for zero-shot GUI navigati...
Ferret: Refer and Ground Anything Anywhere at Any Granularity	Ferret introduces a unified framework for referring and grounding in images, utilizing a hybrid region representation and spatial-aware visual sampler...	Ferret differs from related work by unifying referring and grounding in a single framework, supporting diverse region inputs (points, boxes, shapes),...
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-mod...	The paper introduces Monkey, a method to enhance Large Multimodal Models (LMMs) by addressing high-resolution image processing and detailed scene unde...	Monkey differs from related work by introducing a patch-based processing module with sliding window and LoRA adjustments for efficient high-resolution...
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices	MobileVLM introduces mobile-optimized multimodal vision language models with efficient cross-modality interaction via a lightweight projector, achievi...	MobileVLM differs from prior work by emphasizing mobile-specific optimizations (e.g., efficient projector, compact language models) and achieving comp...
A Long Way to Go: Investigating Length Correlations in RLHF	The paper highlights that optimizing response length is a critical, underappreciated factor in RLHF, demonstrating that length-based rewards can repli...	This work differs from related work by systematically analyzing length as a core feature in RLHF, challenging the assumption that length increases are...
UReader: Universal OCR-free Visually-situated Language Understanding with Multim...	URender introduces an OCR-free approach for visually-situated language understanding using a Multimodal Large Language Model (MLLM). Key contributions...	This work differs from related work by eliminating the need for domain-specific pretraining and fine-tuning, leveraging a minimal parameter fine-tunin...
Learning Interactive Real-World Simulators	The paper introduces UniSim, a universal simulator that integrates diverse datasets (image, video, robotics, navigation) to simulate real-world intera...	Unlike prior work focused on domain-specific simulations or limited data, UniSim unifies heterogeneous datasets in a conditional video generation fram...
GLaMM: Pixel Grounding Large Multimodal Model	GLaMM introduces the first model capable of generating natural language responses with object segmentation masks, enabling visually grounded conversat...	GLaMM unifies region understanding, pixel-level grounding, and conversational abilities through end-to-end training, differing from prior works that e...
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation	This paper introduces AssistGUI, a novel benchmark for evaluating GUI automation on Windows desktop environments, along with an Actor-Critic Embodied...	Unlike prior work on mobile/web or terminal automation, this paper focuses on desktop GUI automation with a comprehensive benchmark (AssistGUI) and a...
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs	The paper introduces V*, an LLM-guided visual search mechanism to enhance multimodal LLMs (MLLMs) with efficient visual querying, leading to the SEAL...	The work differs from related work by leveraging rich common sense knowledge from LLMs for dynamic visual search, incorporating both top-down feature...
NExT-Chat: An LMM for Chat, Detection and Segmentation	The paper introduces the pix2emb paradigm for region-level visual understanding, enabling multiple location formats (bounding boxes, masks) through em...	Unlike prior pix2seq methods limited to discrete coordinates, pix2emb uses embeddings decoded into diverse formats via specialized decoders. This arch...
Octopus: Embodied Vision-Language Programmer from Environmental Feedback	Introduces Octopus, an embodied vision-language programmer that bridges high-level planning and real-world manipulation via executable code generation...	Octopus differs by integrating code generation as a medium between planning and manipulation, addressing gaps in prior works that either output low-le...
SteP: Stacked LLM Policies for Web Actions	SteP introduces a dynamic policy composition framework for web tasks using stacked LLM policies, addressing challenges in handling combinatorial web i...	SteP differs from related work by enabling dynamic policy invocation and stacking, allowing adaptive control over task complexity. Unlike static hiera...
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language P...	The paper introduces ViLa, a novel robotic planning framework that integrates vision-language models (VLMs) for grounded task planning. It emphasizes...	Unlike prior work relying on external affordance models or LLMs with limited environmental grounding, ViLa unifies vision and language processing with...
A Zero-Shot Language Agent for Computer Control with Structured Reflection	The paper introduces a zero-shot language agent for computer control that leverages structured reflection to autonomously learn from mistakes without...	This work differs from related work by eliminating the reliance on expert traces or additional screen information, instead using structured self-refle...
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones	Introduces TinyGPT-V, an efficient multimodal large language model (MLLM) with a compact architecture, enabling low-resource training (24GB) and infer...	Differently from prior MLLMs relying on large backbones (e.g., LLaMA2-7B, Vicuna-13B), TinyGPT-V uses a smaller Phi-2 backbone with optimized architec...
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API	The paper introduces a multimodal model for grounding natural language instructions in UI screenshots, leveraging a visual encoder and language decode...	This work differs by integrating reinforcement learning with visual and language components for spatial decoding, proposing a metadata-free grounding...
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models	The paper introduces LLaVA-Grounding, a model that combines visual chat and grounding capabilities. It creates a new dataset (GVC) for grounded visual...	This work differs from related work by integrating chat and grounding tasks into a single model, creating a dedicated dataset and benchmark for ground...
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots	Habitat 3.0 introduces a simulation platform for collaborative human-robot tasks in home environments, focusing on accurate humanoid simulation, human...	Habitat 3.0 differs from related work by integrating humanoid and robot simulation in the same environment, supporting HITL control via VR/mouse/keybo...
WebWISE: Web Interface Control and Sequential Exploration with Large Language Mo...	The paper introduces a method for web interface control using LLMs with in-context learning, leveraging filtered DOM elements as observations and sequ...	WebWISE differs from related work by using DOM observations and in-context learning with minimal examples (single manual or auto-generated example) in...
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversat...	The paper addresses UI task challenges in Vision-Language Models (VLMs) by generating a large-scale conversational dataset without human annotations....	Unlike prior work requiring human annotations, this study generates UI training data automatically using machine conversations, enabling VLMs to handl...
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem	The paper introduces the AIOS-Agent ecosystem, positioning Large Language Models (LLMs) as the core of an intelligent operating system (AIOS) and agen...	This work differs from related work by proposing a conceptual framework where LLMs function as the system-level OS (LLMOS), enabling agents as applica...
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception	The paper introduces MP5, a multi-modal embodied system for open-ended tasks in Minecraft, emphasizing active perception, modular architecture, and co...	MP5 differs from prior work by introducing a modular architecture with active perception and context-aware planning, addressing limitations of previou...
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE	The paper introduces Octavius, a framework addressing task interference in Multimodal Large Language Models (MLLMs) by integrating Mixture-of-Experts...	The work combines MoE with LoRA to create a novel decoder (LoRA-MoE) for MLLMs, addressing task interference through specialized learning paths. This...
An Embodied Generalist Agent in 3D World	The paper introduces LEO, an embodied multi-modal generalist agent designed for 3D world interaction, addressing limitations in 3D perception, groundi...	LEO extends beyond existing 2D-focused generalist models by introducing a unified 3D vision-language-action architecture, a novel 3D dataset with obje...
See and Think: Embodied Agent in Virtual Environment	The paper introduces STEVE, an embodied agent combining vision perception, language instruction, and code action for Minecraft. It proposes a multi-mo...	STEVE differs from prior work by integrating vision perception with LLM-based reasoning and code action generation, enabling more precise and autonomo...
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Wo...	The paper introduces Steve-Eye, a multimodal model integrating LLMs with visual perception for open-world embodied agents. It addresses limitations in...	Unlike prior work focusing on text-based interactions, Steve-Eye integrates visual perception with LLMs, introduces a large-scale open-world instructi...
GROOT: Learning to Follow Instructions by Watching Gameplay Videos	This work introduces GROOT, a novel agent that learns to follow open-ended instructions by watching gameplay videos, addressing limitations in existin...	GROOT differs from prior work by using reference gameplay videos as goal specifications instead of text or static images, enabling self-imitation lear...
UINav: A Practical Approach to Train On-Device Automation Agents	UINav introduces a demonstration-based approach for training on-device UI automation agents with high accuracy using minimal human demonstrations. It...	UINav differs from prior work by combining demonstration-based training with a referee model for real-time feedback and data augmentation, achieving h...
Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing	The paper introduces DroidAgent, an autonomous GUI testing agent for Android that leverages Large Language Models (LLMs) and memory mechanisms for int...	This work differs from related work by integrating LLMs with memory systems to enable high-level semantic testing, moving beyond traditional code cove...
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM...	The paper introduces VisionTasker, a two-stage framework for mobile task automation combining vision-based UI understanding and LLM-driven step-by-ste...	VisionTasker differs from related work by replacing view hierarchies with vision-based UI interpretation and integrating LLMs for dynamic, step-by-ste...
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Alig...	ReMax introduces a reinforcement learning method tailored for aligning large language models (LLMs) by leveraging RLHF properties (fast simulation, de...	ReMax differs from prior work by exploiting RLHF-specific properties and using a greedy baseline in REINFORCE for variance reduction, eliminating the...
An In-depth Look at Gemini's Language Abilities	The paper provides a third-party, reproducible comparison of Google's Gemini and OpenAI's GPT models across 10 language tasks, identifies performance...	This work differs from related work by offering an independent, transparent evaluation of Gemini's language capabilities through reproducible experime...
Creative Agents: Empowering Agents with Imagination for Creative Tasks	The paper introduces creative agents enhanced with an 'imaginator' component to generate task outcomes from language instructions, enabling novel solu...	This work differs from related work by integrating an imaginator for generating task-specific imaginations, enabling creativity in agents. It also int...
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Mod...	The paper addresses underspecification in vision-language tasks by introducing RepARe, a framework that rephrases and augments questions with visually...	Unlike prior work focusing on image captioning or CoT reasoning, RepARe explicitly leverages visual grounding in question rephrasing, utilizing LVLMs...
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on...	The paper introduces a new benchmark (CompWoB) for evaluating compositional web automation tasks, highlights the limitations of language model agents...	This work differs from related work by focusing on compositional task compositionality rather than isolated tasks, introducing a new benchmark (CompWo...
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model...	The paper introduces PCA-EVAL, a benchmark for evaluating embodied decision-making in perception, cognition, and action. It proposes HOLMES, a multi-a...	This work differs by introducing PCA-EVAL as a comprehensive benchmark and HOLMES as a multi-agent framework that directly integrates MLLMs for end-to...
From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models	The paper investigates the effectiveness of different visual encoders in MLLMs, highlighting the advantages of CLIP's shallow layers for fine-grained...	This work differs from related work by analyzing the role of visual encoder depth and introducing COMM, a multi-level feature merging architecture tha...
"What's important here?": Opportunities and Challenges of Using LLMs in Retrievi...	The paper investigates LLMs' ability to retrieve important UI elements from web pages based on user queries, focusing on prompting strategies (example...	Unlike prior works focused on autonomous web navigation, this study decomposes the problem into atomic operations, analyzing how prompting strategies...
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Ma...	The paper introduces SPOC, a transformer-based architecture that imitates shortest-path planners in simulation to enable real-world navigation and man...	This work differs by using shortest-path expert trajectories instead of human demonstrations or dense rewards, combining a transformer architecture wi...
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models	PG-Video-LLaVA introduces the first video-based Large Multimodal Model (LMM) with pixel-level grounding capabilities, integrating audio cues via trans...	This work differs from related work by introducing pixel-level visual grounding in videos, integrating audio context through transcription, proposing...
InfMLLM: A Unified Framework for Visual-Language Tasks	The paper introduces InfMLLM, a unified framework for visual-language tasks, employing a three-stage training scheme (lightweight alignment pretrainin...	InfMLLM differs from prior MLLMs by introducing a progressive training strategy with stage-specific objectives and a novel pool-adapter architecture t...
Vision-Based Mobile App GUI Testing: A Survey	The paper provides a comprehensive survey of vision-based mobile app GUI testing techniques, highlighting their advantages over traditional code/layou...	This work differs from related work by offering a holistic survey of 271 papers (92 vision-based), systematically categorizing GUI testing topics, and...
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld	The paper introduces EMMA, an Embodied Multi-Modal Agent trained via cross-modality imitation learning between a text-world LLM and a visual-world VLM...	This work differs from related work by combining cross-modal distillation from an LLM expert with a novel DAgger-DPO algorithm, enabling embodied VLM...
Large Language Models as Generalizable Policies for Embodied Tasks	The paper introduces LLaRP, a method that adapts large language models (LLMs) as generalizable policies for embodied visual tasks through reinforcemen...	LLaRP differs from prior work by using reinforcement learning directly on pre-trained LLMs for embodied tasks without requiring task-specific training...
CoPAL: Corrective Planning of Robot Actions with Large Language Models	The paper introduces CoPAL, a hierarchical architecture for robot task planning that integrates multi-level feedback loops with Large Language Models...	This work differs from related work by proposing a closed-loop task planning architecture (CoPAL) that systematically integrates geometric constraints...
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasonin...	LL3DA introduces a novel approach for 3D scene understanding, reasoning, and planning by directly processing point cloud inputs, enabling interaction...	LL3DA differs from prior work by directly encoding 3D point clouds instead of using multi-view projections, employing an attention mechanism to integr...
Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraint...	Ctrl-Room introduces a two-stage framework for text-driven 3D room generation with layout constraints, enabling structurally plausible layouts, high-f...	Unlike prior methods that struggle with global layout consistency or lack editability, Ctrl-Room decouples layout and appearance generation, employs s...
Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation	The paper introduces Zero-shot Interactive Personalized Object Navigation (ZIPON), a task that combines zero-shot object navigation with natural langu...	This work differs from related work by introducing ORION, a framework that leverages LLMs for sequential decision-making in interactive navigation, in...
Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Langu...	The paper introduces MERLIM, a multi-modal benchmark for evaluating Instruction Tuning Large Vision and Language Models (IT-LVLMs) on fundamental comp...	This work differs from related work by introducing MERLIM, a standardized benchmark specifically designed to evaluate IT-LVLMs for fundamental vision...
Improving web element localization by using a large language model	The paper introduces VON Similo LLM, an enhanced web element localization approach leveraging Large Language Models (LLMs) to improve accuracy by inco...	Unlike prior post-repair approaches (e.g., WATER, GPT-2-based methods) that rely on attribute comparisons or heuristic repairs, this work integrates L...
Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Langu...	The paper introduces Co-NavGPT, a framework that integrates Vision Language Models (VLMs) as global planners for multi-robot cooperative visual target...	Co-NavGPT differs from related work by combining VLMs with multi-robot systems for semantic reasoning and global planning, enabling efficient collabor...
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts	The paper introduces M3DBench, a comprehensive multi-modal 3D instruction-following dataset that supports diverse 3D tasks, interleaved visual prompts...	Unlike prior 3D datasets focused on specific tasks (e.g., object detection, navigation), M3DBench unifies region and scene-level 3D tasks with multi-m...
Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficientl...	The paper introduces RLFP framework with FAC algorithm, leveraging foundation models for policy, value, and reward priors to enhance sample efficiency...	This work differs from related work by integrating foundation priors (policy, value, success-reward) into RL, enabling automatic reward functions and...
Large Language Model as a Policy Teacher for Training Reinforcement Learning Age...	This paper introduces a framework that combines Large Language Models (LLMs) with reinforcement learning (RL) to train specialized student agents. The...	This work differs from related work by proposing a knowledge distillation framework where an LLM acts as a teacher to guide a specialized RL student a...
InstructDET: Diversifying Referring Object Detection with Generalized Instructio...	The paper introduces InstructDET, a data-centric approach to referring object detection (ROD) that generates diverse instructions for object localizat...	This work differs from related work by (1) generating diverse, human-like instructions via foundation models rather than manual annotation, (2) creati...
GSVA: Generalized Segmentation via Multimodal Large Language Models	The paper introduces GSVA, a novel approach to Generalized Referring Expression Segmentation (GRES) that addresses multi-target and empty-target scena...	GSVA differs from prior work by introducing the [REJ] token for explicit rejection of empty targets and enabling simultaneous segmentation of multiple...
Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI	The paper introduces a scene-driven multimodal knowledge graph (Scene-MMKG) construction method for embodied AI, combining knowledge engineering with...	The work differs from related work by proposing a unified scene knowledge injection framework that integrates conventional knowledge bases with large...
WebVLN: Vision-and-Language Navigation on Websites	Introduces the WebVLN task for navigating websites using natural language instructions, incorporates HTML content alongside visual and textual data, a...	Differs from related work by explicitly incorporating HTML content (non-visual web-specific data) and introducing a new task and benchmark focused on...

2024: Q1

Title	Summary	Contributions
GPT-4V(ision) is a Generalist Web Agent, if Grounded	This work introduces SEEACT, a generalist web agent leveraging large multimodal models (LMMs) like GPT-4V for integrated visual understanding and web...	This work differs from related work by proposing a novel grounding strategy that integrates HTML structure and visuals for web agents, rather than rel...
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents	The paper introduces SeeClick, a visual GUI agent that automates tasks using screenshots instead of structured data, addresses the GUI grounding chall...	SeeClick differs from prior work by eliminating reliance on structured text (e.g., HTML) and GUI metadata, leveraging LVLMs for direct screenshot-base...
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks	Introduces VisualWebArena, a benchmark for evaluating multimodal agents on visually grounded web tasks, emphasizing integration of visual and textual...	VisualWebArena fills the gap in evaluating multimodal agents on visually grounded tasks, offering a comprehensive benchmark with real-world tasks and...
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception	The paper introduces Mobile-Agent, a multi-modal agent that uses visual perception to operate mobile apps without relying on XML metadata. It presents...	Mobile-Agent differs from prior work by employing a vision-centric approach without requiring XML or system metadata, introducing Mobile-Eval as a new...
DeepSeek-VL: Towards Real-World Vision-Language Understanding	DeepSeek-VL introduces a hybrid vision encoder for efficient high-resolution image processing, a comprehensive real-world dataset with diverse modalit...	DeepSeek-VL differentiates from prior work by combining vision and language pretraining with a hybrid vision encoder, addressing limitations of projec...
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models	This paper introduces WebVoyager, a multimodal web agent leveraging large multimodal models (LMMs) to interact with real-world websites end-to-end. It...	WebVoyager differs from prior work by enabling real-world web navigation through multimodal inputs (screenshots and text), utilizing a novel benchmark...
UFO: A UI-Focused Agent for Windows OS Interaction	UFO introduces a dual-agent framework for Windows OS interaction, leveraging GPT-Vision to analyze GUIs and controls, enabling automated task executio...	Unlike prior LLM agents focused on general task planning or code-centric approaches, UFO specifically targets Windows OS UI automation with a dual-age...
ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models	The paper introduces a synthetic dataset (ALLaVA) generated via GPT4V to enhance lite vision-language models (VLMs), achieving performance comparable...	This work differs by focusing on high-quality synthetic data generation (ALLaVA) to bridge performance gaps between lite and large VLMs, rather than r...
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement	The paper introduces OS-Copilot, a framework for building generalist computer agents capable of interacting with diverse OS elements. It presents FRID...	This work differs from related work by focusing on generalist agents for OS interactions, introducing self-improvement mechanisms, and demonstrating s...
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue	The paper introduces WEBLINX, a large-scale benchmark for conversational web navigation, and proposes a retrieval-inspired model to address the challe...	This work differs from related work by introducing a novel benchmark (WEBLINX) and a retrieval-inspired architecture tailored for web navigation tasks...
Android in the Zoo: Chain-of-Action-Thought for GUI Agents	This work introduces Chain-of-Action-Thought (CoAT) for GUI agents, emphasizing semantic reasoning through screen context, action thinking, targets, a...	Unlike prior works focusing solely on coordinate-based actions or separating element recognition from action inference, CoAT integrates semantic reaso...
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language...	SPHINX-X introduces a scalable family of multi-modal large language models (MLLMs) with architectural optimizations, a comprehensive multimodal datase...	This work differs by scaling data and parameters, modifying the SPHINX architecture with skip tokens and one-stage training, and curating a diverse da...
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous A...	Introduces OmniACT, the first dataset and benchmark for evaluating autonomous agents' ability to generate executable scripts for both desktop and web...	OmniACT differs from prior work by combining desktop and web tasks, requiring executable script generation rather than just action prediction, and emp...
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs	Identifies systematic visual shortcomings in multimodal LLMs (MLLMs) stemming from CLIP-based visual encoders, introduces the MMVP benchmark using CLI...	Systematically exposes the limitations of CLIP-based visual encoders in MLLMs, highlights the disconnect between CLIP's embedding space and vision-onl...
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document	TextMonkey introduces an OCR-Free large multimodal model for document understanding, enhancing text-centric tasks through Shifted Window Attention wit...	Unlike OCR-Model-Driven methods, TextMonkey eliminates reliance on external OCR systems by integrating text detection, layout understanding, and visua...
Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Se...	The paper provides a comprehensive survey of Personal LLM Agents, focusing on their architecture, capabilities, efficiency, and security. It highlight...	This work differs from related work by offering a systematic survey and analysis of Personal LLM Agents, emphasizing their integration with personal d...
ScreenAI: A Vision-Language Model for UI and Infographics Understanding	ScreenAI introduces a vision-language model specialized in understanding UIs and infographics, combining PaLI architecture with Pix2Struct's flexible...	ScreenAI differs from related work by unifying UI and infographic understanding through a novel visual language model architecture, leveraging a hybri...
Cradle: Empowering Foundation Agents Towards General Computer Control	The paper introduces the General Computer Control (GCC) setting to standardize interaction with software via screenshots and low-level actions, propos...	This work differs from related work by introducing a standardized interface (screenshots + keyboard/mouse actions), a modular framework (Cradle) that...
Common 7B Language Models Already Possess Strong Math Capabilities	The paper demonstrates that LLaMA-2 7B models possess strong mathematical capabilities without specialized pre-training, achieving high accuracy on GS...	This work differs from related work by focusing on enhancing small language models' mathematical abilities through efficient SFT and synthetic data sc...
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?	The paper introduces WorkArena, a benchmark for evaluating web agents on enterprise software tasks, and BrowserGym, a unified environment for agent ev...	WorkArena focuses on enterprise software workflows (ServiceNow) unlike prior benchmarks (e.g., WebArena, MiniWoB), and introduces BrowserGym as a flex...
Understanding the planning of LLM agents: A survey	The paper provides the first systematic taxonomy of LLM-based agent planning, categorizing existing works into Task Decomposition, Plan Selection, Ext...	This work differs from related work by offering the first comprehensive survey and structured taxonomy of LLM-Agent planning, synthesizing existing re...
Large Language Model based Multi-Agents: A Survey of Progress and Challenges	The paper provides a comprehensive survey of LLM-based multi-agent systems, focusing on their applications in complex problem-solving and world simula...	This work differentiates itself by offering a holistic survey of LLM-based multi-agent systems, emphasizing collective intelligence, inter-agent inter...
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding	The paper introduces Unified Structure Learning for OCR-free Visual Document Understanding, emphasizing structure-aware parsing tasks and multi-graine...	Unlike prior OCR-dependent or limited-domain approaches, this work unifies structure learning across diverse text-rich images (documents, webpages, ta...
ScreenAgent: A Vision Language Model-driven Computer Control Agent	The paper introduces ScreenAgent, a Vision Language Model (VLM)-driven computer control agent that interacts with real computer screens via GUI manipu...	Unlike prior work, ScreenAgent combines VLM with a task-specific control pipeline and dataset for real-world GUI interaction, achieving superior UI po...
LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model	The paper introduces LLaVA-Phi, a compact multi-modal assistant leveraging the Phi-2 small language model for efficient visual-dialogue tasks. It demo...	LLaVA-Phi differs from related work by combining a small language model (Phi-2) with LLaVA-1.5 training methodology, achieving efficiency without sacr...
AgentStudio: A Toolkit for Building General Virtual Agents	AgentStudio introduces a comprehensive toolkit for building general virtual agents, featuring a lightweight interactive environment with generic obser...	Unlike prior domain-specific simulators (e.g., WebShop, AndroidEnv) or narrow benchmarks, AgentStudio provides a generic, real-world compatible enviro...
TravelPlanner: A Benchmark for Real-World Planning with Language Agents	Introduces TravelPlanner, a benchmark for evaluating language agents in complex real-world planning tasks. Highlights challenges in multi-constraint p...	TravelPlanner differs from prior work by focusing on real-world, multi-constraint travel planning scenarios with a rich sandbox environment, diverse t...
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Eng...	The paper introduces Design2Code, the first real-world benchmark for multimodal code generation in front-end engineering. It evaluates current MLLMs o...	This work differs from related studies by creating the first benchmark specifically for visual-to-code generation, providing detailed evaluation metri...
Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dat...	The paper introduces WebSight, a large-scale synthetic dataset of 2 million HTML-code-to-screenshot pairs, and Sightseer, a vision-language model with...	Unlike prior work that relied on small datasets or heuristic-based approaches, this work introduces WebSight, a synthetic dataset with 2.5x larger siz...
Large Multimodal Agents: A Survey	The paper provides a systematic review of Large Multimodal Agents (LMAs), categorizing research into four types, compiling evaluation methodologies, p...	This work differs from related work by offering a comprehensive survey of LMAs, unifying diverse research categories, standardizing evaluation framewo...
Agent AI: Surveying the Horizons of Multimodal Interaction	The paper defines 'Agent AI' as a framework for embodied multimodal systems that integrate visual, linguistic, and environmental data to enable contex...	This work differs from related research by systematically framing Agent AI as a class of systems that prioritize environmental grounding and multimoda...
MobileAgent: enhancing mobile control via human-machine interaction and SOP inte...	The paper introduces MobileAgent, an LLM-based agent that enhances mobile device control through human-machine interaction and integration of Standard...	This work differs from related work by integrating SOP information into in-context learning for better task comprehension and introducing interactive...
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation	The paper introduces CoCo-Agent, a multimodal large language model (MLLM) agent for smartphone GUI automation, focusing on comprehensive environment p...	Unlike prior work relying on strong pre-trained models or black-box APIs, CoCo-Agent introduces a trainable architecture with CEP and CAP to systemati...
3D-VLA: A 3D Vision-Language-Action Generative World Model	The paper introduces 3D-VLA, a generative world model that integrates 3D perception, reasoning, and action for embodied agents. It addresses limitatio...	3D-VLA differs from prior work by explicitly modeling 3D dynamics and using 3D features for action generation, introducing a novel dataset for 3D embo...
ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Ch...	The paper introduces ChartX, a comprehensive multi-modal benchmark for evaluating chart understanding capabilities, and ChartVLM, a new model designed...	This work differs from related work by introducing ChartX as a comprehensive benchmark with diverse chart types and tasks, and ChartVLM with a cascade...
Understanding the Weakness of Large Language Model Agents within a Complex Andro...	The paper identifies critical weaknesses of LLM agents in complex Android environments, introduces AndroidArena as a benchmark for evaluating LLM agen...	This work differs from related research by introducing AndroidArena, the first benchmark specifically designed to evaluate LLM agents in complex OS en...
Exploring Large Language Model based Intelligent Agents: Definitions, Methods, a...	The paper provides a comprehensive survey of LLM-based intelligent agents, covering definitions, research frameworks, foundational components (plannin...	This work differs from related research by offering an in-depth synthesis of LLM-based agents' methodologies, multi-agent system integration strategie...
SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning...	Introduces SkyEyeGPT, a unified multi-modal large language model for remote sensing (RS) vision-language tasks. Develops a large-scale RS instruction-...	Unlike prior RS models like RSGPT that handle single tasks with separate models, SkyEyeGPT unifies diverse RS tasks through instruction tuning and a s...
Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models	The paper addresses limitations in Multimodal Large Language Models (MLLMs) by introducing P2G, a framework for plug-and-play grounding of visual reas...	The work differs from related work by leveraging external agents for on-the-fly grounding instead of relying on training data or existing modules, and...
CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning	Introduces Chain of Manipulations (CoM) for step-by-step visual reasoning in VLMs, proposes the CogCoM model with 17B parameters, annotates 6K graphic...	Differ from prior work by emphasizing step-by-step visual manipulation reasoning (CoM) instead of direct alignment, introducing a multi-turn multi-ima...
On the Multi-turn Instruction Following for Conversational Web Agents	This work introduces the Conversational Web Navigation task, a novel framework (Self-MAP) for handling multi-turn interactions with users and environm...	Unlike prior work focusing on single-turn web navigation tasks, this paper emphasizes multi-turn conversational interactions, introducing a dataset (M...
Tur[k]ingBench: A Challenge Benchmark for Web Agents	Introduces TurkingBench, a benchmark for web-based agents using real HTML pages from crowdsourcing platforms, evaluates multi-modal models on complex...	TurkingBench differs from prior work by using naturally occurring crowdsourcing HTML pages rather than synthesized environments, offering more realist...
DocLLM: A layout-aware generative language model for multimodal document underst...	DocLLM introduces a layout-aware generative language model for multimodal document understanding, focusing on spatial-textual cross-alignment through...	DocLLM differs from related work by avoiding expensive image encoders, focusing exclusively on bounding box-based spatial layouts, and introducing dis...
Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation	The paper introduces a hierarchical auto-organizing system (HAS) for multi-agent navigation in dynamic environments like Minecraft. Key contributions...	Unlike prior work focusing on single-agent reinforcement learning or pre-trained LLMs for task planning, HAS introduces a novel hierarchical architect...
Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models vi...	This paper addresses object hallucination in Large Vision-Language Models (LVLMs) by introducing CLIP-Guided Decoding (CGD), a training-free method th...	Unlike prior work relying on internal model information (e.g., token likelihoods, hidden states) or complex external tools, this work proposes a train...
Dual-View Visual Contextualization for Web Navigation	The paper introduces a method to enhance HTML element representations for web navigation by leveraging dual views (textual and visual context from scr...	Unlike prior work relying solely on HTML documents or simplified representations, this work integrates visual context from webpage screenshots, contex...
BAGEL: Bootstrapping Agents by Guiding Exploration with Language	BAGEL introduces a method to bootstrap language model agents for digital environment interaction without human supervision. It uses iterative round-tr...	BAGEL differs from prior work by eliminating the need for human demonstrations or reward functions, instead leveraging synthetic demonstrations genera...
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simula...	Introduces MineDreamer, an embodied agent in Minecraft that uses Chain-of-Imagination (CoI) to translate natural language instructions into precise vi...	Proposes CoI mechanism to address sequential instruction-following challenges by breaking tasks into stages and generating state-aware visual prompts,...
Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning	Scene-LLM integrates 3D visual understanding with LLMs for interactive indoor environments, combining scene-level and ego-centric 3D information. It i...	Scene-LLM differs from prior work by jointly leveraging scene-level and ego-centric 3D information, employing a hybrid feature representation for dyna...
Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipane...	The paper introduces MultipanelVQA, a novel benchmark for evaluating Multimodal Large Language Models (MLLMs) on multipanel image understanding. It hi...	This work differs from related studies by focusing specifically on multipanel image understanding, introducing a synthetic benchmark to isolate layout...
VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysi...	The paper introduces VHM, a vision language model tailored for remote sensing image analysis, emphasizing versatility and honesty. It proposes two nov...	Unlike prior work focused on factual questions and sparse captions, VHM introduces comprehensive captions (VersaD) and deceptive question datasets (Hn...
ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grai...	The paper addresses visual grounding issues in LVLMs by introducing ViGoR, a framework that leverages fine-grained reward modeling with human evaluati...	ViGoR differs from related work by focusing on reward modeling with fine-grained human feedback to enhance visual grounding, rather than relying solel...
LLM Multi-Agent Systems: Challenges and Open Problems	The paper identifies key challenges in multi-agent systems (MAS) involving LLMs, such as optimizing task allocation, fostering collaborative reasoning...	This work differs from related studies by focusing on systemic challenges in multi-agent collaboration (e.g., context management, memory, and layered...
The Revolution of Multimodal Large Language Models: A Survey	The paper provides a comprehensive survey of Multimodal Large Language Models (MLLMs), focusing on architectural designs, multimodal alignment strateg...	This work differentiates from related studies by offering an exhaustive review of visual-based MLLMs, emphasizing their integration with visual modali...
DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exe...	DoraemonGPT introduces a novel LLM-driven video agent for dynamic scene understanding, addressing spatial-temporal reasoning, large planning spaces, a...	Unlike prior multimodal systems, DoraemonGPT specifically targets dynamic video tasks with a modular architecture combining symbolic memory, sub-task...
AI Assistance for UX: A Literature Review Through Human-Centered AI	The paper provides a systematic literature review of 359 papers on AI-enabled UX tools, highlighting gaps in empathy-building, multi-screen user exper...	This work differs from related studies by focusing on UX practitioners' unmet needs through a Human-Centered AI lens, mapping findings onto the Double...
Embodied LLM Agents Learn to Cooperate in Organized Teams	The paper introduces a framework for organizing LLM agents using prompt-based structures to enhance cooperation in multi-agent systems. It emphasizes...	This work differs from related work by proposing a novel multi-LLM-agent architecture with hierarchical organization structures and a Criticize-Reflec...
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend Wha...	The paper introduces the Draw-and-Understand framework for integrating visual prompting into MLLMs, supports multiple visual prompt types (points, box...	Unlike prior ROI-based methods that rely on segmentation models or ground truth masks, this work proposes a general architecture adaptable to diverse...
MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments	The paper introduces MEIA, a multimodal embodied agent that integrates visual and linguistic memory for task planning in unknown environments. It prop...	MEIA differs from prior work by combining visual-language memory (MEM) for multimodal scene understanding, enabling real-time environmental grounding...
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction	This paper introduces ShapeLLM, the first 3D Multimodal Large Language Model (LLM) for embodied interaction, combining 3D geometry understanding with...	Unlike prior work focusing on 3D representation learning or vision-language models, ShapeLLM unifies 3D geometry understanding (via ReCon++) with lang...
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worl...	The paper introduces MultiPLY, a multisensory embodied large language model that actively interacts with 3D environments by integrating visual, audio,...	This work differs from related work by enabling active 3D environment interaction through multisensory data integration, introducing object-centric re...
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alig...	The paper introduces GOMA, a framework for proactive verbal communication in embodied agents to align mental states toward shared goals. It addresses...	GOMA differs from prior work by explicitly modeling mental state alignment through goal-oriented planning, enabling proactive and contextually relevan...
Will GPT-4 Run DOOM?	Demonstrates GPT-4's ability to play Doom through reasoning and planning without reinforcement learning, evaluates LLM capabilities in complex environ...	This work differs from related work by demonstrating LLM-based planning in a video game without reinforcement learning or fine-tuning, using Doom as a...
ChatterBox: Multi-round Multimodal Referring and Grounding	The paper introduces the MRG task for instance-level multimodal dialogues, a new benchmark (CB-300K) with multi-round dialogue and complex spatial cha...	This work differs by addressing multi-round dialogues and complex spatial relationships in MRG, which existing benchmarks lack. It introduces CB-300K...
GroundingGPT:Language Enhanced Multi-modal Grounding Model	GroundingGPT introduces a language-enhanced multi-modal grounding model focused on fine-grained understanding of local information across modalities (...	GroundingGPT differs from prior work by explicitly targeting fine-grained multi-modal grounding through modality-specific adapters, coordinate-based s...
LCV2: An Efficient Pretraining-Free Framework for Grounded Visual Question Answe...	The paper introduces LCV2, a modular, pretraining-free framework for grounded visual question answering (VQA) that leverages a frozen large language m...	LCV2 differs from related work by eliminating the need for pre-training, using a modular architecture with a frozen LLM as a mediator, and enabling ef...
Enhancing Vision-Language Pre-training with Rich Supervisions	The paper introduces S4, a pre-training paradigm for Vision-Language Models (VLMs) using web screenshots and rich supervisions. It leverages HTML elem...	Unlike prior work focusing on image-text pairs or object detection tasks, S4 innovates by utilizing web screenshots with structured HTML annotations a...
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Vis...	Griffon v2 enhances multimodal perception by addressing image resolution limitations in LVLMs, enabling high-resolution scaling (up to 1K) and visual-...	Unlike prior methods that rely on division-based resolution enhancement or low-resolution image encoding, Griffon v2 employs a lightweight projector t...
Language-guided Skill Learning with Temporal Variational Inference	The paper introduces a framework for language-guided skill discovery using hierarchical variational inference and an auxiliary objective based on Mini...	The work differs from prior methods by integrating LLM-generated semantic segmentation with variational inference, avoiding language-only skill repres...
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modul...	CREMA introduces a modular, parameter-efficient multimodal fusion framework for video-language reasoning, enabling seamless integration of diverse mod...	Unlike prior work, CREMA achieves scalability and efficiency by decoupling modality processing from the LLM, using parameter-efficient modules and a n...
Object-Centric Instruction Augmentation for Robotic Manipulation	The paper introduces the Object-Centric Instruction Augmentation (OCI) framework to enhance robotic manipulation by integrating object positions into...	Unlike prior work focusing on task planning or vision-language model integration, this work emphasizes augmenting language instructions with explicit...

2024: Q2

Title	Summary	Contributions
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with O...	The paper introduces three key improvements: (1) a strong vision encoder with continuous learning for better visual understanding, (2) dynamic high-re...	InternVL 1.5 addresses gaps in parameter scale, image resolution, and multilingual capability compared to proprietary models by integrating a continuo...
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer En...	The paper introduces OSWorld, a real computer environment for evaluating multimodal agents in open-ended tasks across multiple operating systems. It p...	OSWorld differs from prior work by providing a scalable, real-world interactive environment and benchmark that captures the diversity and complexity o...
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents	The paper introduces AndroidWorld, a dynamic benchmarking environment for autonomous agents on Android, featuring 116 programmatic tasks across 20 rea...	AndroidWorld differs from related work by providing the first comprehensive mobile benchmark with dynamically generated, parameterized tasks across di...
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs	The paper introduces Cambrian-1, a vision-centric multimodal LLM family that evaluates diverse visual representations through visual instruction tunin...	Unlike prior work, Cambrian-1 focuses on vision-centric design and integrates spatial awareness via SVA to enhance visual grounding. It introduces CV-...
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs	Ferret-UI introduces a specialized multimodal large language model (MLLM) for mobile UI understanding, addressing limitations in existing models throu...	This work differs from related work by explicitly addressing UI-specific challenges (e.g., elongated aspect ratios, small objects) through a dual-subi...
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Reso...	The paper introduces InternLM-XComposer2-4KHD, a large vision-language model capable of handling resolutions from 336 pixels to 4K HD. It addresses re...	Unlike prior work that uses fixed resolutions or separate HR/LR encoders, this work proposes dynamic resolution scaling with automatic patch configura...
What matters when building vision-language models?	The paper systematically evaluates critical design choices in vision-language models (VLMs), including architecture, data, and training methods. It id...	This work differs from related research by rigorously ablation-studying key design decisions in VLMs through controlled experiments, providing empiric...
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement...	The paper introduces DigiRL, an autonomous reinforcement learning framework for training device-control agents in real-world GUI environments. Key con...	DigiRL differs from prior work by combining autonomous offline-to-online RL with pre-trained VLMs, addressing real-world stochasticity and non-station...
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via...	The paper introduces a multi-agent architecture (planning, decision, reflection agents) for mobile device operation tasks, addressing navigation chall...	This work differs from related work by proposing a specialized multi-agent collaboration framework tailored for mobile device GUI operations, incorpor...
AutoWebGLM: A Large Language Model-based Web Navigating Agent	The paper introduces AutoWebGLM, a large language model-based web navigating agent designed to overcome challenges in real-world web navigation. Key c...	This work differs from related work by addressing HTML complexity and open-domain task challenges through HTML simplification, curriculum learning, an...
On the Effects of Data Scale on UI Control Agents	The paper investigates the scalability of fine-tuning LLMs for UI control agents, introduces the AndroidControl dataset with high/low-level instructio...	The work introduces AndroidControl, the most diverse UI control dataset with both high/low-level instructions, and provides systematic analysis of how...
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devi...	This work introduces GUI Odyssey, a comprehensive dataset for cross-app GUI navigation, and develops OdysseyAgent, a multimodal navigation agent. It a...	This work differs from related work by focusing on cross-app navigation tasks, which previous datasets and models primarily ignored. It introduces GUI...
GUICourse: From General Vision Language Models to Versatile GUI Agents	The paper introduces GUICourse, a comprehensive suite of datasets (GUIEnv, GUIAct, GUIChat) to enhance Vision Language Models (VLMs) for GUI agent tas...	This work differs from related work by explicitly addressing VLM limitations in OCR, grounding, and GUI-specific knowledge through purpose-built datas...
MANTIS: Interleaved Multi-Image Instruction Tuning	The paper introduces Mantis, a family of large multimodal models trained via instruction tuning on a curated dataset (Mantis-Instruct) to excel in mul...	Unlike prior work that relies on pre-training on large noisy interleaved data, Mantis achieves strong multi-image capabilities through instruction tun...
Autonomous Evaluation and Refinement of Digital Agents	The paper introduces domain-general automatic evaluators for digital agents, demonstrating their effectiveness in improving web navigation and device...	This work differs from related work by proposing automated evaluators that operate without expert demonstrations or supervision, leveraging vision-lan...
From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function	The paper establishes DPO as a token-level Q-learning algorithm within the RLHF framework, bridging the gap between contextual bandit-based DPO and cl...	The work theoretically frames DPO as an inverse Q-learning algorithm in token-level MDPs, resolving discrepancies with classical RLHF by satisfying Be...
OpenVLA: An Open-Source Vision-Language-Action Model	This paper introduces OpenVLA, an open-source vision-language-action (VLA) model trained on 970k real-world robot demonstrations. It combines Llama 2...	OpenVLA differs from prior work by integrating Internet-scale vision-language foundation models with robot demonstrations, enabling generalization to...
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding a...	The paper introduces VisualWebBench, a comprehensive benchmark for evaluating multimodal large language models (MLLMs) in web page understanding and g...	VisualWebBench differs from related work by providing a holistic evaluation framework tailored to web contexts, encompassing perception, comprehension...
A Survey on the Memory Mechanism of Large Language Model based Agents	The paper provides a comprehensive survey on memory mechanisms in LLM-based agents, systematically reviewing existing designs, evaluating their effect...	This work differs from related studies by offering the first holistic review of memory mechanisms in LLM-based agents, abstracting design patterns, an...
Octopus v2: On-device language model for super agent	The paper introduces Octopus v2, an on-device language model with 2B parameters that significantly improves accuracy and reduces latency compared to G...	The work differs by introducing functional tokens for efficient function calling, optimizing latency and accuracy for on-device use, and achieving cos...
WebCanvas: Benchmarking Web Agents in Online Environments	The paper introduces WebCanvas, an online evaluation framework for web agents that addresses the dynamic nature of web environments. Key contributions...	Unlike static benchmarks, WebCanvas focuses on dynamic web environments with real-time task evaluation, introduces a live dataset with intermediate st...
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language...	Ferret-v2 enhances referring and grounding capabilities in MLLMs through high-resolution image processing, multi-granularity visual encoding with DINO...	Unlike prior MLLMs reliant on low-resolution encoders (e.g., CLIP), Ferret-v2 introduces DINOv2 for multi-granularity visual encoding and a three-stag...
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation	LlamaTouch introduces a novel testbed for evaluating mobile UI task automation agents by focusing on faithful and scalable task execution evaluation....	Unlike prior work relying on static datasets or human validation, LlamaTouch introduces a dynamic benchmark that evaluates agents based on traversing...
GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents	Introduces GUI-World, a comprehensive dataset for GUI-oriented multimodal LLM agents, emphasizing dynamic and sequential tasks across diverse GUI scen...	GUI-World addresses gaps in prior work by covering dynamic GUI content, multi-window interactions, and diverse operating environments. It provides a b...
Large Language Models Can Self-Improve At Web Agent Tasks	The paper explores LLM self-improvement for web agent tasks using synthetic training data, achieves a 31% task completion rate improvement on WebArena...	This work differs by focusing on self-improvement through fine-tuning on model-generated data for long-horizon web agent tasks, and by introducing com...
MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents	The paper introduces MobileAgentBench, a benchmark for evaluating mobile LLM agents on Android devices. It addresses the lack of standardized testing...	This work differs from related work by focusing on benchmarking rather than model development. While prior studies explored LLM/MLLM-based agents and...
MMInA: Benchmarking Multihop Multimodal Internet Agents	The paper introduces MMInA, a benchmark for evaluating embodied agents in multihop, multimodal web tasks. Key contributions include evolving real-worl...	MMInA addresses gaps in existing benchmarks by focusing on real-world evolving websites, multihop tasks, and multimodal reasoning. It introduces a hol...
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding	Introduces MuirBench, a comprehensive benchmark for evaluating multi-image understanding in multimodal LLMs, highlighting challenges in spatial-tempor...	MuirBench differs from prior work by providing a broader, more comprehensive assessment of multi-image reasoning capabilities, including 12 diverse ta...
Benchmarking Mobile Device Control Agents across Diverse Configurations	The paper introduces B-MoCA, a novel benchmark for evaluating mobile device control agents across diverse device configurations. It emphasizes tasks m...	Unlike prior work focused on specific agent types or limited task sets, B-MoCA provides a unified benchmark with diverse device configurations and 131...
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundi...	This work introduces the Screen Point-and-Read (ScreenPR) task, a novel GUI-referring task involving screen reading based on user-indicated points. It...	This work differs from related work by focusing on the ScreenPR task, which is underexplored in GUI understanding. It introduces the ToL grounding mec...
MoVA: Adapting Mixture of Vision Experts to Multimodal Context	The paper addresses the limitations of single vision encoders in MLLMs by introducing MoVA, a novel framework that adaptively routes and fuses task-sp...	MoVA differs from prior work by introducing a context-aware expert routing strategy guided by LLMs and a fine-grained MoV-Adapter for task-specific kn...
Octopus: On-device language model for function calling of software APIs	The paper introduces Octopus, an on-device LLM fine-tuned for software API function calling, with techniques like conditional masking and a novel benc...	This work differs by focusing on on-device LLMs for API interactions, introducing conditional masking for format alignment, and creating a specialized...
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Mod...	Groma introduces localized visual tokenization for grounded multimodal understanding, enabling region-level tasks like captioning and grounding. It in...	Groma differs by embedding localization into image tokenization rather than relying on external modules or LLM-based coordinate outputs, enabling unif...
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy	LLaRA introduces a framework for adapting pretrained Vision-Language Models (VLMs) into Vision-Language-Action (VLA) models for robotic control. It ge...	LLaRA differs from related work by focusing on visuomotor instruction tuning for robotics, generating task-specific conversation-style datasets from e...
Efficient Multimodal Large Language Models: A Survey	The paper provides a comprehensive survey of efficient Multimodal Large Language Models (MLLMs), summarizing their development timelines, efficient st...	This work differs from related work by offering a systematic review of efficient MLLMs, highlighting scalability challenges, and proposing future dire...
Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single M...	The paper introduces STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied agents, addressing limitations in multi-LLM syst...	STEVE-2 differs from prior work by distilling complex multi-agent systems into a single model with hierarchical architecture, enabling dynamic adaptat...
Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vi...	The paper identifies 'blind tokens' in LVLMs that cause hallucinations due to excessive attention on irrelevant image regions. It introduces AvisC, a...	Unlike prior output-level methods that contrast whole-image representations or use external models, AvisC focuses on internal attention patterns to id...
VideoGUI: A Benchmark for GUI Automation from Instructional Videos	Introduces VideoGUI, a multi-modal benchmark for advanced GUI automation tasks derived from instructional videos, emphasizing visual-centric tasks, hi...	Differently from prior benchmarks limited to simple text-instruction tasks, VideoGUI targets complex visual-centric GUI tasks via instructional videos...
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents	Wilbur introduces adaptive in-context learning for web agents, combining a differentiable ranking model, instruction synthesis, and an intelligent bac...	Wilbur differs from prior work by introducing backtracking to recover from mistakes, synthesizing both positive and negative task demonstrations, and...
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Langua...	The paper presents TextHawk, a Multimodal Large Language Model (MLLM) specifically designed for document-oriented tasks, featuring four novel componen...	TextHawk differs from prior work by introducing a novel architecture with ReSA, SPEs, QPN, and MLCA for efficient fine-grained document perception, al...
Search Beyond Queries: Training Smaller Language Models for Web Interactions via...	The paper introduces GLAINTEL, a novel agent for intelligent web navigation that leverages reinforcement learning and smaller language models to addre...	GLAINTEL differs from related work by focusing on smaller language models trained via reinforcement learning and unsupervised methods, demonstrating s...
Training a Vision Language Model as Smartphone Assistant	This research introduces a Vision Language Model (VLM) designed to control mobile devices through UI interactions, leveraging sequences of past screen...	The work differs from prior methods by directly interacting with the UI through visual inputs and action sequences, rather than relying on API calls o...
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Mul...	The paper introduces Web2Code, a large-scale dataset and evaluation framework for webpage-to-code generation tasks, addressing the gap in MLLMs' abili...	Unlike existing datasets that lack instruction information or focus on general visual tasks, Web2Code integrates instruction tuning, diverse QA pairs,...
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasonin...	GuardAgent introduces a novel guardrail framework for LLM agents, enabling dynamic safety checks through knowledge-enabled reasoning and code executio...	Unlike prior guardrails focused on input/output moderation, GuardAgent introduces a task-planning and code-execution framework for safeguarding LLM ag...
Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM...	The paper introduces GovSim, a novel simulation platform to study cooperative decision-making in LLM agents, highlighting challenges in achieving sust...	This work differs by focusing on multi-agent cooperation in dynamic, realistic scenarios (GovSim) rather than single-agent benchmarks. It introduces a...
WebSuite: Systematically Evaluating Why Web Agents Fail	The paper introduces WebSuite, a diagnostic benchmark for web agents that systematically evaluates failure patterns by categorizing web actions into a...	Unlike existing benchmarks that only measure task success or failure, WebSuite introduces a taxonomy of web actions and an extensible benchmark suite...
Latent State Estimation Helps UI Agents to Reason	The paper explores how LLMs can estimate latent states in UI agents through zero-shot prompting, demonstrating significant improvements in task comple...	This work differs from related work by formalizing the use of pre-trained LLMs to estimate latent states in a textual space without task-specific fine...
Mitigating Object Hallucinations in Large Vision-Language Models with Assembly o...	The paper addresses object hallucinations in Large Vision-Language Models (LVLMs) by identifying attention deficiency as a root cause. It introduces A...	This work differs from related work by focusing on attention mechanism design rather than instruction tuning or post-hoc revisers. It proposes a dual-...
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LL...	The paper introduces a new learning paradigm called 'list items one by one' to enhance Multimodal Large Language Models' (MLLMs) ability to understand...	This work differs from related studies by focusing on a novel training paradigm ('list items one by one') and a curated dataset to enable SoM promptin...
Octopus v4: Graph of language models	The paper introduces Octopus v4, a system that integrates multiple open-source language models using functional tokens and a graph structure to optimi...	This work differs from related work by introducing a graph-based coordination framework with functional tokens to dynamically route and reformat queri...
Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent	The paper introduces a compact, on-device multimodal AI agent (Octopus v3) optimized for edge devices with <1B parameters, featuring functional tokens...	Octopus v3 differs by introducing functional tokens for AI agent-specific reasoning, optimizing for sub-1B parameters for edge deployment, and demonst...
Octo-planner: On-device Language Model for Planner-Action Agents	The paper introduces an on-device Planner-Action framework for AI agents, separating planning and action execution. It utilizes Phi-3 Mini (a 3.8B par...	The work differs from related work by focusing on on-device execution with fine-tuned LLMs (Phi-3 Mini) and multi-LoRA for multi-domain adaptability,...
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal...	Introduces V-Zen, a novel Multimodal Large Language Model (MLLM) specialized for GUI understanding and grounding, along with the GUIDE dataset. The wo...	V-Zen improves GUI grounding precision with dual-resolution image encoders and a dedicated grounding module, addressing limitations in existing MLLMs....
GUI Action Narrator: Where and When Did That Action Take Place?	The paper introduces a GUI video captioning benchmark (Act2Cap) with 4,189 samples, addressing GUI-specific challenges like dense information and rapi...	This work differs from related work by introducing a specialized GUI video captioning benchmark and framework that explicitly addresses GUI-specific c...
LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions	The paper surveys existing LLM-based single-agent and multi-agent reinforcement learning frameworks, emphasizing communication strategies, coordinatio...	This work differs from related research by focusing on the integration of LLMs into MARL frameworks, emphasizing language-conditioned communication, c...
GUIDE: Graphical User Interface Data for Execution	The paper introduces GUIDE, a novel dataset for MLLMs in RPA, emphasizing multi-platform adaptability and diverse website coverage. It presents V-Zen,...	GUIDE differs from related work by providing annotated data with action grounding, multi-platform adaptability, and diverse website coverage, while V-...
VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Prog...	The paper introduces ICAL, a method that refines suboptimal trajectories into high-quality multimodal programs of thought using VLM self-refinement an...	Unlike prior text-based methods that lack visual cues or introspection, ICAL leverages VLMs to generate causal reasoning and subgoals from noisy demon...
Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based App...	The paper introduces DCGen, a divide-and-conquer approach for generating UI code from screenshots, addressing challenges like element omission, distor...	DCGen differs from related work by introducing a segment-aware divide-and-conquer framework specifically tailored for MLLMs, improving upon CNN-based...
Scaling Instructable Agents Across Many Simulated Worlds	The paper introduces the SIMA project, which develops agents capable of following arbitrary language instructions across diverse 3D environments. Key...	Unlike prior work focused on specific games or environments, SIMA emphasizes training agents across a wide range of 3D worlds (including commercial ga...
DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Te...	DocGenome introduces a large-scale, multi-modal scientific document benchmark addressing limitations in multi-page document extraction, understanding...	DocGenome differs from related work by offering the first comprehensive dataset with structured multi-modal data (including LaTeX, layout attributes,...
Adversarial Robustness for Visual Grounding of Multimodal Large Language Models	This paper addresses the adversarial robustness of visual grounding in Multi-modal Large Language Models (MLLMs) by proposing three novel adversarial...	This work differs from related studies by focusing specifically on adversarial robustness in visual grounding tasks, whereas prior research primarily...
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business...	The paper introduces WONDERBREAD, a benchmark for evaluating multimodal foundation models (FMs) on business process management (BPM) tasks beyond auto...	Unlike existing benchmarks focused on workflow automation, WONDERBREAD addresses underexplored BPM tasks like documentation and knowledge transfer. It...
Large Language Models for Mobile GUI Text Input Generation: An Empirical Study	The paper evaluates the effectiveness of nine state-of-the-art LLMs in generating text inputs for Android GUI testing, demonstrating varying page-pass...	This work differs from related studies by conducting the first large-scale empirical evaluation of LLMs for Android text-input generation in GUI testi...
CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front...	This paper introduces a novel LLM-based GUI agent that operates solely on screenshot inputs, eliminating reliance on HTML/DOM data. It proposes Contex...	Unlike prior work relying on DOM/HTML inputs or integrated architectures, this work uses modular design with LLM-driven action planning via CAAP promp...
Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts	Prompt2Task introduces a multi-agent system for UI task automation on smartphones, translating textual prompts into operation sequences without requir...	Unlike prior UI automation systems that rely on predefined flows or detailed step-by-step instructions, Prompt2Task employs a text-prompt-driven appro...
ChatShop: Interactive Information Seeking with Language Agents	The paper introduces ChatShop, a novel task framework for evaluating language agents' strategic information-seeking capabilities in interactive scenar...	Unlike prior work focused on static information retrieval or limited reference games, ChatShop introduces a realistic, interactive benchmark that eval...
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration	The paper introduces ReAd, a framework for efficient LLM grounding in multi-agent collaboration through Reinforced Advantage feedback. It addresses in...	Unlike prior work relying on physical verification or self-reflection, ReAd introduces a principled advantage-based feedback mechanism grounded in mul...
A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipula...	The paper introduces SC-VLA, a self-correcting vision-language-action model combining fast and slow systems for robust robotic manipulation. It addres...	SC-VLA differs from prior work by explicitly addressing two key limitations: direct correction of low-level SE(3) poses and learning from correction f...
LEGENT: Open Platform for Embodied Agents	Introduces LEGENT, an open platform for embodied agents integrating LLMs and LMMs, offering a 3D environment, data generation pipeline, and demonstrat...	LEGENT differs from related work by providing an open, scalable platform that combines LMMs with embodied training, addressing limitations of existing...
AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language M...	The paper introduces AlignMMBench, a comprehensive Chinese multimodal alignment benchmark addressing gaps in existing benchmarks by focusing on nuance...	AlignMMBench differs from related work by being the first benchmark specifically designed for Chinese visual contexts, incorporating thirteen tasks ac...
Imp: Highly Capable Large Multimodal Models for Mobile Devices	Systematic exploration of lightweight LMMs through model architecture, training strategy, and training data; development of Imp models (2B-4B paramete...	Combines architectural innovations, optimized training strategies, and curated data to achieve high performance in lightweight LMMs, enabling efficien...
Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset f...	The paper introduces BRMData, a comprehensive bimanual-mobile robot manipulation dataset for household tasks, addressing limitations in existing datas...	BRMData differs from related work by providing the first dataset combining bimanual manipulation with mobile mobility, diverse task difficulty levels,...
VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fin...	The paper introduces VGA, a fine-tuned model for GUI comprehension that addresses hallucinations through image-centric fine-tuning. Key contributions...	Unlike prior work that relies on traditional GUI comprehension methods or LVLMs with limited visual alignment, VGA introduces image-centric fine-tunin...
E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion	The paper introduces E-ANT, the first Chinese GUI navigation dataset with real human behavior and high-quality annotations, aiming to improve MLLM dec...	E-ANT differs from prior work by providing a large-scale, Chinese-focused dataset with human-traced interactions and high-quality screenshots, enablin...
On AI-Inspired UI-Design	The paper explores three AI approaches to enhance UI design creativity: using LLMs for direct UI genera

Truncated — view the full README on GitHub.

Similar repos

ZJU-REAL/Awesome-GUI-Agents

A curated collection of resources, tools, and frameworks for developing GUI Agents.

460

showlab/Awesome-GUI-Agent

💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.

1,216

OS-Agent-Survey/OS-Agent-Survey

This is the repo for the paper "OS Agents: A Survey on MLLM-based Agents for Computer, Phone and…

486

trycua/acu

A curated list of resources about AI agents for Computer Use, including research papers, projects,…

1,770

Steve2457/Awesome-RL-GUI-Agents

A curated list of awesome RL in GUI Agent papers

WILLOSCAR/Awesome-HCI-LLM

Awesome-HCI （Ubiquitous, LLM, MLLM, Agent, RAG, Embodied-AI, RLHF)

ai-agents-2030/awesome-deep-research-agent

638

Bingo-W/Awesome-Agent-Search

[Survey] Agent and Tool Search: Foundations, Techniques, and Open Challenges

Contributors

harpreetsahota204

34 commits

Burhan-Q

5 commits

Languages

Python

100.0%

harpreetsahota204/gui_agent_research_landscape

README

GUI Agent Research Landscape

Overview

Data Sources

Initial Dataset

📈 Temporal Analysis of GUI Agent Research Trends

Research Pipeline

1. Data Extraction and Expansion (data_extractor.py)

2. Intelligent Filtering (filter_papers.py)

3. LLM-Powered Content Analysis (paper_analyzer.py)

4. Temporal Trend Analysis (research_trend_analyzer.py)

5. Timeline and Temporal Analysis (research_timeline_analysis.py)

6. Citation Network Analysis (citation_network_analysis.py)

Key Findings

Citation Network Analysis Results

📊 Analysis Outputs

Core Influence Rankings

Research Infrastructure Analysis

Temporal Dynamics

Visualizations

🔍 Key Insights from Citation Analysis

Field Evolution

Most Influential Works

Research Patterns

Emerging Trends

📈 Network Statistics

🎯 How to Use These Results

🔄 Analysis Methodology

Repository Structure

Usage

Requirements

Citation

Acknowledgments

License

Limitations

Papers by Time Period

2016-2021: Early Era

2022: Growth Year

2023: Q1

2023: Q2

2023: Q3

2023: Q4

2024: Q1

2024: Q2

Contributors

Similar repos

Languages

harpreetsahota204/gui_agent_research_landscape

README

GUI Agent Research Landscape

Overview

Data Sources

Initial Dataset

📈 Temporal Analysis of GUI Agent Research Trends

Research Pipeline

1. Data Extraction and Expansion (data_extractor.py)

2. Intelligent Filtering (filter_papers.py)

3. LLM-Powered Content Analysis (paper_analyzer.py)

4. Temporal Trend Analysis (research_trend_analyzer.py)

5. Timeline and Temporal Analysis (research_timeline_analysis.py)

6. Citation Network Analysis (citation_network_analysis.py)

Key Findings

Citation Network Analysis Results

📊 Analysis Outputs

Core Influence Rankings

Research Infrastructure Analysis

Temporal Dynamics

Visualizations

🔍 Key Insights from Citation Analysis

Field Evolution

Most Influential Works

Research Patterns

Emerging Trends

📈 Network Statistics

🎯 How to Use These Results

🔄 Analysis Methodology

Repository Structure

Usage

Requirements

1. Data Extraction and Expansion (`data_extractor.py`)

2. Intelligent Filtering (`filter_papers.py`)

3. LLM-Powered Content Analysis (`paper_analyzer.py`)

4. Temporal Trend Analysis (`research_trend_analyzer.py`)

5. Timeline and Temporal Analysis (`research_timeline_analysis.py`)

6. Citation Network Analysis (`citation_network_analysis.py`)

1. Data Extraction and Expansion (`data_extractor.py`)

2. Intelligent Filtering (`filter_papers.py`)

3. LLM-Powered Content Analysis (`paper_analyzer.py`)

4. Temporal Trend Analysis (`research_trend_analyzer.py`)

5. Timeline and Temporal Analysis (`research_timeline_analysis.py`)

6. Citation Network Analysis (`citation_network_analysis.py`)