Bingo-W/Awesome-Agent-Search

[Survey] Agent and Tool Search: Foundations, Techniques, and Open Challenges

14

5 commits

updated Sep 9, 2026

See the code

README

Awesome Agent Search

A curated reading list accompanying Agent and Tool Search: Foundations, Techniques, and Open Challenges

Awesome arXiv Contribution Welcome GitHub star chart License Visitors

Oryx Video-ChatGPT

📚 Citation

If you find this survey useful in your research and applications, please cite and ⭐ star to watch the repo for updates:

@article{wu2026agentsearch,
  title={Agent and Tool Search: Foundations, Techniques, and Open Challenges},
  author={Wu, Bin and Rahmani, Hossein A and Kim, To Eun and Mammadli, Arastun and 
          Qiao, Shuofei and Fu, Xiao and Ramineni, Varsha and Zhang, Xiaoyu and Meng, 
          Chuan and Drayson, George and Chowdhury, Anu and Ramos, Jerome and Maiga, 
          Abdine and Yilmaz, Emine},
  year={2026},
  publisher={Preprints}
}

[!TIP] 👋 This repository tracks the fast-moving literature on agent and tool search: how systems discover, represent, retrieve, rank, and evaluate agents and tools for a given task. If you know a paper we're missing, or work in this space yourself, PRs are very welcome (see Contributing).

Evolution of agent search research directions, 2023-2026

Evolution of research directions related to agent search from 2023 to 2026. Existing work has progressed from task understanding and planning capabilities, through representation and search mechanisms for tools and agents, toward deployment-oriented concerns such as evaluation, safety, robustness, and domain-specific applications. Together, these developments form the broader research landscape that underpins agent search systems.

Motivating agent search

Conceptual framework of agent search


📋 Table of Contents


1. Task Understanding

Transforming user tasks into actionable requirement specifications for downstream agent search — task decomposition, planning augmentation, and proactive intent clarification.

1.1 Task Decomposition

Sequential/global, iterative, and structured (tree/graph) strategies for breaking a task into sub-tasks.

PaperVenue
Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task AgentsNeurIPS 2023
AdaPlanner: Adaptive Planning from Feedback with Language ModelsNeurIPS 2023
Reasoning with Language Model is Planning with World Model (RAP)EMNLP 2023
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task PlanningNeurIPS 2023
ADAPT: As-Needed Decomposition and Planning with Language ModelsNAACL Findings 2024
TaskLAMA: Probing the Complex Task Understanding of Language ModelsAAAI 2024
TaskBench: Benchmarking Large Language Models for Task AutomationNeurIPS 2024
Can Graph Learning Improve Planning in LLM-based Agents?NeurIPS 2024
What's the Plan? Evaluating and Developing Planning-Aware Techniques for Language ModelsarXiv 2024
Understanding the Planning of LLM Agents: A SurveyarXiv 2024
Tree Search for Language Model AgentsarXiv 2024
LATS: Language Agent Tree Search Unifies Reasoning, Acting, and PlanningICML 2024
WebDART: Dynamic Decomposition and Re-planning for Complex Web TasksarXiv 2025
GAP: Graph-based Agent Planning with Parallel Tool Use and Reinforcement LearningarXiv 2025
ALAS: Transactional and Dynamic Multi-Agent LLM PlanningarXiv 2025
SelfGoal: Your Language Agents Already Know How to Achieve High-Level GoalsNAACL 2025
RECAP: Recursive Context-Aware Reasoning and Planning for LLM AgentsarXiv 2025
THREAD: Thinking Deeper with Recursive SpawningNAACL 2025
ParaCook: On Time-Efficient Planning for Multi-Agent SystemsarXiv 2025
FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized CollaborationarXiv 2025
Reason-Plan-ReAct: A Reasoner-Planner Supervising a ReAct Executor for Complex Enterprise TasksAAAI Workshop
ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task PlanningarXiv 2025
TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding TasksarXiv 2025
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use AgentsarXiv 2025
Verification-Aware Planning for Multi-Agent SystemsarXiv 2025
DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage ShapingarXiv 2025
PlanGenLLMs: A Modern Survey of LLM Planning CapabilitiesACL 2025
End-to-End Planning Framework with Agentic LLMs and PDDLarXiv 2025
ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional PruningICLR 2026

1.2 Planning Augmentation

Agentic post-training (SFT/RL), multi-agent frameworks, and planning offloading to boost planning capability.

PaperVenue
LLM+P: Empowering Large Language Models with Optimal Planning ProficiencyarXiv 2023
RestGPT: Connecting Large Language Models with Real-World RESTful APIsarXiv 2023
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyNeurIPS 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceNeurIPS 2023
FireAct: Toward Language Agent Fine-TuningarXiv 2023
BOLAA: Benchmarking and Orchestrating LLM-Augmented Autonomous AgentsarXiv 2023
MetaGPT: Meta Programming for a Multi-Agent Collaborative FrameworkICLR 2024 (Oral)
ChatDev: Communicative Agents for Software DevelopmentACL 2024
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language ModelsACL Findings 2024
AgentTuning: Enabling Generalized Agent Abilities for LLMsACL Findings 2024
AutoAct: Automatic Agent Learning from Scratch for QA via Self-PlanningACL 2024
Agent Lumos: Unified and Modular Training for Open-Source Language AgentsACL 2024
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents (ETO)ACL 2024
KnowAgent: Knowledge-Augmented Planning for LLM-Based AgentsNAACL 2024
Magentic-One: A Generalist Multi-Agent System for Solving Complex TasksarXiv 2024
Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative IntelligenceICLR 2025
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task AutomationNeurIPS 2025
ToolRL: Reward is All Tool Learning NeedsNeurIPS 2025
EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied IntelligencearXiv 2025
AgentGen: Enhancing Planning Abilities for LLM-Based Agents via Environment and Task GenerationKDD 2025
End-to-End Planning Framework with Agentic LLMs and PDDLarXiv 2025
AOrchestra: Automating Sub-Agent Creation for Agentic OrchestrationICML 2026
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent EvaluationarXiv 2025
Meta-Harness: End-to-End Optimization of Model HarnessesarXiv 2026
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent HarnessesarXiv 2026
Retrospective Harness Optimization: Improving LLM Agents via Self-Preference over Trajectory RolloutsarXiv 2026
Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task StreamsarXiv 2026
MemoHarness: Agent Harnesses That Learn from ExperiencearXiv 2026
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent WorkflowsarXiv 2026
Self-Harness: Harnesses That Improve ThemselvesarXiv 2026

1.3 Proactive User Intent Clarification

Closed-ended attribute-based and open-ended language-based clarification; agent-vs-human user intent.


2. Agent Discovery, Representation and Indexing

How agents and tools are discovered as retrieval units, represented for matching, indexed for scalable search, and kept up to date as the candidate ecosystem evolves.

2.1 Units of Discovery

API-level tools, toolkits/packages, and agents as registries/catalogs — the retrieval unit itself.

2.2 Tool Representation Surfaces

Names, natural-language docs, schemas, usage examples, and learned tool tokens/embeddings.

2.3 Indexing and Retrieval Backends

Sparse/dense indexes, rerankers, hierarchical catalogs, query rewriting, and generation-as-retrieval over tool libraries.

PaperVenue
Enhancing Tool Retrieval with Iterative Feedback from Large Language ModelsEMNLP 2023
ProTIP: Progressive Tool Retrieval Improves PlanningarXiv 2023
ToolNet: Connecting Large Language Models with Massive Tools via Tool GrapharXiv 2024
Towards Completeness-Oriented Tool Retrieval for Large Language Models (COLT)CIKM 2024
Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool RetrievalEMNLP Findings 2024
Tulip Agent: Enabling LLM-Based Agents to Solve Tasks Using Large Tool LibrariesarXiv 2024
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API CallsICML 2024
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized ToolsetsICLR 2024
Efficient and Scalable Estimation of Tool Representations in Vector Space (ToolBank)arXiv 2024
Toolken+: Improving LLM Tool Usage with Reranking and a Reject OptionEMNLP Findings 2024
Data-Efficient Massive Tool Retrieval: An RL Approach for Query-Tool Alignment (QTA)SIGIR-AP 2024
Planning and Editing What You Retrieve for Enhanced Tool Learning (PLUTO)NAACL Findings 2024
AVATAR: Optimizing LLM Agents for Tool-Assisted Knowledge RetrievalNeurIPS 2024
Benchmarking Tool Retrieval for Large Language Models (ToolRet)ACL Findings 2025
MassTool: A Multi-Task Search-Based Tool Retrieval Framework for LLMsarXiv 2025
Improving Tool Retrieval by Leveraging Large Language Models for Query GenerationCOLING Industry 2025
ToolReAGt: Tool Retrieval for LLM-Based Complex Task Solution via Retrieval Augmented GenerationACL Workshop (KnowLLM) 2025
ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM ReasoningarXiv 2025
Tool-Planner: Task Planning with Clusters across Multiple ToolsICLR 2025
Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent SystemsarXiv 2025
Tools Are Under-Documented: Simple Document Expansion Boosts Tool RetrievalICLR 2026
ToolDreamer: Instilling LLM Reasoning into Tool RetrieversACL 2026
Beyond Single-Shot: Multi-Step Tool Retrieval via Query PlanningACL 2026
Multi-Field Tool RetrievalarXiv 2026

2.4 Agent Representations

Policy/prompt, capability/profile, and routing-metadata representations; automated agent design and composition.

2.5 Maintenance: Drift, Updates, and Missing Information

Keeping representations/indexes accurate as tools, APIs, and agents evolve.


3. Retrieval, Reranking, and Selection of Agents

How candidates are ranked and selected once represented and indexed — similarity matching, LLM-based reasoning, structure-aware search, iterative retrieval, and preference-driven recommendation.

3.1 Agent/Tool Selection

Similarity matching, LLM-based (training-free/training-based) selection, and structure-aware retrieval.

3.2 Iterative Tool Retrieval

Reason-act-reflection loops and iterative query reformulation for retrieval that co-evolves with task execution.

3.3 Agent and Tool Recommendation

Mining behavioral history / preferences to personalize agent or tool selection.


Intrinsic evaluation of the search process itself, and extrinsic evaluation of downstream task outcomes.

4.1 Intrinsic Evaluation

Retrieval quality, invocation correctness, and tool/agent-use benchmarks assessed against annotated ground truth.

PaperVenue
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMsEMNLP 2023
ToolQA: A Dataset for LLM Question Answering with External ToolsNeurIPS 2023
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated CasesarXiv 2023
Can Large Language Models Be an Alternative to Human Evaluations?ACL 2023
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (STE)ACL 2024
T-Eval: Evaluating the Tool Utilization Capability of LLMs Step by StepACL 2024
MetaTool: Deciding Whether to Use Tools and Which to UseICLR 2024
Gorilla: Large Language Model Connected with Massive APIsNeurIPS 2024
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of LLMsACL Findings 2024
LLMJudge: LLMs for Relevance JudgmentsLLM4Eval Workshop (SIGIR) 2024
MCP-Zero: Active Tool Discovery for Autonomous LLM AgentsarXiv 2025
Benchmarking Tool Retrieval for Large Language Models (ToolRet)ACL Findings 2025
Judging the Judges: A Collection of LLM-Generated Relevance JudgementsSIGIR 2025
MasRouter: Learning to Route LLMs for Multi-Agent SystemsACL 2025
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsICLR 2025
τ²-Bench: Evaluating Conversational Agents in a Dual-Control EnvironmentarXiv 2025
The Berkeley Function Calling Leaderboard (BFCL)ICML 2025
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous AgentsarXiv 2025
Humanity's Last Exam (HLE)Nature 2025
DeepAgent: A General Reasoning Agent with Scalable ToolsetsWWW 2026
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP ServersarXiv 2026
The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task ExecutionICLR 2026
SealQA: Raising the Bar for Reasoning in Search-Augmented Language ModelsICLR 2026
DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable ConstraintsACL 2026
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-World ApplicationsICLR 2026

4.2 Extrinsic Evaluation

Downstream task success on agentic/deep-research/web benchmarks, incl. agent-as-a-judge evaluation.

PaperVenue
WebCanvas: Benchmarking Web Agents in Online Environments (Mind2Web-Live)Agentic Markets Workshop 2024
VisualWebArena: Evaluating Multimodal Agents on Realistic Visually Grounded Web TasksACL 2024
GAIA: A Benchmark for General AI AssistantsICLR 2024
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM AgentsNeurIPS 2024
Identifying the Risks of LM Agents with an LM-Emulated SandboxICLR 2024
G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentEMNLP 2023
A Survey on LLM-as-a-Judge (survey)arXiv 2024
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation MethodsarXiv 2024
Deep Research Comparator: A Platform for Fine-Grained Human Annotations of Deep Research AgentsarXiv 2025
ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific InquiryarXiv 2025
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-JudgeNeurIPS Datasets 2025
Agent-as-a-Judge: Evaluate Agents with AgentsICML 2025
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate ThemarXiv 2025
Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation SystemsarXiv 2025
BearCubs: A Benchmark for Computer-Using Web AgentsCOLM 2025
BrowseComparXiv 2025
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research AgentACL 2026
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing AgentsarXiv 2025
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World UsersNeurIPS 2025
DeepShop: A Benchmark for Deep Research Shopping AgentsarXiv 2025
WideSearch: Benchmarking Agentic Broad Info-SeekingICLR 2025
xBench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World EvaluationsarXiv 2025
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent TrajectoriesCOLM 2025
BrowserArena: Evaluating LLM Agents on Real-World Web Navigation TasksNeurIPS Workshop 2025
DeepResearch Bench: A Comprehensive Benchmark for Deep Research AgentsICLR 2026
Automated Rubrics for Reliable Evaluation of Medical Dialogue SystemsarXiv 2026
UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-JudgeAAAI 2026
A Survey on Agent-as-a-JudgearXiv 2026
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-JudgearXiv 2026

5. System-Level Dimensions

System-level concerns that shape whether agent search is trustworthy, fair, secure, personalized, and inspectable in practice.

5.1 Safety

PaperVenue
Evil Geniuses: Delving into the Safety of LLM-Based AgentsarXiv 2023
Testing Language Model Agents Safely in the WildarXiv 2023
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingarXiv 2024
BadAgent: Inserting and Activating Backdoor Attacks in LLM AgentsACL 2024
Towards Tool Use Alignment of Large Language Models (ToolAlign)ACL 2024
R-Judge: Benchmarking Safety Risk Awareness for LLM AgentsEMNLP Findings 2024
TrustAgent: Towards Safe and Trustworthy LLM-based AgentsEMNLP Findings 2024
SG-Bench: Evaluating LLM Safety Generalization across Diverse Tasks and Prompt TypesNeurIPS 2024
BELLS: A Framework Towards Future-Proof Benchmarks for the Evaluation of LLM SafeguardsICML Workshop 2024
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM AgentsNeurIPS 2024
Identifying the Risks of LM Agents with an LM-Emulated SandboxICLR 2024
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based AgentsNeurIPS 2024
Refusal-Trained LLMs Are Easily Jailbroken As Browser AgentsarXiv 2024
Agent-SafetyBench: Evaluating the Safety of LLM AgentsarXiv 2024
SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsarXiv 2024
SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search AgentsarXiv 2025
MiniScope: A Least Privilege Framework for Authorizing Tool Calling AgentsarXiv 2025
AgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsICLR 2025
Securing AI Agents with Information-Flow ControlarXiv 2025
OS-Harm: A Benchmark for Measuring Safety of Computer Use AgentsNeurIPS 2025
SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific DiscoveryEMNLP 2025
SafeArena: Evaluating the Safety of Autonomous Web AgentsICML 2025
NetSafe: Exploring the Topological Safety of Multi-Agent NetworksACL 2025
ToolSafety: A Comprehensive Dataset for Enhancing Safety in LLM-Based Agent Tool InvocationsEMNLP 2025
Position: Trustworthy AI Agents Require the Integration of Large Language Models and Formal MethodsICML Position 2025
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent TrajectoriesCOLM 2025
BrowserArena: Evaluating LLM Agents on Real-World Web Navigation TasksNeurIPS Workshop 2025
AgenTRIM: Tool Risk Mitigation for Agentic AIarXiv 2026
SafeSearch: Do Not Trade Safety for Utility in LLM Search AgentsEACL Findings 2026
Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM AgentsarXiv 2026
MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device ControlAAAI 2026
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent SafetyICLR 2026
Your Agent May Misevolve: Emergent Risks in Self-Evolving LLM AgentsICLR 2026
The Alignment Waltz: Jointly Training Agents to Collaborate for SafetyICLR 2026
Superficial Safety Alignment HypothesisICLR 2026
Risk-Sensitive Agent CompositionsICLR 2026

5.2 Bias and Fairness

PaperVenue
Fairness in Multi-Agent Sequential Decision-MakingNeurIPS 2014
FA*IR: A Fair Top-K Ranking AlgorithmCIKM 2017
Measuring Fairness in Ranked OutputsSSDBM 2017
Policy Learning for Fairness in RankingNeurIPS 2019
Learning Fairness in Multi-Agent SystemsNeurIPS 2019
Reducing Disparate Exposure in Ranking: A Learning to Rank Approach (DELTR)Web Conference 2020
Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankers (AdvBert)SIGIR 2021
Cooperative Multi-Agent Fairness and Equivariant PoliciesAAAI 2022
Fairness-Guided Few-Shot Prompting for Large Language ModelsNeurIPS 2023
Fair Division with Prioritized AgentsAAAI 2023
Fairness and Optimization in Dynamic Multiagent Allocation ProblemsIJCAI 2024
Fairness-Aware Exposure Allocation via Adaptive RerankingSIGIR 2024
Using Protected Attributes to Consider Fairness in Multi-Agent SystemsAEQUITAS Workshop 2024
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM InteractionsEMNLP Findings 2024
Unmasking Conversational Bias in AI Multiagent SystemsarXiv 2025
MALIBU Benchmark: Multi-Agent LLM Implicit Bias UncoveredICLR 2025
Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge RetrievalKDD 2025
Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation SystemsACL 2025
Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive ProcessesRANLP 2025
ToolTweak: An Attack on Tool Selection in LLM-Based AgentsarXiv 2025
Bias-Aware Agent: Enhancing Fairness in AI-Driven Knowledge RetrievalWeb Conference 2025
Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language ModelsFAccT 2025
FairTopia: Envisioning Multi-Agent Guardianship for Disrupting Unfair AI PipelinesarXiv 2025
Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework (MOMA)AAAI 2025
BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language ModelsICLR 2026
From Personalization to Prejudice: Bias and Discrimination in Memory-Enhanced AI Agents for RecruitmentWSDM 2026
From Biased Chatbots to Biased Agents: Examining Role Assignment Effects on LLM Agent RobustnessarXiv 2026
Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent SystemsICLR 2026

5.3 Security Risks

PaperVenue
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionAISec Workshop (CCS) 2023
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model AgentsACL Findings 2024
AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge BasesNeurIPS 2024
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled ReasoningICML 2025
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM AgentsACL 2025
Red-Teaming LLM Multi-Agent Systems via Communication AttacksACL Findings 2025
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language ModelsKDD 2025
Breaking Agents: Compromising Autonomous LLM Agents through Malfunction AmplificationEMNLP 2025
AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM AgentsEMNLP Findings 2025
Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious ToolsNeurIPS 2025
A Practical Memory Injection Attack against LLM Agents (MINJA)arXiv 2025
Persuade Me If You Can: Evaluating AI Agent Influence on Safety MonitorsICML Workshop 2025
AI Agents under Threat: A Survey of Key Security Challenges and Future PathwaysACM Computing Surveys 2025
TRiSM for Agentic AI: Trust, Risk and Security Management FrameworkarXiv 2025
Prompt Injection Attack to Tool Selection in LLM Agents (ToolHijacker)NDSS 2026
Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI AgentsICLR 2026
Optimizing Agent Planning for Security and AutonomyICLR 2026
Reliable Weak-to-Strong Monitoring of LLM AgentsICLR 2026
A2ASecBench: A Protocol-Aware Security Benchmark for Agent-to-Agent Multi-Agent SystemsICLR 2026
Breaking and Fixing Defenses against Control-Flow Hijacking in Multi-Agent SystemsICLR 2026
The Attack and Defense Landscape of Agentic AI: A Comprehensive SurveyarXiv 2026

5.4 Personalization

PaperVenue
Personalisation of Web SearchIJCAI Workshop 2003
Personalisation in Web Computing and Informatics: Theories, Techniques, Applications, and Future ResearchInformation Systems Frontiers 2010
RouteLLM: Learning to Route LLMs from Preference DataICLR 2025
A Survey of Personalization: From RAG to AgentarXiv 2025
FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real UsersarXiv 2025
PersonaAgent: When Large Language Model Agents Meet Personalization at Test TimeNeurIPS Workshop (MTI-LLM) 2025
MAPS: Motivation-Aware Personalized Search via LLM-Driven Consultation AlignmentACL 2025
Large Language Models Empowered Personalized Web Agents (PWAB / PUMA)WWW 2025
Towards Adaptive Personalized Conversational Information Retrieval (APCIR)CIKM 2025
UXAgent: An LLM-Agent-Based Usability Testing Framework for Web DesignCHI Extended Abstracts 2025
Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement LearningICLR 2025
Customer-R1: Personalized Simulation of Human Behaviors via RL-Based LLM Agent in Online ShoppingarXiv 2025
Deep Research: A Survey of Autonomous Research AgentsarXiv 2025
PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement LearningarXiv 2025
ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification QuestionsEMNLP Industry 2025
Towards Personalized Deep Research: Benchmarks and Evaluations (PDR-Bench)ICLR 2026
SPARK: Search Personalization via Agent-Driven Retrieval and Knowledge-SharingWSDM 2026
Latent Preference Modeling for Cross-Session Personalized Tool CallingarXiv 2026

5.5 Transparency and Explainability


6. Applications

6.1 Healthcare

PaperVenue
Empowering Biomedical Discovery with AI AgentsCell 2024
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-MakingNeurIPS 2024
MedAgents: Large Language Models as Collaborators for Zero-Shot Medical ReasoningACL Findings 2024
Agent Hospital: A Simulacrum of Hospital with Evolvable Medical AgentsarXiv 2024
AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical EnvironmentsarXiv 2024
MMedAgent: Learning to Use Medical Tools with Multi-Modal AgentEMNLP Findings 2024
TriageAgent: Towards Better Multi-Agent Collaborations for LLM-Based Clinical TriageEMNLP Findings 2024
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical TasksNeurIPS 2025
MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM AgentsNEJM AI 2025
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical ReasoningEMNLP 2025
Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare SafetyICML Workshop 2025
The Anatomy of a Personal Health AgentarXiv 2025
MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential DiagnosisACL 2025
ColaCare: Enhancing Electronic Health Record Modeling through LLM-Driven Multi-Agent CollaborationACM Web Conference 2025
HealthBenchOpenAI 2025
A Survey of LLM-Based Agents in Medicine: How Far Are We from Baymax?ACL Findings 2025
LLM-Based Agentic Systems in Medicine and HealthcareNature Machine Intelligence 2025
Large Language Model Agents Can Use Tools to Perform Clinical Calculationsnpj Digital Medicine 2025
Healthcare Agent: Eliciting the Power of Large Language Models for Medical Consultationnpj Artificial Intelligence 2025
MedHELM: Holistic Evaluation of Large Language Models for Medical TasksNature Medicine 2026
CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic FrameworkICLR 2026
MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical ReasoningICLR 2026
AI Agent in Healthcare: Applications, Evaluations, and Future Directionsnpj Digital Medicine 2026

6.2 Law

6.3 Deep Research


The position paper motivating agent search as a distinct problem, its companion benchmark, and the workshop this survey is affiliated with.

ResourceDescriptionLinks
AgentSearch: Indexing, Retrieval, and Ranking of AI AgentsPosition paper (SIGIR 2026) motivating agent search as a distinct research problem📄 Paper
Agent-Search Workshop @ SIGIR 2026The workshop this survey is affiliated with🌐 Website
AgentSearchBench: A Benchmark for AI Agent Search in the WildCompanion benchmark for evaluating agent search systems📄 Paper · 🌐 Project Page · 🤗 Dataset · 💻 Code

🤝 Contributing

This collection is an ongoing effort. We are actively expanding and refining its coverage, and welcome contributions from the community. You can:

  • Submit a pull request to add papers or resources
  • Open an issue to suggest additional papers or resources
  • Email us at bin.wu.23@ucl.ac.uk

We regularly update the repository to include new research on agent and tool search.

Star History Chart




Licensed under MIT.

Contributors

arastun-m

4 commits

Bingo-W

1 commits

Bingo-W/Awesome-Agent-Search

[Survey] Agent and Tool Search: Foundations, Techniques, and Open Challenges

14

5 commits

updated Sep 9, 2026

See the code

README

Awesome Agent Search

A curated reading list accompanying Agent and Tool Search: Foundations, Techniques, and Open Challenges

Awesome arXiv Contribution Welcome GitHub star chart License Visitors

Oryx Video-ChatGPT

📚 Citation

If you find this survey useful in your research and applications, please cite and ⭐ star to watch the repo for updates:

@article{wu2026agentsearch,
  title={Agent and Tool Search: Foundations, Techniques, and Open Challenges},
  author={Wu, Bin and Rahmani, Hossein A and Kim, To Eun and Mammadli, Arastun and 
          Qiao, Shuofei and Fu, Xiao and Ramineni, Varsha and Zhang, Xiaoyu and Meng, 
          Chuan and Drayson, George and Chowdhury, Anu and Ramos, Jerome and Maiga, 
          Abdine and Yilmaz, Emine},
  year={2026},
  publisher={Preprints}
}

[!TIP] 👋 This repository tracks the fast-moving literature on agent and tool search: how systems discover, represent, retrieve, rank, and evaluate agents and tools for a given task. If you know a paper we're missing, or work in this space yourself, PRs are very welcome (see Contributing).

Evolution of agent search research directions, 2023-2026

Evolution of research directions related to agent search from 2023 to 2026. Existing work has progressed from task understanding and planning capabilities, through representation and search mechanisms for tools and agents, toward deployment-oriented concerns such as evaluation, safety, robustness, and domain-specific applications. Together, these developments form the broader research landscape that underpins agent search systems.

Motivating agent search

Conceptual framework of agent search


📋 Table of Contents


1. Task Understanding

Transforming user tasks into actionable requirement specifications for downstream agent search — task decomposition, planning augmentation, and proactive intent clarification.

1.1 Task Decomposition

Sequential/global, iterative, and structured (tree/graph) strategies for breaking a task into sub-tasks.

PaperVenue
Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task AgentsNeurIPS 2023
AdaPlanner: Adaptive Planning from Feedback with Language ModelsNeurIPS 2023
Reasoning with Language Model is Planning with World Model (RAP)EMNLP 2023
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task PlanningNeurIPS 2023
ADAPT: As-Needed Decomposition and Planning with Language ModelsNAACL Findings 2024
TaskLAMA: Probing the Complex Task Understanding of Language ModelsAAAI 2024
TaskBench: Benchmarking Large Language Models for Task AutomationNeurIPS 2024
Can Graph Learning Improve Planning in LLM-based Agents?NeurIPS 2024
What's the Plan? Evaluating and Developing Planning-Aware Techniques for Language ModelsarXiv 2024
Understanding the Planning of LLM Agents: A SurveyarXiv 2024
Tree Search for Language Model AgentsarXiv 2024
LATS: Language Agent Tree Search Unifies Reasoning, Acting, and PlanningICML 2024
WebDART: Dynamic Decomposition and Re-planning for Complex Web TasksarXiv 2025
GAP: Graph-based Agent Planning with Parallel Tool Use and Reinforcement LearningarXiv 2025
ALAS: Transactional and Dynamic Multi-Agent LLM PlanningarXiv 2025
SelfGoal: Your Language Agents Already Know How to Achieve High-Level GoalsNAACL 2025
RECAP: Recursive Context-Aware Reasoning and Planning for LLM AgentsarXiv 2025
THREAD: Thinking Deeper with Recursive SpawningNAACL 2025
ParaCook: On Time-Efficient Planning for Multi-Agent SystemsarXiv 2025
FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized CollaborationarXiv 2025
Reason-Plan-ReAct: A Reasoner-Planner Supervising a ReAct Executor for Complex Enterprise TasksAAAI Workshop
ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task PlanningarXiv 2025
TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding TasksarXiv 2025
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use AgentsarXiv 2025
Verification-Aware Planning for Multi-Agent SystemsarXiv 2025
DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage ShapingarXiv 2025
PlanGenLLMs: A Modern Survey of LLM Planning CapabilitiesACL 2025
End-to-End Planning Framework with Agentic LLMs and PDDLarXiv 2025
ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional PruningICLR 2026

1.2 Planning Augmentation

Agentic post-training (SFT/RL), multi-agent frameworks, and planning offloading to boost planning capability.

PaperVenue
LLM+P: Empowering Large Language Models with Optimal Planning ProficiencyarXiv 2023
RestGPT: Connecting Large Language Models with Real-World RESTful APIsarXiv 2023
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyNeurIPS 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceNeurIPS 2023
FireAct: Toward Language Agent Fine-TuningarXiv 2023
BOLAA: Benchmarking and Orchestrating LLM-Augmented Autonomous AgentsarXiv 2023
MetaGPT: Meta Programming for a Multi-Agent Collaborative FrameworkICLR 2024 (Oral)
ChatDev: Communicative Agents for Software DevelopmentACL 2024
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language ModelsACL Findings 2024
AgentTuning: Enabling Generalized Agent Abilities for LLMsACL Findings 2024
AutoAct: Automatic Agent Learning from Scratch for QA via Self-PlanningACL 2024
Agent Lumos: Unified and Modular Training for Open-Source Language AgentsACL 2024
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents (ETO)ACL 2024
KnowAgent: Knowledge-Augmented Planning for LLM-Based AgentsNAACL 2024
Magentic-One: A Generalist Multi-Agent System for Solving Complex TasksarXiv 2024
Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative IntelligenceICLR 2025
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task AutomationNeurIPS 2025
ToolRL: Reward is All Tool Learning NeedsNeurIPS 2025
EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied IntelligencearXiv 2025
AgentGen: Enhancing Planning Abilities for LLM-Based Agents via Environment and Task GenerationKDD 2025
End-to-End Planning Framework with Agentic LLMs and PDDLarXiv 2025
AOrchestra: Automating Sub-Agent Creation for Agentic OrchestrationICML 2026
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent EvaluationarXiv 2025
Meta-Harness: End-to-End Optimization of Model HarnessesarXiv 2026
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent HarnessesarXiv 2026
Retrospective Harness Optimization: Improving LLM Agents via Self-Preference over Trajectory RolloutsarXiv 2026
Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task StreamsarXiv 2026
MemoHarness: Agent Harnesses That Learn from ExperiencearXiv 2026
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent WorkflowsarXiv 2026
Self-Harness: Harnesses That Improve ThemselvesarXiv 2026

1.3 Proactive User Intent Clarification

Closed-ended attribute-based and open-ended language-based clarification; agent-vs-human user intent.


2. Agent Discovery, Representation and Indexing

How agents and tools are discovered as retrieval units, represented for matching, indexed for scalable search, and kept up to date as the candidate ecosystem evolves.

2.1 Units of Discovery

API-level tools, toolkits/packages, and agents as registries/catalogs — the retrieval unit itself.

2.2 Tool Representation Surfaces

Names, natural-language docs, schemas, usage examples, and learned tool tokens/embeddings.

2.3 Indexing and Retrieval Backends

Sparse/dense indexes, rerankers, hierarchical catalogs, query rewriting, and generation-as-retrieval over tool libraries.

PaperVenue
Enhancing Tool Retrieval with Iterative Feedback from Large Language ModelsEMNLP 2023
ProTIP: Progressive Tool Retrieval Improves PlanningarXiv 2023
ToolNet: Connecting Large Language Models with Massive Tools via Tool GrapharXiv 2024
Towards Completeness-Oriented Tool Retrieval for Large Language Models (COLT)CIKM 2024
Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool RetrievalEMNLP Findings 2024
Tulip Agent: Enabling LLM-Based Agents to Solve Tasks Using Large Tool LibrariesarXiv 2024
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API CallsICML 2024
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized ToolsetsICLR 2024
Efficient and Scalable Estimation of Tool Representations in Vector Space (ToolBank)arXiv 2024
Toolken+: Improving LLM Tool Usage with Reranking and a Reject OptionEMNLP Findings 2024
Data-Efficient Massive Tool Retrieval: An RL Approach for Query-Tool Alignment (QTA)SIGIR-AP 2024
Planning and Editing What You Retrieve for Enhanced Tool Learning (PLUTO)NAACL Findings 2024
AVATAR: Optimizing LLM Agents for Tool-Assisted Knowledge RetrievalNeurIPS 2024
Benchmarking Tool Retrieval for Large Language Models (ToolRet)ACL Findings 2025
MassTool: A Multi-Task Search-Based Tool Retrieval Framework for LLMsarXiv 2025
Improving Tool Retrieval by Leveraging Large Language Models for Query GenerationCOLING Industry 2025
ToolReAGt: Tool Retrieval for LLM-Based Complex Task Solution via Retrieval Augmented GenerationACL Workshop (KnowLLM) 2025
ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM ReasoningarXiv 2025
Tool-Planner: Task Planning with Clusters across Multiple ToolsICLR 2025
Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent SystemsarXiv 2025
Tools Are Under-Documented: Simple Document Expansion Boosts Tool RetrievalICLR 2026
ToolDreamer: Instilling LLM Reasoning into Tool RetrieversACL 2026
Beyond Single-Shot: Multi-Step Tool Retrieval via Query PlanningACL 2026
Multi-Field Tool RetrievalarXiv 2026

2.4 Agent Representations

Policy/prompt, capability/profile, and routing-metadata representations; automated agent design and composition.

2.5 Maintenance: Drift, Updates, and Missing Information

Keeping representations/indexes accurate as tools, APIs, and agents evolve.


3. Retrieval, Reranking, and Selection of Agents

How candidates are ranked and selected once represented and indexed — similarity matching, LLM-based reasoning, structure-aware search, iterative retrieval, and preference-driven recommendation.

3.1 Agent/Tool Selection

Similarity matching, LLM-based (training-free/training-based) selection, and structure-aware retrieval.

3.2 Iterative Tool Retrieval

Reason-act-reflection loops and iterative query reformulation for retrieval that co-evolves with task execution.

3.3 Agent and Tool Recommendation

Mining behavioral history / preferences to personalize agent or tool selection.


Intrinsic evaluation of the search process itself, and extrinsic evaluation of downstream task outcomes.

4.1 Intrinsic Evaluation

Retrieval quality, invocation correctness, and tool/agent-use benchmarks assessed against annotated ground truth.

PaperVenue
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMsEMNLP 2023
ToolQA: A Dataset for LLM Question Answering with External ToolsNeurIPS 2023
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated CasesarXiv 2023
Can Large Language Models Be an Alternative to Human Evaluations?ACL 2023
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (STE)ACL 2024
T-Eval: Evaluating the Tool Utilization Capability of LLMs Step by StepACL 2024
MetaTool: Deciding Whether to Use Tools and Which to UseICLR 2024
Gorilla: Large Language Model Connected with Massive APIsNeurIPS 2024
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of LLMsACL Findings 2024
LLMJudge: LLMs for Relevance JudgmentsLLM4Eval Workshop (SIGIR) 2024
MCP-Zero: Active Tool Discovery for Autonomous LLM AgentsarXiv 2025
Benchmarking Tool Retrieval for Large Language Models (ToolRet)ACL Findings 2025
Judging the Judges: A Collection of LLM-Generated Relevance JudgementsSIGIR 2025
MasRouter: Learning to Route LLMs for Multi-Agent SystemsACL 2025
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsICLR 2025
τ²-Bench: Evaluating Conversational Agents in a Dual-Control EnvironmentarXiv 2025
The Berkeley Function Calling Leaderboard (BFCL)ICML 2025
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous AgentsarXiv 2025
Humanity's Last Exam (HLE)Nature 2025
DeepAgent: A General Reasoning Agent with Scalable ToolsetsWWW 2026
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP ServersarXiv 2026
The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task ExecutionICLR 2026
SealQA: Raising the Bar for Reasoning in Search-Augmented Language ModelsICLR 2026
DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable ConstraintsACL 2026
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-World ApplicationsICLR 2026

4.2 Extrinsic Evaluation

Downstream task success on agentic/deep-research/web benchmarks, incl. agent-as-a-judge evaluation.

PaperVenue
WebCanvas: Benchmarking Web Agents in Online Environments (Mind2Web-Live)Agentic Markets Workshop 2024
VisualWebArena: Evaluating Multimodal Agents on Realistic Visually Grounded Web TasksACL 2024
GAIA: A Benchmark for General AI AssistantsICLR 2024
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM AgentsNeurIPS 2024
Identifying the Risks of LM Agents with an LM-Emulated SandboxICLR 2024
G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentEMNLP 2023
A Survey on LLM-as-a-Judge (survey)arXiv 2024
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation MethodsarXiv 2024
Deep Research Comparator: A Platform for Fine-Grained Human Annotations of Deep Research AgentsarXiv 2025
ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific InquiryarXiv 2025
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-JudgeNeurIPS Datasets 2025
Agent-as-a-Judge: Evaluate Agents with AgentsICML 2025
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate ThemarXiv 2025
Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation SystemsarXiv 2025
BearCubs: A Benchmark for Computer-Using Web AgentsCOLM 2025
BrowseComparXiv 2025
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research AgentACL 2026
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing AgentsarXiv 2025
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World UsersNeurIPS 2025
DeepShop: A Benchmark for Deep Research Shopping AgentsarXiv 2025
WideSearch: Benchmarking Agentic Broad Info-SeekingICLR 2025
xBench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World EvaluationsarXiv 2025
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent TrajectoriesCOLM 2025
BrowserArena: Evaluating LLM Agents on Real-World Web Navigation TasksNeurIPS Workshop 2025
DeepResearch Bench: A Comprehensive Benchmark for Deep Research AgentsICLR 2026
Automated Rubrics for Reliable Evaluation of Medical Dialogue SystemsarXiv 2026
UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-JudgeAAAI 2026
A Survey on Agent-as-a-JudgearXiv 2026
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-JudgearXiv 2026

5. System-Level Dimensions

System-level concerns that shape whether agent search is trustworthy, fair, secure, personalized, and inspectable in practice.

5.1 Safety

PaperVenue
Evil Geniuses: Delving into the Safety of LLM-Based AgentsarXiv 2023
Testing Language Model Agents Safely in the WildarXiv 2023
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingarXiv 2024
BadAgent: Inserting and Activating Backdoor Attacks in LLM AgentsACL 2024
Towards Tool Use Alignment of Large Language Models (ToolAlign)ACL 2024
R-Judge: Benchmarking Safety Risk Awareness for LLM AgentsEMNLP Findings 2024
TrustAgent: Towards Safe and Trustworthy LLM-based AgentsEMNLP Findings 2024
SG-Bench: Evaluating LLM Safety Generalization across Diverse Tasks and Prompt TypesNeurIPS 2024
BELLS: A Framework Towards Future-Proof Benchmarks for the Evaluation of LLM SafeguardsICML Workshop 2024
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM AgentsNeurIPS 2024
Identifying the Risks of LM Agents with an LM-Emulated SandboxICLR 2024
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based AgentsNeurIPS 2024
Refusal-Trained LLMs Are Easily Jailbroken As Browser AgentsarXiv 2024
Agent-SafetyBench: Evaluating the Safety of LLM AgentsarXiv 2024
SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsarXiv 2024
SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search AgentsarXiv 2025
MiniScope: A Least Privilege Framework for Authorizing Tool Calling AgentsarXiv 2025
AgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsICLR 2025
Securing AI Agents with Information-Flow ControlarXiv 2025
OS-Harm: A Benchmark for Measuring Safety of Computer Use AgentsNeurIPS 2025
SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific DiscoveryEMNLP 2025
SafeArena: Evaluating the Safety of Autonomous Web AgentsICML 2025
NetSafe: Exploring the Topological Safety of Multi-Agent NetworksACL 2025
ToolSafety: A Comprehensive Dataset for Enhancing Safety in LLM-Based Agent Tool InvocationsEMNLP 2025
Position: Trustworthy AI Agents Require the Integration of Large Language Models and Formal MethodsICML Position 2025
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent TrajectoriesCOLM 2025
BrowserArena: Evaluating LLM Agents on Real-World Web Navigation TasksNeurIPS Workshop 2025
AgenTRIM: Tool Risk Mitigation for Agentic AIarXiv 2026
SafeSearch: Do Not Trade Safety for Utility in LLM Search AgentsEACL Findings 2026
Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM AgentsarXiv 2026
MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device ControlAAAI 2026
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent SafetyICLR 2026
Your Agent May Misevolve: Emergent Risks in Self-Evolving LLM AgentsICLR 2026
The Alignment Waltz: Jointly Training Agents to Collaborate for SafetyICLR 2026
Superficial Safety Alignment HypothesisICLR 2026
Risk-Sensitive Agent CompositionsICLR 2026

5.2 Bias and Fairness

PaperVenue
Fairness in Multi-Agent Sequential Decision-MakingNeurIPS 2014
FA*IR: A Fair Top-K Ranking AlgorithmCIKM 2017
Measuring Fairness in Ranked OutputsSSDBM 2017
Policy Learning for Fairness in RankingNeurIPS 2019
Learning Fairness in Multi-Agent SystemsNeurIPS 2019
Reducing Disparate Exposure in Ranking: A Learning to Rank Approach (DELTR)Web Conference 2020
Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankers (AdvBert)SIGIR 2021
Cooperative Multi-Agent Fairness and Equivariant PoliciesAAAI 2022
Fairness-Guided Few-Shot Prompting for Large Language ModelsNeurIPS 2023
Fair Division with Prioritized AgentsAAAI 2023
Fairness and Optimization in Dynamic Multiagent Allocation ProblemsIJCAI 2024
Fairness-Aware Exposure Allocation via Adaptive RerankingSIGIR 2024
Using Protected Attributes to Consider Fairness in Multi-Agent SystemsAEQUITAS Workshop 2024
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM InteractionsEMNLP Findings 2024
Unmasking Conversational Bias in AI Multiagent SystemsarXiv 2025
MALIBU Benchmark: Multi-Agent LLM Implicit Bias UncoveredICLR 2025
Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge RetrievalKDD 2025
Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation SystemsACL 2025
Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive ProcessesRANLP 2025
ToolTweak: An Attack on Tool Selection in LLM-Based AgentsarXiv 2025
Bias-Aware Agent: Enhancing Fairness in AI-Driven Knowledge RetrievalWeb Conference 2025
Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language ModelsFAccT 2025
FairTopia: Envisioning Multi-Agent Guardianship for Disrupting Unfair AI PipelinesarXiv 2025
Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework (MOMA)AAAI 2025
BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language ModelsICLR 2026
From Personalization to Prejudice: Bias and Discrimination in Memory-Enhanced AI Agents for RecruitmentWSDM 2026
From Biased Chatbots to Biased Agents: Examining Role Assignment Effects on LLM Agent RobustnessarXiv 2026
Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent SystemsICLR 2026

5.3 Security Risks

PaperVenue
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionAISec Workshop (CCS) 2023
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model AgentsACL Findings 2024
AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge BasesNeurIPS 2024
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled ReasoningICML 2025
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM AgentsACL 2025
Red-Teaming LLM Multi-Agent Systems via Communication AttacksACL Findings 2025
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language ModelsKDD 2025
Breaking Agents: Compromising Autonomous LLM Agents through Malfunction AmplificationEMNLP 2025
AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM AgentsEMNLP Findings 2025
Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious ToolsNeurIPS 2025
A Practical Memory Injection Attack against LLM Agents (MINJA)arXiv 2025
Persuade Me If You Can: Evaluating AI Agent Influence on Safety MonitorsICML Workshop 2025
AI Agents under Threat: A Survey of Key Security Challenges and Future PathwaysACM Computing Surveys 2025
TRiSM for Agentic AI: Trust, Risk and Security Management FrameworkarXiv 2025
Prompt Injection Attack to Tool Selection in LLM Agents (ToolHijacker)NDSS 2026
Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI AgentsICLR 2026
Optimizing Agent Planning for Security and AutonomyICLR 2026
Reliable Weak-to-Strong Monitoring of LLM AgentsICLR 2026
A2ASecBench: A Protocol-Aware Security Benchmark for Agent-to-Agent Multi-Agent SystemsICLR 2026
Breaking and Fixing Defenses against Control-Flow Hijacking in Multi-Agent SystemsICLR 2026
The Attack and Defense Landscape of Agentic AI: A Comprehensive SurveyarXiv 2026

5.4 Personalization

PaperVenue
Personalisation of Web SearchIJCAI Workshop 2003
Personalisation in Web Computing and Informatics: Theories, Techniques, Applications, and Future ResearchInformation Systems Frontiers 2010
RouteLLM: Learning to Route LLMs from Preference DataICLR 2025
A Survey of Personalization: From RAG to AgentarXiv 2025
FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real UsersarXiv 2025
PersonaAgent: When Large Language Model Agents Meet Personalization at Test TimeNeurIPS Workshop (MTI-LLM) 2025
MAPS: Motivation-Aware Personalized Search via LLM-Driven Consultation AlignmentACL 2025
Large Language Models Empowered Personalized Web Agents (PWAB / PUMA)WWW 2025
Towards Adaptive Personalized Conversational Information Retrieval (APCIR)CIKM 2025
UXAgent: An LLM-Agent-Based Usability Testing Framework for Web DesignCHI Extended Abstracts 2025
Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement LearningICLR 2025
Customer-R1: Personalized Simulation of Human Behaviors via RL-Based LLM Agent in Online ShoppingarXiv 2025
Deep Research: A Survey of Autonomous Research AgentsarXiv 2025
PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement LearningarXiv 2025
ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification QuestionsEMNLP Industry 2025
Towards Personalized Deep Research: Benchmarks and Evaluations (PDR-Bench)ICLR 2026
SPARK: Search Personalization via Agent-Driven Retrieval and Knowledge-SharingWSDM 2026
Latent Preference Modeling for Cross-Session Personalized Tool CallingarXiv 2026

5.5 Transparency and Explainability


6. Applications

6.1 Healthcare

PaperVenue
Empowering Biomedical Discovery with AI AgentsCell 2024
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-MakingNeurIPS 2024
MedAgents: Large Language Models as Collaborators for Zero-Shot Medical ReasoningACL Findings 2024
Agent Hospital: A Simulacrum of Hospital with Evolvable Medical AgentsarXiv 2024
AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical EnvironmentsarXiv 2024
MMedAgent: Learning to Use Medical Tools with Multi-Modal AgentEMNLP Findings 2024
TriageAgent: Towards Better Multi-Agent Collaborations for LLM-Based Clinical TriageEMNLP Findings 2024
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical TasksNeurIPS 2025
MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM AgentsNEJM AI 2025
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical ReasoningEMNLP 2025
Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare SafetyICML Workshop 2025
The Anatomy of a Personal Health AgentarXiv 2025
MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential DiagnosisACL 2025
ColaCare: Enhancing Electronic Health Record Modeling through LLM-Driven Multi-Agent CollaborationACM Web Conference 2025
HealthBenchOpenAI 2025
A Survey of LLM-Based Agents in Medicine: How Far Are We from Baymax?ACL Findings 2025
LLM-Based Agentic Systems in Medicine and HealthcareNature Machine Intelligence 2025
Large Language Model Agents Can Use Tools to Perform Clinical Calculationsnpj Digital Medicine 2025
Healthcare Agent: Eliciting the Power of Large Language Models for Medical Consultationnpj Artificial Intelligence 2025
MedHELM: Holistic Evaluation of Large Language Models for Medical TasksNature Medicine 2026
CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic FrameworkICLR 2026
MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical ReasoningICLR 2026
AI Agent in Healthcare: Applications, Evaluations, and Future Directionsnpj Digital Medicine 2026

6.2 Law

6.3 Deep Research


The position paper motivating agent search as a distinct problem, its companion benchmark, and the workshop this survey is affiliated with.

ResourceDescriptionLinks
AgentSearch: Indexing, Retrieval, and Ranking of AI AgentsPosition paper (SIGIR 2026) motivating agent search as a distinct research problem📄 Paper
Agent-Search Workshop @ SIGIR 2026The workshop this survey is affiliated with🌐 Website
AgentSearchBench: A Benchmark for AI Agent Search in the WildCompanion benchmark for evaluating agent search systems📄 Paper · 🌐 Project Page · 🤗 Dataset · 💻 Code

🤝 Contributing

This collection is an ongoing effort. We are actively expanding and refining its coverage, and welcome contributions from the community. You can:

  • Submit a pull request to add papers or resources
  • Open an issue to suggest additional papers or resources
  • Email us at bin.wu.23@ucl.ac.uk

We regularly update the repository to include new research on agent and tool search.

Star History Chart




Licensed under MIT.

Contributors

arastun-m

4 commits

Bingo-W

1 commits