Empowering LLMs: Tool Learning with Real-World Interactions

This is the official repo for SIGIR 2024 tutorial: Empowering LLMs: Tool Learning with Real-World Interactions. More details can be found in https://rulegreen.github.io/services/tools-meet-llm/

We record the recent progress of tool learning based on LLMs. We list works following the structure of tutorail, and will constantly update it, welcome to raise a issue to add new works!!

Table of Contents

table of contents

• 0 Survey
• 1 Defnition and Scope of Tools
  • 1.1 Cognitive Tools
  • 1.2 Physical Tools
• 2 Components and Architecture of Tool Learning
  • 2.1 Tool Set
  • 2.2 Controller/Planner
  • 2.3 Environments
  • 2.4 Perceiver
• 3 Tool Learning based on LLMs
  • 3.1 Tool-oriented Learning
  • 3.2 Tool-augmented Learning
  • 3.3 Learning of Tool Learning
• 4 Application of Tool Learning
  • 4.1 Tool Creation, Selection and Utilization
  • 4.2 Tool Learning in IR
  • 4.3 Tool Learning in Embodied Env
• 5 Advanced Topics and Future Directions
  • 5.1 Multi-modal and Multi-agent Tool Learning
  • 5.2 Safe, Trsutworthy and Personalized Tool Learning
  • 5.3 Emerging Trends and Future Opportunities

0 Survey

• Tool Learning with Large Language Models: A Survey 2024/05

• What Are Tools Anyway? A Survey from the Language Model Perspective 2024/03

• Augmented language models: a survey

• Tool Learning with Foundation Models 2023/04

• Interactive Natural Language Processing

• A Survey on Large Language Model based Autonomous Agents

• The Rise and Potential of Large Language Model Based Agents: A Survey

• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

• Agent AI: Surveying the Horizons of Multimodal Interaction

• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security

1 Defnition and Scope of Tools

defnition and scope of tools

• 0 Cognitive Tools
• 1 Physical Tools

1.1 Cognitive Tools

relevant cognitive tools

• TPE: Towards Better Compositional Reasoning over Conceptual Tools with Multi-persona Collaboration :fire::fire::fire:

• Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues :fire::fire::fire:

• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving

• Meta Reasoning for Large Language Models

• Meta-Reasoning: Monitoring and Control of Thinking and Reasoning :fire::fire::fire::fire::fire: personally like this

1.2 Physical Tools

relevant physical tools

• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs 🔥🔥🔥 important work, also for dialogues

• Toolformer: Language Models Can Teach Themselves to Use Tools

• TravelPlanner: A Benchmark for Real-World Planning with Language Agents

• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

2 Components and Architecture of Tool Learning

2.1 Tool Set

see above

2.2 Controller / Planner

• Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees :fire::fire::fire:

• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases multi-modal

• ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models

• ART: Automatic multi-step reasoning and tool-use for large language models

• Toolformer: Language Models Can Teach Themselves to Use Tools

• ReAct: Synergizing Reasoning and Acting in Language Models

• Self-Refine: Iterative Refinement with Self-Feedback

• Teaching Large Language Models to Self-Debug

• WizardLM: Empowering Large Language Models to Follow Complex Instructions

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

• Tree of Thoughts: Deliberate Problem Solving with Large Language Models

• Knowledge-enhanced Agents for Interactive Text Games

• SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks

• AdaPlanner: Adaptive Planning from Feedback with Language Models

• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

• Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach

• User Behavior Simulation with Large Language Model based Agents

• Towards A Unified Agent with Foundation Models

• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

• PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

• Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning

• ExpeL: LLM Agents Are Experiential Learners

• ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

• Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives

• Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

• AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning

• SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents

• AutoGuide: Automated Generation and Selection ofState-Aware Guidelines for Large Language Model Agents

2.3 Environments / Benchmarks

• ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios

• [CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions] chinese benchmark

• TOOLTALK: EVALUATING TOOL USAGE IN A CONVERSATIONAL SETTING

• MINT: Evaluating llms in multi-turn interaction with tools and language feedback

• Metatool benchmark for large language models: Deciding whether to use tools and which to use

2.4 Perceiver

• Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives

• Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

• CRITIC: LARGE LANGUAGE MODELS CAN SELFCORRECT WITH TOOL-INTERACTIVE CRITIQUING first to use external feedback from tools to critic/refine outputs of LLMs? [code]

• Reflexion: Language Agents with Verbal Reinforcement Learning

• Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

3 Tool Learning based on LLMs

3.1 Tool-oriented Learning

• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

3.2 Tool-augmented Learning

• JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

• Chain of Code: Reasoning with a Language Model-Augmented Code Emulator

• KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

• ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models

3.3 Learning of Tool Learning

• START: Self-taught Reasoner with Tools

• GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

• Making Language Models Better Tool Learners with Execution Feedback

4 Application of Tool Learning

4.1 Tool Creation Selection and Utilization

Tool Creation

• LARGE LANGUAGE MODELS AS TOOL MAKERS

• Craft: Customizing llms by creating and retrieving from specialized toolsets

• CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

Tool Selection and Utilization

• T1: Tool-integrated Self-verification for Test-time Compute Scaling in Small Language Models

• EASYTOOL: ENHANCING LLM-BASED AGENTS WITH CONCISE TOOL INSTRUCTION optimizate tool documentation

• TOOLVERIFIER: Generalization to New Tools via Self-Verification finetune a model to select tool based on desc, and propose questions to self-refine decisions

• TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems ICLR2024

• CRUXEval: A Benchmark for Code Reasoning Understanding and Execution

• Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval

• Empowering Large Language Model Agents through Action Learning

4.2 Tool Learning in IR

• UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems :fire::fire::fire:

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection :fire::fire::fire::fire::fire:

• UniRetriever: Multi-task Candidates Selection for Various Context-Adaptive Conversational Retrieval

• Active Retrieval Augmented Generation EMNLP 2023 :fire::fire::fire: interesting and useful -> may can be used in dialogues

• I^3 Retriever: Incorporating Implicit Interaction in Pre-trained Language Models for Passage Retrieval

• Learning Retrieval Augmentation for Personalized Dialogue Generation EMNLP 2023

• PK-ICR: Persona-Knowledge Interactive Multi-Context Retrieval for Grounded Dialogue EMNLP 2023

• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions ACL 2023

• Self-Knowledge Guided Retrieval Augmentation for Large Language Models EMNLP 2023

4.3 Tool Learning in Embodied Environment

• A Solution-based LLM API-using Methodology for Academic Information Seeking

• Voyager: An Open-Ended Embodied Agent with Large Language Models 2023.05

• SCIENCEWORLD: Is your Agent Smarter than a 5th Grader? 22.03 Interactive Env

• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning 2020.10

• VirtualHome: Simulating Household Activities via Programs

• On the tool manipulation capability of open-source large language models

• Code as Policies: Language Model Programs for Embodied Control

• Plan4MC: Skill Reinforcement Learning and Planning for Open-World Minecraft Tasks

• Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents

• Language Models Meet World Models: Embodied Experiences Enhance Language Models

• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

• Reasoning with Language Model is Planning with World Model

• An Embodied Generalist Agent in 3D World

4.4 Tool Learning for All

• FACTOOL: Factuality Detection in Generative AI A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios [code]

• Tool-Augmented Reward Modeling

5 Advanced Topic and Future Directions

defnition and scope of tools

• 0 Multi-modal and Multi-agent Tool Learning
• 1 Safe, Trustworthy and Personalized Tool Learning
• 2 Emerging Trends and Future Opportunities

5.1 Multi-modal and Multi-agent Tool Learning

• AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction :fire::fire::fire::fire::fire:

• Learning to Use Tools via Cooperative and Interactive Agents with Large Language Models :fire::fire:

• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

• [OS-Copilot: Towards Generalist Computer Agents with Self-Improvement]

• [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents]

• Scaling Large-Language-Model-based Multi-Agent Collaboration multi-agent

• Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

• MOBILE-AGENT: AUTONOMOUS MULTI-MODAL MOBILE DEVICE AGENT WITH VISUAL PERCEPTION multi-modal

• WEBARENA: A REALISTIC WEB ENVIRONMENT FOR BUILDING AUTONOMOUS AGENTS multi-modal

5.2 Safe, Trustworthy and Personalized Tool Learning

• RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning

• R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

• GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning

• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

• INJECAGENT: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

• IDENTIFYING THE RISKS OF LM AGENTS WITH AN LM-EMULATED SANDBOX

• SECGPT: An Execution Isolation Architecture for LLM-Based Systems :fire::fire::fire::fire:

• [Towards Tool Use Alignment of Large Language Models] safety and autonmy

• ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages ACL 2024

• Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agent

5.3 Emerging Trends and Future Opportunities

• Alignment for Efficient Tool Calling of Large Language Models only one tool for one problem, just one call

• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger

• SMART: Self-Aware Agent for Tool Overuse Mitigation

• Self-DC: When to retrieve and When to generate? Self Divide-and-Conquer for Compositional Unknown Questions NAACL 2025 Oral :fire::fire::fire:

• Knowledge Conflicts for LLMs: A Survey EMNLP 2024

• Metacognitive Retrieval-Augmented Large Language Models tools conflicts

• TORA: A TOOL-INTEGRATED REASONING AGENT FOR MATHEMATICAL PROBLEM SOLVING :fire::fire::fire: [code]

5.4 Tool as Basic Unit in Reinforcement Learning

• TORL: Scaling Tool-Integrated RL

@inproceedings{toolmeetllm,
        author = {Wang, Hongru and Qin, Yujia and Lin, Yankai and Pan, Jeff Z. and Wong, Kam-Fai},
        title = {Empowering Large Language Models: Tool Learning for Real-World Interaction},
        year = {2024},
        isbn = {9798400704314},
        publisher = {Association for Computing Machinery},
        address = {New York, NY, USA},
        url = {https://doi.org/10.1145/3626772.3661381},
        doi = {10.1145/3626772.3661381},
        abstract = {Since the advent of large language models (LLMs), the field of tool learning has remained very active in solving various tasks in practice, including but not limited to information retrieval. This half-day tutorial provides basic concepts of this field and an overview of recent advancements with several applications. In specific, we start with some foundational components and architecture of tool learning (i.e., cognitive tool and physical tool), and then we categorize existing studies in this field into tool-augmented learning and tool-oriented learning, and introduce various learning methods to empower LLMs this kind of capability. Furthermore, we provide several cases about when, what, and how to use tools in different applications. We end with some open challenges and several potential research directions for future studies. We believe this tutorial is suited for both researchers at different stages (introductory, intermediate, and advanced) and industry practitioners who are interested in LLMs and tool learning.},
        booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
        pages = {2983–2986},
        numpages = {4},
        keywords = {language agents, large language models, tool learning},
        location = {Washington DC, USA},
        series = {SIGIR '24}
}