[ACL 2025] ToolHop

[ACL 2025] ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

Data for the paper ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

Junjie Ye

jjye23@m.fudan.edu.cn

Jan. 07, 2025

Introduction

Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches.

What's New

• [2026/04/11] The leaderboard is released.
• [2025/05/15] The paper has been accepted by ACL 2025 Main Conference.
• [2025/01/07] Release the data and code for ToolHop.
• [2025/01/07] Paper available on Arxiv.

Learderboard

Int the paper, we conduct a detailed analysis of 14 LLMs, covering five distinct families.

In addition, we release a more comprehensive leaderboard that shows how the latest LLMs perform on our dataset.

Usage

Requirement

• Run the command to install the packages required.

  pip install -r requirements.txt
  

Evaluation for Open-Source LLMs

• Run the command to evaluate the Open-Source LLMs. We currently support evaluation for LLaMA3.1 and Qwen2.5 families.

  cd code
  python3 evaluation_open.py --scenario [Direct/Mandatory/Free] --series [llama31/qwen25] --model_path ${model_path} --output_file ${output_file}
  

Evaluation for Closed-Source LLMs

• Run the command to evaluate the Closed-Source LLMs. We currently support evaluation for Gemini1.5, Claude3.5, and GPT families.

  cd code
  python3 evaluation_closed.py --scenario [Direct/Mandatory/Free] --series [gemini15/claude35/gpt] --model ${model} --base_url ${base_url} --api_key ${api_key} --output_file ${output_file}
  

License

The code is licensed under the Apache License 2.0.

The ToolHop dataset is licensed under the CC BY 4.0 License.

Acknowledgement

The dataset is built upon MorehopQA.

Citation

If you find this project useful in your research, please cite:

@inproceedings{ToolHop,
  author       = {Junjie Ye and
                  Zhengyin Du and
                  Xuesong Yao and
                  Weijian Lin and
                  Yufei Xu and
                  Zehui Chen and
                  Zaiyuan Wang and
                  Sining Zhu and
                  Zhiheng Xi and
                  Siyu Yuan and
                  Tao Gui and
                  Qi Zhang and
                  Xuanjing Huang and
                  Jiecao Chen},
  editor       = {Wanxiang Che and
                  Joyce Nabende and
                  Ekaterina Shutova and
                  Mohammad Taher Pilehvar},
  title        = {ToolHop: {A} Query-Driven Benchmark for Evaluating Large Language
                  Models in Multi-Hop Tool Use},
  booktitle    = {Proceedings of the 63rd Annual Meeting of the Association for Computational
                  Linguistics (Volume 1: Long Papers), {ACL} 2025, Vienna, Austria,
                  July 27 - August 1, 2025},
  pages        = {2995--3021},
  publisher    = {Association for Computational Linguistics},
  year         = {2025},
  url          = {https://aclanthology.org/2025.acl-long.150/},
  timestamp    = {Thu, 24 Jul 2025 21:25:39 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/YeDYLXCWZXYGZ0C25.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}