README
Introduction
Complex Function Calling Benchmark (ComplexFuncBench) is specillly designed for complex function calling evaluation. The ComplexFuncBench dataset encompass 1,000 complex function calling samples from five aspects: (1) Function calling with multi-step in single turn; (2) Function calling with user-provided constraints; (3) Function calling that requires parameter value reasoning from implicit information; (4) Function calling with long parameter values that exceed 500 tokens; and (5) Function calling with 128k long-context length.
If you wish to use this dataset for automated evaluation, please refer to our github.
Paper: https://huggingface.co/papers/2501.10132
Leaderboard
| Model | Overall Success Rate | Overall Call Acc. | Completeness | Correctness |
|---|---|---|---|---|
| Claude-3.5-Sonnet (20241022) | 61.00 | 79.27 | 1.84 | 1.85 |
| GPT-4o (2024-08-06) | 60.50 | 80.55 | 1.66 | 1.75 |
| GLM-4-Long | 57.10 | 76.35 | 1.72 | 1.74 |
| GPT-4-Turbo (2024-04-09) | 49.50 | 71.38 | 1.72 | 1.81 |
| Claude-3.5-Haiku (20241022) | 45.80 | 69.50 | 1.79 | 1.71 |
| Qwen2.5-72B | 40.10 | 58.32 | 1.80 | 1.75 |
| Mistral Large 2 | 20.10 | 48.78 | 0.94 | 1.0 |
| GLM-4-9B | 9.40 | 27.97 | 1.15 | 1.03 |
| Qwen2.5-7B | 5.0 | 18.19 | 1.5 | 1.47 |
| Llama-3.1-405B | 4.00 | 11.87 | 0.43 | 0.30 |
| Llama-3.1-70B | 2.70 | 8.17 | 0.67 | 0.36 |
| Llama-3.1-8B | 0.10 | 1.34 | 0.18 | 0.09 |
Dataset Statistics
| Hotels | Flights | Car Rental | Attraction | Cross | Total | |
|---|---|---|---|---|---|---|
| Num Samples | 150 | 150 | 150 | 150 | 400 | 600 |
| Avg. Steps | 3.33 | 3.33 | 2.87 | 2.86 | 3.5 | 3.26 |
| Avg. Calls | 4.29 | 5.33 | 4.57 | 3.6 | 6.0 | 5.07 |
Citation
If you find our work helpful for your research, please consider citing our work.
@misc{zhong2025complexfuncbench,
title={ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario},
author={Lucen Zhong and Zhengxiao Du and Xiaohan Zhang and Haiyi Hu and Jie Tang},
year={2025},
eprint={2501.10132},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.10132},
}
Contributors
Similar repos
README
Introduction
Complex Function Calling Benchmark (ComplexFuncBench) is specillly designed for complex function calling evaluation. The ComplexFuncBench dataset encompass 1,000 complex function calling samples from five aspects: (1) Function calling with multi-step in single turn; (2) Function calling with user-provided constraints; (3) Function calling that requires parameter value reasoning from implicit information; (4) Function calling with long parameter values that exceed 500 tokens; and (5) Function calling with 128k long-context length.
If you wish to use this dataset for automated evaluation, please refer to our github.
Paper: https://huggingface.co/papers/2501.10132
Leaderboard
| Model | Overall Success Rate | Overall Call Acc. | Completeness | Correctness |
|---|---|---|---|---|
| Claude-3.5-Sonnet (20241022) | 61.00 | 79.27 | 1.84 | 1.85 |
| GPT-4o (2024-08-06) | 60.50 | 80.55 | 1.66 | 1.75 |
| GLM-4-Long | 57.10 | 76.35 | 1.72 | 1.74 |
| GPT-4-Turbo (2024-04-09) | 49.50 | 71.38 | 1.72 | 1.81 |
| Claude-3.5-Haiku (20241022) | 45.80 | 69.50 | 1.79 | 1.71 |
| Qwen2.5-72B | 40.10 | 58.32 | 1.80 | 1.75 |
| Mistral Large 2 | 20.10 | 48.78 | 0.94 | 1.0 |
| GLM-4-9B | 9.40 | 27.97 | 1.15 | 1.03 |
| Qwen2.5-7B | 5.0 | 18.19 | 1.5 | 1.47 |
| Llama-3.1-405B | 4.00 | 11.87 | 0.43 | 0.30 |
| Llama-3.1-70B | 2.70 | 8.17 | 0.67 | 0.36 |
| Llama-3.1-8B | 0.10 | 1.34 | 0.18 | 0.09 |
Dataset Statistics
| Hotels | Flights | Car Rental | Attraction | Cross | Total | |
|---|---|---|---|---|---|---|
| Num Samples | 150 | 150 | 150 | 150 | 400 | 600 |
| Avg. Steps | 3.33 | 3.33 | 2.87 | 2.86 | 3.5 | 3.26 |
| Avg. Calls | 4.29 | 5.33 | 4.57 | 3.6 | 6.0 | 5.07 |
Citation
If you find our work helpful for your research, please consider citing our work.
@misc{zhong2025complexfuncbench,
title={ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario},
author={Lucen Zhong and Zhengxiao Du and Xiaohan Zhang and Haiyi Hu and Jie Tang},
year={2025},
eprint={2501.10132},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.10132},
}