Introduction

We introduce a benchmark for evaluating LLM's agentic tool-usage capability. This benchmark treat tool use as a first‑class evaluation target. We gather high-quality and executable tools from diverse domains. Specifically, these tools exhibit capabilities beyong LLM's own ability, such as retrieve real-time information. To simulate real-world usage of agent, we collect representative tasks in each domain and generate detailed tool-calling trajectories. The queries are generated based on the trajectories and exhibits different difficulty levels (e.g. simple and hard). Trajectories are well controlled in length and bredth, enabling a controlled evaluation. The dataset can also serve as training data to enhance LLM's tool usage capability.

Dataset Sources

Repository: https://github.com/PengfeiHePower/TRAJECT-Bench

Paper: https://arxiv.org/pdf/2510.04550

Dataset Structure

tools/           # Tool definitions and APIs for each domain
parallel/        # Parallel trajectory sets
├── Education/   # Education domain
│   ├──hard_ver.json   # Hard queries
│   └──simple_ver.json # Simple queries
├── Finance/     # Domain-specific test cases
├── Travel/      # Multi-tool scenarios
└── ...          # Other domains
sequential/        # Sequential trajectory sets
├── Education/   # Education domain
│   └──traj_query.json # Trajectory and query data
├── Finance/     # Domain-specific test cases
├── Travel/      # Multi-tool scenarios
└── ...          # Other domains

Dataset Overview

🌟 Hightlights:

• 🔧 Multi-Tool Selection: Combining multiple executable, production-style tools to solve complex queries
• 🎯 Practical Task Solving: Incorporating practical tasks across diverse domains
• 🛤️ Trajectory Structure Support: First to evaluate both parallel and sequential tool-calling trajectories
• 📊 Advanced Metrics: Trajectory-aware evaluation metrics for comprehensive assessment
• 📈 Query Difficulty Control: Structured difficulty levels for progressive evaluation
• 🎲 Multiple Tool-Pool Regimes: Support for various tool pool setups, including whole toolset (mixture of tools), domain-specific tools, tool retrieval, small-scale fixed tool pool
• 🤖 Agentic Method Support: Evaluation frameworks for ReAct and other agentic approaches

¹ Trajectory structure: Evaluates support for different tool-calling patterns, including parallel (independent tools) and sequential (dependent tool chains) execution strategies
² Trajectory scaling: Tests model performance across varying task complexity levels, from simple 3-tool scenarios to complex 10+ tool orchestration
³ Trajectory-aware metrics: Provides comprehensive evaluation beyond final results, measuring the quality of the entire tool-calling trajectory
⁴ Tool-pool regimes: Supports diverse evaluation strategies including whole toolset, domain-specific pools, retrieval-based selection, and fixed small-scale tool pools

🌍 Domains Covered

The benchmark covers 10 carefully selected domains that require external tools rather than internal model capabilities:

• ✈️ Travel: Hotel booking, flight information, trip planning, transportation
• 🌤️ Weather: Forecasts, meteorological data, climate information
• 💰 Finance: Market data, trading platforms, cryptocurrency, banking
• 🗺️ Mapping: Location services, routing, geographic data
• 🎵 Music: Streaming, lyrics, artist information, music metadata
• 📰 News & Media: News articles, multimedia content, current events
• 📚 Education: Learning resources, academic data, research tools
• 📧 Email: Communication, automation, folder management
• 🎮 Gaming: Game data, statistics, account management
• 🛒 eCommerce: Online shopping, inventory, product information

📊 Dataset at a Glance (for parallel, sequential on the way)

• 🌍 10 domains: Education, Email, Finance, Gaming, Mapping, Music, News_Media, Travel, Weather, eCommerce
• 🎯 50 practical task types: 5 representative user task types for each domain
• 🛤️ 2,000 total parallel trajectories with different depth: 5 trajectory per number of tools (from 3-10) for each task
• 📝 2,000 queries per difficulty: one simple and one hard for each trajectory
• 🔢 4,000 total parallel trajectories: simple (2,000) + hard (2,000)
• 🛠️ Tool metadata: public_data/tools/*.json

🔄 Query/Trajectories Types

We take a trajectory->query strategy

1. 🔗 Parallel tool-calling trajectory: Independent tools collaborate for one task. Test sub-task planning and tool-usage capability.
2. ⏭️ Sequential tool-calling trajectory: Trajectories with strong dependency among tools, i.e. latter tools require former tools' outputs.
3. 📝 Simple Queries: Straightforward and clear instructions requiring multiple tools
4. 🧠 Hard Queries: Indirect and indicating queries that are challenging, but avoid vagueness and much openness.

📋 Query JSON Structure

Each entry in public_data/parallel/<Domain>/*.json follows this structure:

• query: Natural language instruction
• tool_list: Tool-calling trajectory for solving the query
  • tool name: <parent tool name>: <API name>
  • tool description: Brief description of the API endpoint
  • required parameters: List of objects { "name": string, "value": string }
  • optional parameters: List of objects { "name": string, "value": string } (can be empty)
  • parent tool name: Name of the tool/provider
  • API name: Endpoint name
  • domain name: Domain category
• trajectory_type: parallel or sequential
• task_name: Short title of the general task type
• task_description: Extended description of the task type

Example:

{
  "query": "First, retrieve the post with ID P36288 from FluentMe. Then, get information about the university with ID 394596. Finally, show me a list of video game adaptations from the Transmedia Storytelling database.",
  "tool_list": [
    {
      "tool name": "thefluentme: Get post by id",
      "tool description": "Returns a specific post using its post_id identifier...",
      "required parameters": [{ "name": "post_id", "value": "P36288" }],
      "optional parameters": [],
      "parent tool name": "thefluentme",
      "API name": "Get post by id",
      "domain name": "Education"
    },
    ...
  ],
  "trajectory_type": "parallel",
  "task_name": "Language and Culture Homework Helper",
  "task_description": "This task assists students with language and culture assignments. It involves finding definitions, translations, character information, and contextual examples for various languages."
}

📝 Minimal sequential example (illustrative)

{
  "query": "Find a flight from NYC to SFO next Friday and then get the 3-day weather forecast for the arrival dates in SFO.",
  "tool_list": [
    {
      "tool name": "skyscanner_flights: search_flights",
      "required parameters": [{ "name": "origin", "value": "NYC" }, { "name": "destination", "value": "SFO" }, { "name": "date", "value": "<next_friday>" }],
      "optional parameters": [],
      "parent tool name": "skyscanner_flights",
      "API name": "search_flights",
      "domain name": "Travel"
    },
    {
      "tool name": "forecast_lookup: 3day",
      "required parameters": [{ "name": "location", "value": "SFO" }, { "name": "date", "value": "<arrival_date>" }],
      "optional parameters": [],
      "parent tool name": "forecast_lookup",
      "API name": "3day",
      "domain name": "Weather"
    }
  ],
  "trajectory_type": "sequential",
  "task_name": "Travel planning with weather guardrails",
  "task_description": "Plan travel and check weather constraints based on retrieved itinerary."
}

📊 Evaluation Metrics

The benchmark evaluates models on:

• 🎯 Exact Match: Exact same selected tools with/without order consideration
• 🔍 Tool includion: Fraction of ground-truth tools that appear in the predicted trajectory
• ⚙️ Tool parameterization: Proper usage of tools, including input values and formats
• 🏆 Trajectory win rate: Given the predicted vs. reference trajectories, an LLM judge picks which better satisfies the task
• ✅ Trajectory satisfication: Judge rates whether the predicted trajectory sufficiently satisfies the query (LLM judge)
• 🎯 Solution Accuracy: Compare predicted solution with the ground truth solution (executation involved)

📊 Evaluation Results

Results of individual model’s tool-use capability with domain-specific tools on parallel queries (direct query).

Results of individual model’s tool-use capability with domain-specific tools on sequential queries.

Results of individual models combined with retrieval-based tool-selection strategy.

Evaluation of model’s inherent agentic tool-use capability.