Agent-STAR

This repository is the official implementation of our paper: Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe.

We use TravelPlanner as a long-horizon tool-use testbed, where agents must iteratively call tools to satisfy multifaceted constraints, i.e., commonsense and hard constraints. We implement STAR [Data Synthesis → SFT → RL], a unified post-training pipeline that systematically studies the agentic RL design space across five axes: reward shaping, model scaling, data composition, algorithm selection, and environmental stability.

Please consider citing or giving a 🌟 if Agent-STAR is helpful to your work!

@misc{wu2026agentstar,
      title={Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe}, 
      author={Xixi Wu and Qianguo Sun and Ruiyang Zhang and Chao Song and Junlong Wu and Yiyan Qi and Hong Cheng},
      year={2026},
      eprint={2603.21972},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.21972}, 
}

🎙️ News

🎉 [2026-08-22] Our paper is accepted by EMNLP 2026!

🏋️ [2026-08-22] We release the full RL training framework and scripts under RL/, covering TravelPlanner RL training and out-of-domain inference on BrowseComp-Plus and Knowledge-Intensive QA.

📑 [2026-03-24] Our paper is released on arXiv!

🌱 [2026-03-23] We release the data synthesis and inference framework of our STAR-pipeline, along with three STAR-SFT models and three STAR-RL models.

👀 Stay tuned for continuous updates, new STAR checkpoints will be added over time!

---

---

📖 Table of Contents

• 📚 Dataset
• 🔧 Environment
• 🤗 Model
• 🚀 Run Inference
• 🏋️ RL Training
• 📮 Contact
• 🙏 Acknowledgements

📚 Dataset

We open-source our data synthesis codes and 17K+ synthetic queries.

Synthetic datasets are available at https://huggingface.co/datasets/xxwu/Agent-STAR-TravelDataset.

To generate your own training samples, follow the three-step pipeline:

• Step 0 - Prepare the travel database

  • Download: https://huggingface.co/datasets/xxwu/Agent-STAR-TravelDatabase
  • Put all CSV files under database/.

• Step 1 - Element sampling

  cd DataSynthesis
  python3 step1_generate_dict.py --generate_dict --easy_num 1000 --medium_num 1000 --hard_num 1000
  

• Step 2 - Feasibility checking

  Merge the sampled files into a single file, e.g., combine_3k.jsonl, then:

  python3 step2_dict_feasibility.py --path output/combine_3k.jsonl
  

  Output files: *_feasible.jsonl and *_failed.jsonl.

• Step 3 - Query generation

  Use LLM back-translation to turn feasible JSON elements into natural language TravelPlanner queries

  Fill API keys in step3_generate_query.py first:

  python3 step3_generate_query.py --input_file output/combine_3k_feasible.jsonl --output_file output/combine_3k_final.jsonl
  

🔧 Environment

For inference, the required packages are:

vllm==0.11.0
transformers==4.57.3
torch==2.8.0

Before running the inference, you have to download the travel database, and put all the CSV files under the database folder.

If you use commercial LLMs like Gemini3-Pro, Seed-1.8 / Seed-2.0, Kimi-K2.5, DeepSeek-V3.2, Qwen3-397B-A17B, etc, fill up the corresponding API keys in Inference/config.json.

🤗 Model

We release both the STAR-SFT and STAR-RL tuned models across 1.5B / 3B / 7B scales for reproducing and further research.

Backbones: Qwen2.5-Instruct.

SFT: fine-tune from the Qwen2.5-Instruct base using 1K successful trajectories, which are generated by DeepSeek-V3.2-Exp-Thinking on synthetic queries. The corresponding SFT scripts are provided under SFTScripts/. The scripts are based on LLaMAFactory==0.9.4.dev0.

RL: we follow the paper’s scale-aware recipe, as smaller models benefit from curriculum-style rewards and exploration-heavy algorithms, while 7B can leverage GRPO with the dense SUM reward for stronger performance and faster convergence.

The figure summarizes TravelPlanner test-set success across training variants and model scales.

🚀 Run Inference

The inference code lives under Inference/. The pipeline is: ReAct inference → Post-processing [NL → structured JSON] → Evaluation.

Quick scripts

We provide ready-to-use scripts under Inference/scripts/:

• infer_sotamodel.sh: run inference with strong commercial LLMs
• infer_testset.sh: deploy a local model using vLLM and run inference on the TravelPlanner test set

Step-by-step

• Step 0: Prepare the environment

  • Put the TravelPlanner travel CSVs under database/.
  • If you use commercial LLMs, fill Inference/config.json including API endpoints and keys.

• Step 1: Run ReAct inference

  From the repo root, you can run:

  cd Inference
  python3 -u main.py \
    --model YOUR_MODEL_PATH_OR_NAME \
    --save_suffix YOUR_SUFFIX \
    --max_workers 20 \
    --split validation  # or test
    --max_context 32768 \
    --max_turns 60 
  

  If you want to run inference on your own synthesized dataset, add: --use_custom_query --input_file YOUR_FILE.jsonl.

• Step 2: Post-processing to format plans into the evaluation JSON

  Set the DashScope API key used by Inference/utils.py:

  export DASHSCOPE_API_KEY="YOUR_DASHSCOPE_API_KEY"
  python3 -u post_process.py \
    --path PREDICTIONS.jsonl \
    --split validation  # or test
    --format_model deepseek-v3.2-exp
  

  post_process.py converts the model’s natural-language itinerary into the strict JSON format used by the evaluators.

  ⚠️ IMPORTANT & 🚨 REWARD-HACKING PREVENTION

  • For test set, post_process.py runs utils.py::format_planning_for_official_test to normalize transportation, i.e., driving/... → Self-driving, so TravelPlanner’s checker matches correctly.
  • For local evaluation like validation set, we use Inference/eval_commonsense.py::detect_transportation_type for stricter transport matching.

• Step 3: Evaluation

  • Validation set:

    python3 -u eval.py --path YOUR_FORMATTED.jsonl --save_score
    

  • Test set:

    Use the official TravelPlanner leaderboard

🏋️ RL Training

The RL code lives under RL/, built on rLLM with a lightly patched verl==0.5.0 vendored in RL/verl/.

📘 Full documentation: RL Guide (English) · RL 使用文档 (中文)

The guides cover the local sandbox database, reward/API configuration, the complete variable cheat sheet, training dynamics, checkpoint export, and OOD inference.

Quick start

• Step 1: Set up the environment

  conda create -n rllm_0511 python=3.10 && conda activate rllm_0511
  cd RL
  bash scripts/install_verl.sh
  pip install -e .
  pip install swanlab
  

  RL/rllm_env.txt records a known-good, fully reproducible environment.

• Step 2: Verify tools and reward

  TravelPlanner tools read from the local sandbox at rllm/tools/travel_tools/database/. Smoke-test before training:

  export DEEPSEEK_API_KEY="YOUR_DEEPSEEK_API_KEY"   # used by the reward formatter
  python3 -m rllm.rewards.travel_reward
  python3 -m rllm.tools.travel_tools.search_restaurant
  

• Step 3: Launch training

  Pick a script, fill in the model / data / logging variables at the top, then run it. Starting from a STAR-SFT checkpoint is recommended.

  bash examples/travel/3b_scripts/train_qwen2_5_sft_3b_curriculum.sh
  

  Scripts follow the paper's scale-aware recipe — the staged Curriculum reward for smaller models, the dense Sum reward with GRPO at 7B:

  Scale	Script	Reward	Algorithm
  1.5B	examples/travel/1_5b_scripts/train_qwen2_5_sft_1_5b_curriculum.sh	Curriculum	GRPO
  3B	examples/travel/3b_scripts/train_qwen2_5_sft_3b_curriculum.sh	Curriculum	GRPO
  3B	examples/travel/3b_scripts/train_qwen2_5_sft_3b_macro.sh	Macro	GRPO
  3B	examples/travel/3b_scripts/train_qwen2_5_sft_3b_macro_arpo.sh	Macro	ARPO
  3B	examples/travel/3b_scripts/train_qwen2_5_sft_3b_macro_dapo.sh	Macro	DAPO
  7B	examples/travel/7b_scripts/train_qwen2_5_sft_7b_sum.sh	Sum	GRPO

  Reward designs are switched via REWARD_TYPE: SUM, MACRO, SUCCESS, and PLANNER_R1 for the Curriculum reward, which additionally requires TOTAL_STEPS.

• Step 4: Export the checkpoint

  cd helper
  python3 convert_safetensor.py --local_dir /PATH/TO/SAVE_FOLDER/global_step_140/actor
  

Out-of-domain inference

To check whether the tool-use ability transfers beyond TravelPlanner, we evaluate on two retrieval-heavy suites. Both need at least 2 GPUs and their own corpus/index download — see the RL Guide for details.

📮 Contact

If you have any questions about usage, reproducibility, or would like to discuss, please feel free to open an issue on GitHub or contact the authors via email at xxwu@se.cuhk.edu.hk

🙏 Acknowledgements

We sincerely thank the authors of TravelPlanner for creating a realistic and challenging benchmark, and for providing open-source resources and ongoing support for consistent evaluation. We also appreciate the open-sourced rLLM framework that supports our RL training. The first author, X. Wu, would also like to thank her previous colleagues from DeepResearch Team@Tongyi Lab for helpful hands-on experiences in agentic RL and valuable insights.