[SIGIR 2026] Learning to Retrieve from Agent Trajectories

Retrieval is no longer optimized only for human searchers. As large language model agents increasingly issue queries, inspect snippets, browse documents, and reason over retrieved evidence, the target of retrieval training has shifted from human interaction to agent interaction. LRAT studies this paradigm shift and learns retrievers directly from multi-step agent trajectories.

Training data for agent native search should match how search agents actually search, browse, and consume evidence.

News

• 2026/08/06: Released the CCIR-B BrowseComp-Plus multi-model inference code, including the 50-call autonomous-agent contract.
• 2026/07/01: Organized a competition at CCIR CUP based on LRAT. Welcome!
• 2026/04/08: Released the full paper on arXiv.
• 2026/04/03: Paper accepted to SIGIR 2026.
• 2026/03/24: Open-sourced model checkpoints and the LRAT-Train dataset.

Introduction

LRAT studies how to train retrievers from the intermediate behaviors of strong search agents rather than from only final answers. The repository focuses on a practical pipeline for:

• collecting long-horizon search trajectories from agentic systems,
• converting trajectories into retrieval supervision,
• training retrievers on the resulting samples, and
• evaluating both retrieval quality and end-to-end task success.

Chinese Blog / 中文解读

For Chinese readers, we provide an unofficial Chinese walkthrough of LRAT covering the motivation, methodology, and experimental findings:

• Chinese Blog / 中文解读：https://mp.weixin.qq.com/s/RWfw6CA4kSwTPLV81uDMJQ?scene=1

Highlights

• Trajectory-first retrieval learning: build retriever supervision from agent search and browse traces instead of relying only on static relevance labels.
• Agent-friendly data collection: run local or API-based research agents and save each query as structured trajectory JSON.
• Training data construction with an LLM judge: turn trajectories into (query, pos, neg, ...) training pairs with reasoning-aware annotations.
• Benchmark-oriented evaluation: evaluate outputs on BrowseComp-Plus and InfoSeek-Eval with a local vLLM judge.

Resources

Dataset Statistics

The released LRAT-Train dataset contains only completed trajectories with correct final answers.

• 26,482 successful agent trajectories.
• Collected from both sparse (BM25) and dense (Qwen3-Embedding) retrievers.
• Each trajectory contains search and browse interactions performed during multi-step information seeking.

Repository Overview

Vendored Dependencies

• FlagEmbedding/ is a vendored and locally modified copy based on the upstream FlagEmbedding project. In this repository, it reflects user-side modifications layered on top of upstream work and earlier external changes.
• tevatron/ is a vendored upstream dependency used to support dense retrieval utilities and encoding workflows.
• More details are documented in THIRD_PARTY_NOTICES.md.

Supported Components

Retriever Backends

• bm25
• faiss

Agent / LLM Backends

• Alibaba-NLP/Tongyi-DeepResearch-30B-A3B
• hkust-nlp/WebExplorer-8B
• openbmb/AgentCPM-Explore
• openai/gpt-oss-120b
• OpenAI-compatible API services such as MiniMax / GLM-style endpoints

Quick Start

The quickest way to understand the repository is:

1. set up the environment,
2. build a retrieval index,
3. generate agent trajectories,
4. convert trajectories into retriever training data,
5. train a retriever, and
6. run benchmark evaluation.

1. Environment Setup

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Sync environment
uv sync

# Optional: activate the environment
source .venv/bin/activate

# Install flash-attn if needed by your environment
uv pip install --no-build-isolation flash-attn

Install Java 21 for the Lucene / Pyserini-based BM25 pipeline:

conda install -c conda-forge openjdk=21

Install the local FlagEmbedding package:

cd FlagEmbedding
pip install -e .
cd ..

2. Build Retrieval Indexes

See docs/index_construction.md for the full indexing notes.

BM25 template:

python src/index_builder.py \
  --retrieval_method bm25 \
  --corpus_path /path/to/your/corpus.jsonl \
  --save_dir /path/to/save/index

Dense embedding template via Tevatron:

CUDA_VISIBLE_DEVICES=0 python -m tevatron.retriever.driver.encode \
  --model_name_or_path /path/to/your/embedding_model \
  --dataset_path /path/to/your/corpus.jsonl \
  --encode_output_path /path/to/save/embeddings.pkl \
  --passage_max_len 512 \
  --normalize \
  --pooling <eos|mean> \
  --passage_prefix "" \
  --per_device_eval_batch_size 512 \
  --padding_side left \
  --fp16

3. Generate Agent Trajectories

See docs/trajectory_construction.md for full examples.

Example with Tongyi and a bm25 backend:

python search_agent/tongyi_client.py \
  --output-dir /path/to/output/dir \
  --searcher-type bm25 \
  --index-path /path/to/bm25/index/dir \
  --num-threads 32 \
  --model /path/to/agent_or_llm_dir \
  --snippet-max-tokens 64 \
  --query /path/to/queries.tsv \
  --port <PORT> \
  --k 10

Example with Tongyi and a faiss backend:

python search_agent/tongyi_client.py \
  --output-dir /path/to/output/dir \
  --searcher-type faiss \
  --index-path "/path/to/embeddings/index-*.pkl" \
  --model-name /path/to/embedding/model \
  --pooling <mean|eos> \
  --normalize \
  --num-threads 32 \
  --snippet-max-tokens 64 \
  --query /path/to/queries.tsv \
  --port <PORT> \
  --dataset-name /path/to/corpus_or_dataset \
  --model /path/to/agent_or_llm_dir \
  --k 10

4. Build Training Data from Trajectories

See docs/training_data_construction.md.

If you do not want to build training data from scratch, you can directly use the released LRAT-Train dataset. If you prefer to control filtering or supervision design yourself, you can also start from saved agent trajectories and rerun pair extraction with src/data_builder.py.

python src/data_builder.py \
  --corpus-path /path/to/your/corpus.jsonl \
  --traj-dir /path/to/your/trajectory_dir \
  --output-path /path/to/save/output.jsonl \
  --tokenizer-path /path/to/your/tokenizer_or_model_dir \
  --judge-api-url http://<JUDGE_HOST>:<PORT>/v1/chat/completions \
  --judge-model <JUDGE_MODEL_NAME> \
  --max-workers 32 \
  --future-timeout 30

5. Train the Retriever

The repository currently uses the local FlagEmbedding training recipe. Start from:

• FlagEmbedding/examples/finetune/embedder/run.sh
• FlagEmbedding/examples/finetune/embedder/README.md

You can plug the JSONL generated by src/data_builder.py into your existing training setup without changing the repository-level presentation structure.

6. Evaluate End-to-End Performance

See docs/evaluate.md.

python scripts_evaluation/evaluate.py \
  --input_dir /path/to/agent_output_json_dir \
  --gt_path /path/to/InfoSeek-Eval.tsv \
  --dataset_type InfoSeek-Eval \
  --output_file /path/to/save/eval_results.json \
  --model_path /path/to/local_judge_model \
  --tensor_parallel_size <NUM_GPUS> \
  --gpu_memory_utilization <GPU_MEM_UTIL> \
  --batch_size 32

7. Reproduce the CCIR competition settings

The competition-specific code is kept separate from the core LRAT pipeline:

• A-leaderboard retriever evaluation
• B-test multi-model BrowseComp-Plus inference

The B-test runner evaluates all six configured retrievers with autonomous search/get_document tool use and an explicit upper bound of 50 model calls per trajectory.

Documentation

Core Guides

Advanced Notes

Data and Outputs

• Example benchmark files are stored in datasets/.
• Query and qrel files are stored in topics-qrels/.
• Example trajectory outputs are stored in trajectory/.
• Generated run artifacts are saved as one JSON file per query by the agent clients.

Acknowledgements

This repository builds on and benefits from several excellent open-source projects and public resources:

• Alibaba-NLP/DeepResearch
• FlagOpen/FlagEmbedding
• texttron/BrowseComp-Plus
• Tevatron

License

This repository is released under the Apache License 2.0. See LICENSE.

Vendored components keep their own upstream licenses, especially:

• FlagEmbedding/ under its upstream MIT license
• tevatron/ under Apache License 2.0

Citation

If you find this repository useful, please cite our SIGIR 2026 paper below. The latest public version is available on arXiv.

@inproceedings{zhou2026lrat,
  title={Learning to Retrieve from Agent Trajectories},
  author={Zhou, Yuqi and Dai, Sunhao and Qu, Changle and Pang, Liang and Xu, Jun and Wen, Ji-Rong},
  booktitle={Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year={2026}
}