Deep Research as Rubric for Reinforcement Learning

English | 中文

Official code for the paper "Deep Research as Rubric for Reinforcement Learning".

A toolkit for synthesizing evaluation data from Deep Research agents. Given a set of queries, the pipeline orchestrates multiple Deep Research frameworks to produce research reports and generates evaluation rubrics from those reports.

Overview

The core idea: use Deep Research agents as a data generation engine for evaluation and rubric synthesis.

1. Unified Router routes queries to different Deep Research frameworks (DeerFlow, Open Deep Research, DeepResearch, Simple Inference)
2. Rubric Generation takes each (query, research_report) pair and generates structured evaluation rubrics via an LLM
3. Benchmark Evaluation scores model outputs against standard benchmarks (ResearchQA, HealthBench, GPQA, MMLU, etc.)
4. SFT Data Synthesis converts research trajectories into training data, which can then be used for subsequent RL training
5. Bootstrap Loop iteratively synthesizes data, trains, and re-evaluates for self-improvement

Architecture

query CSV
  -> src/rubric_generation.py (single sample)
    -> unified_router_server -> framework (deer_flow / open_deep_research / ...)
    -> final_report
    -> openai_wrapper_server -> rubric

  -> src/benchmark_evaluation_2modes.py (single benchmark batch)
    -> predicted_answer + benchmark_eval + rubric_eval -> CSV

  -> src/run_all_benchmarks.py (all benchmarks)
    -> aggregate_summary

  -> src/dr_as_rubric_sft/synthesize_sft_data.py
    -> SFT training data -> RL training

Dataset

The companion dataset is available on HuggingFace: miiikiik/dr-as-rubric-data

It contains:

• SFT training data: 991 multi-turn trajectories in ShareGPT format for fine-tuning an 8B model
• Rubric data: Evaluation rubrics from the 4-step bootstrap loop (8B model) and from GPT-5 / Gemini 3.1 Pro

Models

The 8B model checkpoints at each step of the bootstrap loop (described in the paper) are available on HuggingFace:

• miiikiik/DR-Rubric-8B-BS-1 -- Step 1
• miiikiik/DR-Rubric-8B-BS-2 -- Step 2
• miiikiik/DR-Rubric-8B-BS-3 -- Step 3

Quick Start

1. Prerequisites

• Python 3.10+
• uv (recommended) or pip

pip install -r requirements.txt

2. Configuration

cp config.yaml.example config.yaml
cp baselines.yaml.example baselines.yaml
cp .env.example .env
# Edit these files with your API keys and model endpoints

Key settings in config.yaml:

• openai_wrapper_server: your LLM endpoint (vLLM, OpenAI API, etc.)
• tavily_search / zhipu_search: search API configuration
• benchmark_oss_path: (optional) OSS path for benchmark data
• benchmarks: paths to benchmark data files

3. Prepare Benchmark Data

Benchmark datasets are not included. Download them from their respective sources:

Place each dataset under data/<benchmark_name>/ following the paths in config.yaml.example.

4. Run the Pipeline

Start services and run end-to-end test

cd example
bash unified_router_start.sh

Generate rubrics for a query set

UNIFIED_ROUTER_CONFIG=config.yaml \
python src/rubric_generation.py \
  --query-csv-path data/my_data/your_queries.csv \
  --framework deer_flow \
  --n-parallel-tasks 8 \
  --batch-size 8 \
  --output-folder-name my_rubrics

Evaluate all benchmarks

cd example
bash run_simple_all_benchmarks.sh /path/to/your/model

Synthesize SFT training data

python src/dr_as_rubric_sft/synthesize_sft_data.py \
  --input-dir output/my_rubrics \
  --output-dir artifacts/sft_data

The synthesized data can be used for SFT training, followed by RL training to further improve model performance.

Run bootstrap self-improvement loop

python scripts/bootstrap_loop.py --config scripts/bootstrap_config.yaml

Two-Workspace Architecture

The bootstrap loop operates across two separate directories on the same machine (or cluster):

These are configured in scripts/bootstrap_config.yaml:

dr_rubric_repo: /path/to/this/repo            # where rubric generation runs
training_workspace: /path/to/training-workspace  # where RL training runs

How the Bootstrap Loop Crosses the Two Workspaces

Each bootstrap step alternates between the two workspaces:

                   dr-rubric Repo                     Training Workspace
                   ──────────────                     ──────────────────
Step N:
  1. Start vLLM with current model  ─────────────>  (model checkpoint lives here)
  2. Start framework servers
  3. Run rubric generation          ─>  output/rubric-bootstrap-*-step-N/
  4. Repair missing rubrics
  5. Merge to CSV                   ─>  data/my_data/{model}/bootstrap-step-N.csv
  6. Stop all servers
                                                     7. Generate reward config
                                                     8. Generate training script
                                                     9. Run RL training (verl)
                                                    10. Checkpoint ─> next step's model

• Steps 1--6 run inside the dr-rubric repo: they produce rubric CSVs that serve as training data.
• Steps 7--10 run inside the training workspace: they consume the rubric CSV and produce a new model checkpoint.
• The next step starts back at step 1, loading the checkpoint from the training workspace as the new model.

Setting Up the Training Workspace

The training workspace is not part of this repository. It must contain:

• reward_config/ -- reward model configs and environment plugins
• env.sh -- shell environment setup sourced by generated training scripts
• The RL training CLI (verl) must be available in $PATH

See scripts/bootstrap_config.yaml.example for the full configuration reference, including vLLM, rubric generation, and training parameters.

Project Structure

src/                          # Core pipeline code
  rubric_generation.py            # Single-sample rubric generation workflow
  benchmark_evaluation_2modes.py  # Batch benchmark data generation
  run_all_benchmarks.py           # Multi-benchmark orchestrator
  benchmark_loader.py             # Benchmark dataset loaders
  evaluation_handler.py           # Evaluation dispatcher
  config_loader.py                # Configuration loading utilities
  llm_handler.py                  # LLM call handler
  openai_wrapper_server.py        # LLM proxy with thinking removal
  unified_router_server.py        # Framework router
  deer_flow_server.py             # DeerFlow framework server
  open_deep_research_server.py    # Open Deep Research framework server
  deep_research_server.py         # DeepResearch framework server
  simple_inference_server.py      # Simple inference framework server
  tavily_mock_server.py           # Mock search server (uses LLM)
  rubric_evaluation.py            # Rubric-based evaluation
  run_baselines.py                # Baseline comparison runner
  dr_as_rubric_sft/               # SFT data synthesis

scripts/
  bootstrap_loop.py               # Iterative self-improvement loop
  bootstrap_config.yaml.example   # Bootstrap configuration template
  merge_rubric_batched_to_csv.py  # Batch merging utility
  repair_missing_rubrics.py       # Repair missing rubric entries

example/                      # Shell scripts for starting services
framework/                    # Research frameworks (with local patches)
  deer_flow/                     # ByteDance DeerFlow (patched)
  open_deep_research/            # LangChain Open Deep Research (patched)
  DeepResearch/                  # Alibaba DeepResearch (patched)
  simple_inference/              # Minimal inference adapter

prompts/                      # Jinja2 prompt templates
tests/                        # Test files

Framework Patches

The framework/ directory contains third-party Deep Research frameworks with local patches:

• deer_flow: Modified llm.py for model forwarding, added mock Tavily search, JSON truncation for repeated fields
• open_deep_research: Modified deep_researcher.py and utils.py for OpenAI wrapper integration
• DeepResearch: Modified react_agent.py and tool files for local model endpoint support

Citation

@article{mei2026dr,
  title={Deep Research as Rubric for Reinforcement Learning},
  author={Mei, Wangyi and Gu, Zhouhong and Bai, Zhenhan and Cai, Yin and Zhang, Lefan and Ding, Zhenxin and Chen, Bo and Gao, Yan and Wu, Yi and Hu, Yao and Liang, Jiaqing and Yang, Deqing},
  year={2026}
}

License

Apache License 2.0. See LICENSE.

Third-party frameworks in framework/ retain their original licenses:

• deer_flow: MIT License
• open_deep_research: MIT License
• DeepResearch: See framework/DeepResearch/LICENSE