To Mix or To Merge?
Toward Multi-Domain Reinforcement Learning for Large Language Models

Haoqing Wang^†, Xiang Long^†, Ziheng Li^†, Yilong Xu, Tingguang Li, Yehui Tang^✉
Samsung Research, Beijing, China   ·   Peking University

---

📰 News

• [2026.09.07] 🎉 The model checkpoints are now open-sourced on Hugging Face and ModelScope! Feel free to use our checkpoints for your post-training research (e.g., weight merging, multi-teacher on-policy distillation)!
• [2026.07.09] 🎉 Our paper is accepted to COLM 2026!

---

🔍 Overview

Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs), achieving expert-level performance in specific domains such as coding and math. When building a general multi-domain expert-level model, however, we must carefully consider how RLVR collaborates across different domains.

State-of-the-art models mainly adopt two training paradigms for multi-domain RLVR, yet few works provide a detailed comparison and analysis between them:

1. Mixed multi-task RLVR — learn from heterogeneous rewards across all domains simultaneously.
2. Separate RLVR + Model Merging — train domain-specific experts separately, then merge them via weight merging (Average, Task Arithmetic, Ties, SCE) or multi-teacher off/on-policy distillation.

We choose five commonly used high-level domains — math, coding, science, instruction following, and agent — and design extensive qualitative and quantitative experiments on open-source datasets, analyzing the internal mechanisms from the perspectives of information constraints, model prediction behavior, and self-verification.

---

✨ Key Findings

• Efficiency. Mixed multi-task RLVR achieves comparable performance with separate RLVR followed by model merging while using only 58.3% of the GPU hours. Under a matched compute budget, multi-task RLVR is even +4.15 points better on average across 9 benchmarks.
• Little interference, strong synergy. Cross-domain RLVR exhibits small mutual interference; reasoning-intensive domains (math, coding, science) are mutually synergistic, and instruction following also benefits the reasoning domains.
• Policy neighborhood transfer. Using KL divergence to analyze weight merging, we find that neighborhood policy transfer shapes domain policies toward the optimal policy, explaining the gains from merging (e.g., merging the coding expert into the math one improves math).
• Capability acquisition differs. Weight merging primarily inherits single-task capabilities, whereas multi-task training and on-policy distillation develop emergent capabilities that diverge more from single-task learning.
• Self-verification trade-off. RLVR induces an emergent self-verification ability that is highly sensitive to task characteristics and training paradigm. Agentic, multi-turn interaction is a strong catalyst for robust process-level verification, while multi-task RLVR — though efficient — causes a severe degradation in process verification.

---

🧪 Experimental Setup

Base Model: Qwen3-4B-Base  ·  RL Algorithm: GRPO  ·  Framework: slime

Pipeline: a simplified SFT → RL recipe for controllable experiments.

1. SFT on ~14M samples (one epoch) to develop basic conversational and task-solving capabilities.
2. RLVR with GRPO, per-domain or mixed multi-task, followed by model merging for the separate-training paradigm.

Domains & Datasets (open-source, from Nemotron-3 Nano):

SFT data blend follows the Nemotron-3 Nano proportions (RL blend):

---

📊 Main Results

Benchmarks (Avg@K): AIME'24 / AIME'25 (Math), LiveCodeBench v5 / v6 (Coding), HLE / GPQA-Diamond (Science), IFEval / IFBench (Instruction Following), BFCL v3 (Agent).

Evaluation across 5 domains

The highest and second-best scores are in bold and underlined.

Each single-domain RLVR model reaches state-of-the-art performance within its own domain, while both Merging and RL-Multi aggregate these gains and even improve further (e.g., AIME'24, AIME'25, HLE, IFEval, BFCL v3) — confirming the cross-domain gain effect.

Efficiency: Mix vs. Merge

Multi-task RLVR (10050.6 GPU hours) reaches comparable performance to separate RLVR + merging (5 experts, 15778.5 GPU hours) using only 58.3% of the compute. Under a matched budget (5×200-step experts vs. 1000-step multi-task), multi-task RLVR is +4.15 points better on average.

Comparison of model merging methods

TA = task arithmetic, MT-OPD/MT-OFF = multi-teacher on/off-policy distillation. On-policy distillation is only marginally better than direct weight merging but requires extra compute; different merging methods show a seesaw effect across benchmarks.

Notably, our best model trained purely on open-source data reaches performance comparable to the official Qwen3-4B (Thinking mode): 61.18 vs. 52.94 average across the 9 benchmarks.

Cross-domain synergy during training

The accuracy trajectories during each single-domain RLVR process show that the three reasoning domains (math, coding, science) stably improve each other, and instruction following further helps the reasoning domains. The agent domain benefits only when tool-use trajectories are present (i.e., jointly trained), while cross-domain interference on it remains small.

---

🔬 Mechanism Analysis

Explore Policy Neighborhoods

We cross-compare the KL divergence between domain experts and the merged multi-domain model on trajectories across domains. For a given data domain, experts from other domains can also exhibit low KL divergence with the merged policy — we formalize these as policy neighborhoods. Merging with a policy-neighborhood expert further improves the target domain (e.g., Math+Coding: AIME'24 ↑2.63, AIME'25 ↑2.02), while non-neighboring experts yield no extra gains. The relation is asymmetric (coding is a neighbor of math, but not vice versa), consistent with the asymmetry of KL divergence.

Do multi-task and merged models acquire the same skills as single-task models?

Measuring the cosine similarity between each multi-domain model's per-instance gain vector and the union of single-task gains (right figure above), we find that weight merging inherits the most single-task skills, whereas RL-Multi and MT-OPD diverge more — evidence of emergent capabilities arising from cross-domain synergy during joint training.

Dynamics of self-verification

Evaluating RL-trained models as Generative Reward Models (GenRMs) on their own trajectories reveals:

• Task characteristic dictates the verification modality. For logic-intensive tasks (math, coding), process-based verification wins; for constraint-intensive tasks (instruction following), outcome-based verification wins.
• The agent expert excels at process verification (e.g., 95.3 on AIME, 99.4 on LCB, 88.5 on IFEval), because multi-turn tool-use training forces dynamic self-monitoring.
• Robustness–efficiency trade-off. Multi-task RLVR is efficient but its average process-verification score collapses to 58.1, far behind merging methods, which decouple expert training and preserve robust verification.

Explore Weight Shift

The weight-update footprints of different domain-specific RL models significantly overlap (Jaccard ≈ 0.46–0.48 vs. ~0.18 random), and the weight-shift vectors have positive cosine similarity in overlapping regions — highest among the three reasoning domains. This holds robustly across layers and thresholds.

---

💡 Practical Guidance

• Prefer multi-task RLVR when compute is limited or emergent cross-domain capabilities are desired; prefer separate RLVR + merging when maximizing single-domain performance or preserving self-verification ability.
• To build a stronger single-domain expert, mix in data from beneficial neighboring domains (e.g., coding/STEM data helps math).
• Align the verification modality with the error locus: process-based rewards for logic-intensive tasks, outcome-based rewards for constraint-intensive tasks.

---

🚀 Reproduction

Full instructions for environment setup, data processing, SFT, single/multi-task RL, model merging, and multi-teacher distillation are in M2RL/README.md. The pipeline is verified on 32 × H800 GPUs.

---

📚 Citation

If you find this work useful, please consider citing:

@inproceedings{
wang2026to,
title={To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models},
author={Haoqing Wang and Xiang Long and Ziheng Li and Yilong Xu and Tingguang Li and Yehui Tang},
booktitle={Third Conference on Language Modeling},
year={2026},
url={https://openreview.net/forum?id=jP7j5XkG8J}
}