A curated paper list on On-Policy Distillation (OPD) & On-Policy Self-Distillation (OPSD) for LLMs — student rollouts + teacher feedback, covering white-box/black-box methods, OPD-RL hybrids, agentic & multimodal applications, and industrial multi-teacher recipes.
24
4 commits
updated Aug 6, 2026
README
Awesome On-Policy Distillation (OPD)
A curated paper list on On-Policy Distillation (OPD) — the student model samples its own rollouts, and a teacher model provides feedback on them, eliminating the exposure bias of offline distillation.
- 🎯 OPD core requirements:
C1the student samples its own rollouts during training,y ~ π_student(·|x);C2the teacher provides token- or sequence-level feedback on those student samples. Both conditions must hold for a strict definition of OPD. - 🪞 OPSD (On-Policy Self-Distillation) is a special case of OPD: the teacher is the same model, but conditioned on privileged information (the correct answer / a longer context / an earlier checkpoint).
- 📚 This repo tags each paper with one or more technical directions (white-box / black-box / PI-based OPSD / OPD×RL hybrid / CoT / multimodal / agent / industrial multi-teacher), plus a one-line method summary and key experimental numbers.
- ⚠️ Content is compiled from close readings of paper PDFs. PRs to fix errors are welcome.
Table of Contents
Surveys
| Paper | Summary | Link |
|---|---|---|
| A Survey of On-Policy Distillation for LLMs (2026.04) | The first systematic survey; proposes a 3D taxonomy (feedback signal × teacher access × loss granularity) and outlines 9 open research directions | arXiv 2604.00626 |
Direction Overview
| Direction | Description |
|---|---|
| 🔬 White-Box OPD | Teacher logits are accessible; direct KL / JSD distillation |
| 🎭 Black-Box OPD | Only discrete teacher feedback available (verbal score / ranking / verification result), no logits |
| ♻️ PI-based OPSD | Teacher is the same model but conditioned on privileged information (PI) |
| 🤝 OPD × RL Hybrid | OPD signal fused with RLVR advantage/reward during training |
| 🧠 CoT OPD | Distillation targeting long chain-of-thought reasoning patterns |
| 🖼️ Multimodal OPD | Vision-language / image generation and other multimodal settings |
| 🤖 Agent / Embodied OPD | Multi-turn agent, embodied AI, and other long-horizon decision-making settings |
| 🏭 Industrial Multi-Teacher | Multi-domain / multi-teacher distillation in industrial practice |
Methods
| Paper | Direction | Summary | Link |
|---|---|---|---|
| KDRL (2025.06) | OPD × RL Hybrid | Jointly optimizes GRPO (RL direction) and KD-RKL (reverse-KL distillation direction) in a single loss, with β-annealing and reward-guided masking; +2.6% vs GRPO with 10% fewer distillation tokens, showing joint training beats sequential staging | arXiv 2506.02208 |
| SCoRe (2025.09) | White-Box OPD | Student generates rollouts, teacher only corrects the earliest error step (aligns only the first deviation); a 7B model approaches 72B-teacher performance on 12 benchmarks | arXiv 2509.14257 |
| DDT / On-Policy SFT (2026.02) | White-Box OPD | Uses signal-detection theory to derive CLL = log p + H[p] as the optimal criterion for distinguishing on-policy vs. OOD tokens; IDFT dynamically down-weights OOD-token gradients via exp(−φ); Hinted Decoding adaptively mixes distributions in log-space by imitator entropy to align training data; HD+IDFT surpasses DPO/SimPO at SFT-level GPU cost | arXiv 2602.12222 |
| SDPO (2026.01) | OPD × RL Hybrid | Uses the model itself as a self-teacher under feedback conditioning (RLVR environment feedback plus correct outputs in the same batch), providing dense token-level KL signal; beats GRPO with 4× fewer generations | arXiv 2601.20802 |
| OVD (2026.01) | Black-Box OPD · Agent OPD | Replaces token-logit matching with discrete verbal scores (0–9); on web tasks an environment agent simulates a search engine and produces the score, on math tasks the teacher model scores directly; 48,000× memory reduction, up to +26% on math reasoning | arXiv 2601.21968 |
| PID (2026.02) | PI-based OPSD · Agent OPD | Proposes a privileged-information distillation paradigm for agentic tasks with "observable behavior, hidden reasoning"; both π-Distill (joint teacher-student parameter training) and OPSD (on-policy RL + reverse-KL penalty) surpass the SFT+RL industry standard | arXiv 2602.04942 |
| SSD (2026.04) | Black-Box OPD | No verifier/teacher/RL needed — SFT on the model's own high-temperature truncated-sampling outputs; adaptively reshapes the token distribution via an "accuracy-exploration conflict" mechanism; Qwen3-30B pass@1 on LiveCodeBench v6 rises from 42.4% to 55.3% (+12.9pp) | arXiv 2604.01193 |
| Skill-SD (2026.04) | White-Box OPD · Agent OPD | Dynamically extracts a skill-conditioned teacher from the student's own historical trajectories (asymmetric prompting), combined with importance-weighted reverse KL + dynamic parameter sync; +14.0%/+10.9% over vanilla GRPO and +42.1%/+40.6% over vanilla OPD on AppWorld/Sokoban | arXiv 2604.10674 |
| SCOPE (2026.04) | White-Box OPD | Diagnoses the OPD Pass@k paradox (PSR/OPD raises Pass@1 but drops Pass@32, i.e., mode collapse) plus a prefix trap (high teacher PPL means the teacher can't parse the student's bad prefix either); routes by correctness — correct samples go through a student MLE self-bootstrap path, incorrect ones through a teacher-KL consultation path, using the same PPL in opposite directions on each path; +11.42% Avg@32, +7.30% Pass@32 — the only method to improve both simultaneously | arXiv 2604.10688 |
| Lightning OPD (2026.04) | White-Box OPD · CoT OPD | Proposes a "teacher consistency" principle (SFT and OPD must use the same teacher or gradients become biased), and accordingly moves the teacher to offline pre-computation; 4× training efficiency gain, a single node can train a 30B model to 71.0% on AIME 2024 | arXiv 2604.13010 |
| StableOPD (2026.04) | White-Box OPD | Diagnoses OPD's truncation collapse: repeated tokens receive 4–9× the reverse-KL advantage, on-policy sampling forms a self-reinforcing loop, and within ~30 steps the truncation rate goes 0.5→1.0, repetition rate 0→0.6, with accuracy crashing in sync; fix is mixture distillation (blending in gold solutions as an SFT distribution anchor) + a KL gate (limiting policy drift from the reference model); ablations show the gold anchor contributes +6.0 vs. only +1.7 from the KL gate; 1.5B student scores 36.1 vs. OPD 28.9 / SFT 31.9 / GRPO 30.1, and the 7B student's 47.6 beats all zero-style RLVR; OPD is actually worse than SFT here (collapse makes a strong signal less reliable than a stable weak one) | arXiv 2604.08527 |
| OPD Recipe (2026.04) | White-Box OPD · Agent OPD | Systematically analyzes when OPD succeeds/fails: two preconditions (compatible thinking patterns + a teacher with genuinely new capability), token-level progressive alignment signal, off-policy cold-start remedies, and the root cause of signal degradation on long sequences (>7K tokens), which stems from agent environments producing very long sequences where OPD's entropy signal from the teacher becomes unreliable deep into those sequences | arXiv 2604.13016 |
| REOPOLD (2026.03) | White-Box OPD | Reinterprets OPD as RL (log-probability ratio = token reward), fixing three issues with the RL toolbox: a mixture-distribution-derived reward-clipping floor log(λ/(1−λ)) to prevent gradient explosion, entropy-guided masking that skips 60% of near-zero-reward tokens, and a two-stage explore→refine learning-mode switch; 7B student gets 3.32× faster inference vs. a 32B teacher, and vanilla RKL collapses at matched teacher-student scale while REOPOLD remains unaffected | arXiv 2603.11137 |
| OPD Fixes (2026.03) | White-Box OPD | Identifies three engineering flaws in sampled-token OPD (imbalanced signal / unreliable teacher guidance / tokenizer mismatch), and proposes Teacher Top-K Local Support Matching (truncated reverse KL + support-set normalization + top-pp sampling + special-token masking); observes that the teacher's probability on a given token is usually lower than the student's, so the fix for the long-tail is to forcibly compute the student's probability over the teacher's top-k, effectively pulling the student toward the teacher off-policy; +19.8% in a multi-task setting | arXiv 2603.25562 |
| RLAD (2026.02) | White-Box OPD · OPD × RL Hybrid | Folds the teacher policy into the GRPO importance ratio (geometric mean) to construct a mixed anchor TRRD: r^TRRD = (r^GRPO)^α × (r^T)^(1−α), implicitly equivalent to α·KL(student‖old policy)+(1−α)·KL(student‖teacher); selective imitation automatically aligns with the advantage direction, removing objective interference; α=0.5 works well by default across tasks; +5.5pp on math reasoning, AIME24 Pass@32 77.8→85.4, with only +12% extra training overhead | arXiv 2602.22495 |
| G-OPD / ExOPD (2026.02) | White-Box OPD · OPD × RL Hybrid · Industrial Multi-Teacher | Proves OPD is a special case of dense KL-constrained RL (with an implicit reward = teacher/reference log-probability ratio); introduces a reward scale λ: λ=1.25 (ExOPD) lets the student extrapolate along the teacher's improvement direction, surpassing the teacher; supports multi-teacher merging and strong-to-weak distillation reward correction | arXiv 2602.12125 |
| EOPD (2026.03) | White-Box OPD | Identifies a dual failure of reverse KL at high-entropy positions (unstable gradients + diversity collapse to 6.8%), proposing entropy-aware switching: reverse KL at low entropy, forward-KL top-K approximation at high entropy; +5.05 Pass@8 on Qwen3-4B | arXiv 2603.07079 |
| NPO (2026.04) | OPD × RL Hybrid · Multimodal OPD | Replaces one rollout with a near-future checkpoint (Δ* steps ahead) from the same training run, optimizing S=Q/V to balance trajectory quality and variance; AutoNPO auto-detects plateaus and selects the optimal Δ*; +5.27 average across 8 benchmarks on Qwen3-VL-8B, 2.1× faster convergence | arXiv 2604.20733 |
| SRPO (2026.04) | OPD × RL Hybrid | Routes by correctness: correct samples reinforced via GRPO, incorrect samples distilled via SDPO (using correct answers from the same batch as the teacher); an entropy-aware exponential weight exp(−β·H_t) suppresses contributions at teacher-uncertain positions; the mixing ratio adapts automatically to the success rate; +3.4pp vs. GRPO and +6.3pp vs. SDPO across 5 benchmarks on Qwen3-8B, with −17.2% compute cost; key finding: SDPO applied only to correct samples is actually worse than GRPO | arXiv 2604.02288 |
| RLSD (2026.04) | OPD × RL Hybrid · Multimodal OPD | Reframes OPSD's self-distillation signal from "distribution matching" to "magnitude modulation": the environmental reward (RLVR) determines the update direction (sign(A)), while the teacher/student log-probability ratio Δ (OPD) determines the token-level update magnitude; mathematically proves OPSD suffers from an unremovable mutual-information leak causing KL stagnation; Qwen3-VL-8B averages 56.18% across five benchmarks, +2.32pp over GRPO and +3.69pp over OPSD | arXiv 2604.03128 |
| GLM-5 (2026.02) | OPD × RL Hybrid · Agent OPD · Industrial Multi-Teacher | Industrial-scale OPD deployment within multi-stage RL post-training: uses earlier-stage checkpoints as teachers, embedding the teacher-student log-probability ratio directly into the advantage function for cross-stage anti-forgetting and capability fusion; agentic RL uses an asynchronous decoupled framework (slime) + TITO gateway + two-sided importance sampling supporting 1000+ concurrent rollouts; #1 open-source on Vending-Bench 2, 50 points on the Artificial Analysis Intelligence Index v4.0 | arXiv 2602.15763 |
| MiMo-V2-Flash (2026.01) | Agent OPD · Industrial Multi-Teacher | Proposes multi-teacher on-policy distillation (MOPD): domain-specific teachers are each trained independently via RL, then distilled on-policy into a single student using token-level advantage (teacher/student log-probability ratio + ORM reward), with clipped importance weights to prevent off-policy drift, and each teacher distills only prompts from its own domain; multiple types of agentic RL (code/search/web/terminal) plus R3 routing replay resolve MoE expert-routing inconsistency; 73.4% on SWE-Bench Verified, the highest among open-source models | arXiv 2601.02780 |
| Nemotron-Cascade 2 (2026.03) | OPD × RL Hybrid · Agent OPD · Industrial Multi-Teacher | Proposes a Cascade RL + MOPD post-training framework: MOPD uses historical pipeline checkpoints as teachers (Math/RLHF/Multi-domain categories), with token-level advantage = teacher-student log-probability ratio and clipping weights ε=0.5/2.0, acting as a "stable anchor" between RL stages to recover from capability drift, converging 3× faster than GRPO; SWE RL supports both Agentless and Execution-based modes, and joint training transfers across frameworks; the 30B-A3B model achieves gold-medal level on IMO/IOI/ICPC | arXiv 2603.19220 |
| HDPO (2026.03) | OPD × RL Hybrid | Addresses GRPO's "cliff" problem (all rollouts fail, advantage is zero, no gradient): the same model is given the correct answer "open-book" to generate a solution, R=1 filters correct trajectories, and top-k=64 JSD distills them back into the "closed-book" model itself; identical teacher/student weights (differing only in input) bound the realizability gap; drifting λ=0.01 barely hurts pass@1 while pass@4 gains +1.1pp; λ=0.1 pushes pass@8 to 0.84 (+1.7pp) at a cost of −2.3pp pass@1; counter-intuitive: a frozen teacher, by preserving pre-training diversity, actually beats the drifting teacher on pass@8 | arXiv 2603.23871 |
| MAD-OPD (2026.05) | White-Box OPD · Agent OPD · Industrial Multi-Teacher | Uses multi-teacher debate (R=2 rounds) to break through the single-teacher capability ceiling: debate history is injected as privileged information via force-decoding, and confidence-based softmax weighting merges the distributions; agentic tasks use JSD, code generation uses reverse KL; paired with OPAD (on-policy sampling + environment interaction at every step) to eliminate long-trajectory error accumulation; ranks first across all 6 teacher-student configurations, and a 4B student beats its 14B teacher by +4.26% on a code benchmark | arXiv 2605.01347 |
| Uni-OPD (2026.05) | White-Box OPD · Industrial Multi-Teacher · Multimodal OPD | Fixes two major OPD pain points from dual perspectives: offline difficulty-awareness plus online correctness-balancing addresses insufficient exploration, and margin calibration (Mask/Shift) uses outcome rewards to correct ordering errors in the teacher signal; on Qwen3-4B multi-teacher settings, 48.7 vs. 47.0 on math and 63.6 vs. 60.2 on code, with consistent gains across LLM/MLLM and strong-to-weak settings | arXiv 2605.03677 |
| SDAR (2026.05) | OPD × RL Hybrid · Agent OPD | Computes the teacher-student log-probability difference per token, adaptively adjusting distillation trust via a sigmoid gate (with detach to prevent self-sabotage); remains effective even when the teacher's average level is below the student's; +10.2pp accuracy on WebShop and +4.7pp on ALFWorld with Qwen2.5-7B, and SDAR still converges stably in settings where directly combining GRPO+OPSD collapses | arXiv 2605.15155 |
| TCOD (2026.04) | Agent OPD | Identifies "trajectory-level KL instability" in multi-turn agent OPD (errors compound step by step, distorting the teacher signal), fixed via a progressive trajectory-depth curriculum: F2B expands from a short prefix to the full trajectory, B2F has the teacher hand off from the early segment to the student progressively; +15.71pp on ALFWorld with Qwen2.5-3B, a 1.7B model recovers from near-0% to usable, and training time drops 32% | arXiv 2604.24005 |
| SOD (2026.05) | White-Box OPD · Agent OPD | Addresses distribution jumps caused by a single tool-call error in tool-integrated reasoning (TIR), via step-wise divergence-adaptive weighting: the student-teacher log-probability gap is computed in real time at each step, lowering that step's KL weight when the gap is large and auto-recovering once it narrows; a 1.7B student beats vanilla OPD by an average of +6.71pp (50.83% on AIME 2024), and stronger teachers become even more effective under SOD | arXiv 2605.07725 |
| vOPD (2026.05) | White-Box OPD | Casts OPD as policy-gradient RL and stabilizes the high-variance single-sample estimator by subtracting a control variate baseline; shows the OPD value function has a closed form as the per-token negative reverse KL between student and teacher, computable directly from the existing forward pass with no extra critic; a top-k approximation of this baseline lowers cost further with no bias; matches full-vocabulary OPD while cutting wall-clock time by up to 57.7%, with up to +6.2% on MATH500 and gradient norms 1–2 orders of magnitude lower than vanilla OPD | arXiv 2605.07865 |
| EffOPD (2026.05) | White-Box OPD | The first mechanistic explanation of why OPD is 3× faster than RL: (1) RL wastes a large share of updates on ineffective layers like embeddings, while OPD naturally concentrates on reasoning layers; (2) after just 10% of OPD training, the update direction already has extremely high cosine similarity to the final direction (low-rank concentration). Based on this, proposes EffOPD: exponentially spaced snapshotting + extrapolation along the locked direction + 50-problem validation filtering; roughly 3× faster convergence on math/code tasks across 1.5B–32B models, with performance matching standard OPD | arXiv 2605.11739 |
| OpenClaw-RL (2026.03) | OPD × RL Hybrid · Black-Box OPD · Agent OPD | The first open-source RL framework unifying Terminal/GUI/SWE/Tool-call agents; extracts an evaluative signal (GRPO) and a guidance signal (OPD hint) from user interactions, with a mixed objective ℒ=w_RL·ℒ_GRPO+w_OPD·ℒ_OPD; Overlap-Guided Hint Selection picks the hint with maximal top-k overlap with the student, plus log-prob clipping for gradient stability; tool-call accuracy rises 0.19→0.25, and a personalized agent converges in just 10.3 sessions | arXiv 2603.10165 |
| TrOPD (2026.06) | White-Box OPD | Uses the teacher-student probability ratio instead of entropy as the trust criterion: reverse KL (trustworthy) where π_teacher ≥ π_student, forward KL for outlier positions where π_teacher < π_student; off-policy guidance lets the student start from the teacher's prefix; +3.44~+3.52 across domains vs. OPD, and entropy filtering (the EOPD route) is actually worse than the probability-ratio criterion under matched conditions | arXiv 2606.01249 |
| SDPG (2026.06) | OPD × RL Hybrid · PI-based OPSD | RL's sparse reward acts as a gate, OPD's dense signal does the refinement: positive-advantage gating (distillation activates only on RL-approved trajectories) + a β warm-up-then-decay schedule (actively turning off distillation late in training); full-vocabulary reverse KL is mathematically equivalent to a policy gradient with token-level advantage; Qwen3-4B AIME2025 goes 0.300→0.327 vs. RLSD, maintaining high entropy throughout to prevent mode collapse | arXiv 2606.04036 |
| OPRD (2026.06) | White-Box OPD | Moves distillation from output space (vocabulary probabilities) to hidden-state space: MSE alignment across all layers, supervising only the last 2000 tokens; proves theoretically that output-space OPD noise doesn't converge late in training (null-space masking), while hidden-state MSE has zero conditional gradient variance; a 1.5B model matches its teacher on AIME2024 (49.8% vs. teacher 50.8%), 1.44× faster and 54% less memory than top-16 OPD | arXiv 2606.06021 |
| PBSD (2026.06) | OPD × RL Hybrid · PI-based OPSD · Agent OPD | Uses Bayesian self-distillation to score every step of a long-horizon search agent: subtracting the likelihood of the same model under "answer given" vs. "answer withheld" conditions (a Bayesian flip turning an intractable posterior into a tractable likelihood ratio) yields a per-step evidence score s_t, which is used only as a weight to calibrate the magnitude of the GRPO advantage (direction still comes from the outcome reward) — never as a training target, avoiding the leakage of directly imitating a privileged teacher as in OPSD; Qwen3-30B-A3B scores 40.87 on validation vs. GRPO's 38.25 and BC(300)'s +3.50, with consistent gains across 256K-long-context benchmarks (BrowseComp 46.21 vs. GRPO's 40.05, where GRPO actually drops 4.66 below SFT); counter-intuitive: OPSD that directly imitates the privileged teacher (33.25) is worse than GRPO, showing that privileged information is safe as a weight but toxic as a target | arXiv 2606.09348 |
| RLCSD (2026.06) | OPD × RL Hybrid | Diagnoses "privilege-induced style drift" in OPSD: 3× of the teacher-student gap concentrates on stylistic words (e.g., "Wait", "Therefore") rather than task-relevant ones; uses incorrect answers as a control for common-mode suppression — subtracting the teacher-student gap under a wrong "cheat sheet" from the gap under the correct one cancels shared stylistic noise while retaining task signal; the contrastive signal only modulates (never replaces) the GRPO outcome advantage, with dual independent normalization to prevent minority signals from being drowned out; Qwen3-8B scores 74.0 on K&K vs. GRPO's 66.0 (+14.4 over base), and removing the verifier anchor drops it by 4.5 in ablation; counter-intuitive: in cross-model distillation, a weaker but stylistically closer Base teacher actually produces a stronger student than an Instruct teacher (20.7 vs. 15.7) | arXiv 2606.11709 |
| PowerOPD (2026.06) | White-Box OPD | Traces OPD's sample inefficiency, unstable generation dynamics, and gap to full-vocabulary OPD back to the log-ratio reward being unbounded by construction, which produces extremely high-variance gradients concentrated at early positions; proposes a family of natively bounded, sign-consistent rewards via the Box-Cox power transformation (parameterized by α>0, with the log-ratio reward as the degenerate α→0 limit); across six math benchmarks and four Qwen3 teacher-student pairs, gains up to +6.37/+5.71 Avg@8/Pass@8 over vanilla OPD and +2.59/+8.90 over full-vocabulary OPD, at 59.2% less wall-clock time and 23.1% less peak memory, with gradient norms over 3,000× smaller than vanilla OPD | arXiv 2606.17199 |
| IW-OPD (2026.06) | White-Box OPD | Discovers a positional bias: supervising only the first 30% of tokens matches full-token supervision, while the last 30% is nearly useless; derives the theoretically optimal weight as the teacher/student probability ratio, implemented via accumulated unsigned prefix-deviation weighting + γ-mixing with standard OPD; +6.9 points at step 10 on AIME2025, +1.7 points at convergence, +1.3~+2.2 points at extreme 235B→30B scale; side finding: KL drops only 20% after OPD convergence, revealing that OPD is fundamentally local-neighborhood optimization rather than global alignment | arXiv 2606.22600 |
| ATOD (2026.06) | OPD × RL Hybrid · Agent OPD | An annealing schedule lets OPD dominate early cold-start while RL takes over later to surpass the teacher; T-DUR uses a divergence × uncertainty soft weight to focus on key steps within multi-turn trajectories; a 0.6B student surpasses its 4B teacher on both ALFWorld and WebShop, averaging +3.03 points over OPD and +23.62 points over GRPO | arXiv 2606.27814 |
| DanceOPD (2026.06) | Image-Generation OPD · Multi-Capability Fusion | Abstracts multiple frozen image-generation experts (text-to-image / local editing / global style) each as a velocity field, using sample-level hard routing (each sample assigned to exactly one field) + single-point on-policy semantic-side queries to eliminate gradient conflicts and distribution bias; a combined text-to-image + editing model gets +8.1% on GEditBench; key finding: dense querying (K=16) is actually 16.6% worse than a single query, and the merged student surpasses any individual teacher | arXiv 2606.27377 |
| DEAR (2026.06) | White-Box OPD | Splits the reasoning chain into "decisions" (high-entropy branch points for the student) and "evidence" (positions where the hidden state resembles a decision but the student is confidently wrong, with a large teacher-student divergence); uses cosine similarity of the student's hidden states as a relevance gate combined with the teacher-student log-probability gap as a gap amplifier, multiplicatively scoring and selecting the top-q% evidence, supervising only decisions ∪ evidence (~36% of tokens); 2.2×/2.7× improvement over standard OPD on AIME 2024/2025, +5.74pp on MBPP code; counter-intuitive: decision-only supervision covers just 39.1% of the gradient mass (vs. 35.9% random), meaning evidence alone drives almost all the gain, with DEAR reaching 75.8% coverage | arXiv 2606.22830 |
| MOPD (2026.06) | Industrial Multi-Teacher · White-Box OPD | A multi-domain post-training paradigm: starting from the same SFT base, same-origin domain-expert teachers are trained in parallel (math/code/instruction each via RL), then each problem is routed to only its corresponding domain teacher for per-token reverse-KL distillation, with no signal merging whatsoever; the teacher-student log-difference is fed directly into the PPO/GRPO framework as the advantage; a 30B student scores a normalized 0.937 (vs. 0.882 for mixed training and 0.775 for sequential training), with the tightest spread across five domains (0.044), and sample efficiency of 25–30K vs. 150–180K for mixed training; already deployed in the 309B MiMo-V2-Flash; counter-intuitive: swapping in a stronger but distant 235B external teacher actually breaks the student (5× higher initial KL, entropy drops 0.30→0.21, top-k collapses at step 18) — same-origin matters more than raw capability | arXiv 2606.30406 |
| OPID (2026.06) | Agent OPD · OPD × RL Hybrid | Extracts hierarchical hindsight skills (episode-level global workflows + step-level critical decisions) from successful trajectories, uses critical-first routing for adaptive selection, and injects token-level self-distillation advantage into the advantage function; 84.3% on ALFWorld (+9.3pp), 74.2% on WebShop (+10.9pp), matches full-data GRPO with only 60% data, zero external dependencies at inference; counter-intuitive: Skill-GRPO drops −14.8~−25.0 points when inference-time skills are removed, proving skill-crutch dependency is toxic | arXiv 2606.26790 |
| SEED (2026.07) | Agent OPD · OPD × RL Hybrid | Self-evolving hindsight distillation: the model serves simultaneously as actor (collecting trajectories) and analyzer (extracting skills), with supervision signals co-evolving with the policy; two-stage training (cold-start SFT + self-evolving RL), skill-induced log-probability shift gated into advantage; 91.8% on ALFWorld (+16.8pp vs. GRPO), 78.9% on WebShop (+15.6pp), beats full-data GRPO with 60% data; counter-intuitive: static skill library drops 7.4 points, skills as prompts underperform distillation internalization | arXiv 2607.14777 |
Related Work
The following are neighboring-field papers frequently cited or borrowed from in OPD research, grouped by topic.
Related Work: Agentic RL Environments
| Paper | Summary | Link |
|---|---|---|
| R2E-Gym (2025.04) | A programmatically synthesized SWE training environment: generates 8135 problems via commit backtranslation (only 0.2pp gap between synthetic and real issues); a hybrid verifier combines execution-based and execution-free checks to cover each other's blind spots; a 32B model reaches 51% Best@26, the first open-source result approaching Claude Sonnet's commercial-level performance | arXiv 2504.07164 |
| SWE-Smith (2025.04) | Actively manufactures bugs to scale up SWE training data: four bug-injection strategies (program transformation / LM rewriting / PR reversal / multi-hole merging) generate 50K problems across 128 repos, with repo-level image reuse compressing storage 500× at $0.025/problem; a 32B model reaches 40.2% on SWE-Bench, open-source SOTA; key finding: harder problems aren't more valuable for training than easy ones — repo diversity is the deciding factor | arXiv 2504.21798 |
| SWE-Factory (2025.06) | A multilingual SWE data pipeline (Python/Java/TS/Go): a 4-agent SWE-Builder system auto-configures test environments, using exit codes instead of log parsing (F1=0.99), with an environment-image pool reusing configs across versions; GPT-4.1 mini achieves a 50.2% F2P rate at $0.047/instance; Qwen2.5-Coder-14B jumps from 5.8% to 21.0% on SWE-Bench after training | arXiv 2506.10954 |
| LLM-in-Sandbox (2026.01) | (Sandbox construction) A minimal three-tool sandbox (bash/file_editor/finish) gives rise to three general meta-abilities: external resource access, file management, and code execution; 7 frontier models gain up to +15.5% across 6 domains with no training and 8× lower token cost; LLM-in-Sandbox-RL trains weaker models on general data, raising the tool-call rate from <3% to 4–7% | arXiv 2601.16206 |
| SkillRL (2026.02) | Distills success/failure trajectories into a hierarchical skill library (general + task-specific, 10–20× compression), with recursive co-evolution during GRPO training (55→100 skills); 89.9% on ALFWorld (+12.3pp vs. GRPO), surpassing GPT-4o's 41.9%; abstraction beats memorization (replacing with raw trajectories costs −25%) | arXiv 2602.08234 |
| AgentRL (2025.10) | A fully asynchronous Rollout/Actor/Reference three-worker decoupled pipeline with a unified Function-Call API; cross-policy sampling (mixing current + historical checkpoints) resolves multi-turn exploration collapse; task-level advantage normalization stabilizes multi-task training; surpasses GPT-4/Claude-Sonnet-4/DeepSeek-R1, with multi-task performance matching single-task experts | arXiv 2510.04206 |
| AgentGym-RL (2025.09) | Proposes a ScalingInter-RL curriculum: limiting then expanding the number of interaction steps to address sparse-reward cold-start in long-horizon RL, with no SFT needed; a decoupled EnvServer/EnvClient architecture supports 14 environments and 27 tasks; pure RL from scratch matches or surpasses commercial models like GPT-4 | arXiv 2509.08755 |
| Chain-of-Agents (2025.08) | Distills three specialized multi-agent systems (Web/Code/MHQA) into a single AFM (masked fine-tuning computing loss only on action-reasoning tokens), then refines it with DAPO/PPO agentic RL; 55.3% Pass@1 / 69.9% Pass@3 on GAIA; fully open-sourced at 3B/7B/32B | arXiv 2508.13167 |
| SPA-RL (2025.05) | Trains a lightweight MLP progress estimator, using a potential function to decompose the terminal reward into per-step contributions (telescoping preserves the total sum); combines progress reward + grounding reward to replace the sparse terminal reward; +2.5% success rate across three benchmarks, +25% vs. PPO on long-horizon settings of 25–29 steps | arXiv 2505.20732 |
| GiGPO (2025.05) | (NeurIPS 2025) Adds a step-level advantage on top of GRPO's trajectory-level advantage: identifies identical environment states across rollouts (anchors) to build step-level contrastive groups, A=A^E+ω·A^S; requires no extra model, <0.002% overhead; +13.9pp on ALFWorld (72.8%→86.7%), +7.7 on WebShop vs. GRPO | arXiv 2505.10978 |
| SkyRL-Agent (2025.11) | A fully asynchronous three-stage pipeline (init/execution/reward run in parallel) reaching ~90% GPU utilization and 1.55× throughput; an AST-based code-search tool reduces code-localization failures, hitting SOTA at 125 steps; SA-SWE-32B: SWE-Bench 24.4%→39.4%, at 2× the training-cost efficiency of DeepSWE | arXiv 2511.16108 |
| VAGEN (2025.10) | (NeurIPS 2025) A POMDP framework with explicit world-model reasoning (structured output tags for state estimation + transition prediction); world-model reward provides dense intermediate signal; Bi-GAE estimates advantage at both the turn and token level; a 3B model scores 0.82 overall, surpassing GPT-5 (0.75) / Gemini 2.5 Pro (0.67) / Claude 4.5 (0.62) | arXiv 2510.16907 |
Related Work: RL Post-Training Data Engineering
| Paper | Summary | Link |
|---|---|---|
| SAERL (2026.05) | Uses SAE internal activations to extract semantic coordinates for 103K training problems, then applies quality filtering (99.92% precision), curriculum difficulty ranking (Spearman 0.749), and moderate batch mixing (mix₈ is the sweet spot; over-mixing hurts); +3.00% vs. vanilla GRPO with 20% fewer steps, at a preprocessing cost of just 0.5 GPU-hours (34× cheaper than fast-rollout approaches) | arXiv 2605.27354 |
Related Work: Multi-Reward RL
| Paper | Summary | Link |
|---|---|---|
| MVA (2025.11) | Two-stage decoupled multi-value alignment: HSIC regularization pushes mutual information between value vectors toward zero (eliminating parameter interference), while an extrapolation weight in [0,C] breaks past the limits of convex interpolation to explore the Pareto front; full MVA vs. without HSIC: helpfulness 1.66 vs. 0.66 (2.5×), harmlessness 0.61 vs. 0.33; remains effective when extended to three value dimensions | arXiv 2511.17579 |
| GDPO (2026.01) | Fixes the normalization order for multi-reward RL: normalize each reward independently within-group before summing (rather than summing then normalizing), eliminating "advantage collapse" (different reward combinations compressing to the same advantage value); a conditioning strategy addresses weight-adjustment failure when reward difficulty is highly skewed; +2.6pp on tool-calling, +6.3pp on math (AIME) at 1.5B, and bug rate on a 3-reward coding task drops from 3.9% to 2.5% | arXiv 2601.05242 |
| RVPO (2026.05) | A variance penalty for multi-reward RL: replaces "mean aggregation" with SoftMin (a soft-minimum operator), equivalent to mean − k/2 × variance, forcing the model to shore up weak dimensions rather than letting high-scoring ones carry the average; a k-annealing curriculum (0.5→2.0) ensures stability; a 14B model's HealthBench score rises from 0.215 to 0.261, while GDPO collapses to zero late in 14B training and RVPO holds at 0.236 | arXiv 2605.05750 |
| RDPO (2026.05) | Multi-objective mixed-reward RL training: MAQ quantile normalization unifies heterogeneous reward scales (handling binary/continuous/piecewise rewards), and Mahalanobis whitening removes redundancy across reward dimensions; introduces η_eff = η_proj × η_corr to diagnose the effective information content of multiple rewards; +10.91pp on IFEval vs. GRPO, and ArenaHard-Hard (large-scale) rises from 65.80% to 76.10% | arXiv 2605.13641 |
Related Work: RLVR Mechanism Analysis
| Paper | Summary | Link |
|---|---|---|
| RM Overoptimization Scaling Laws (2022.10) | The first quantitative study of RLHF reward-model overoptimization: using synthetic labels from dual gold/proxy RMs, finds that both BoN and RL gold scores first rise then fall, with the curve's coefficients scaling smoothly (log-linearly) with RM size; key finding: KL penalty is equivalent to early stopping (it doesn't improve the KL-gold frontier), and larger policy models plateau at the same KL as smaller ones | arXiv 2210.10760 |
| Sparse but Critical (2026.03) | A token-level dissection of RLVR: after DAPO training, 83%+ of positions have near-zero divergence (98%+ for SimpleRL); injecting <4% of RL tokens into the base model fully recovers the gain (AIME 2024 8%→25%), while removing the top 5% critical positions collapses everything; RL changes candidate ranking rather than the candidate set (80% of critical tokens are already in the base model's top-3), and the SFT distribution shift is far wider than RLVR's | arXiv 2603.22446 |
| Inference Manifold (2026.05) | A geometric diagnosis of reasoning quality: reasoning trajectories spontaneously compress to 5–10 intrinsic dimensions (out of 4096), but compression plus rising information volume (deeper layers have lower dimension yet more information) is the true mark of healthy reasoning; a three-factor health score H (base expressiveness × information volume ÷ dimensional-scatter penalty) correlates with all tested benchmarks at Spearman >0.9, entirely label-free | arXiv 2605.08142 |
| Negative RL (2026.06) | Splits RLVR into PSR (rewarding correct answers) and NSR (penalizing wrong ones): NSR trained alone reaches 53.3% AIME Pass@256, beating GRPO (50.0%) and PPO (43.3%), while PSR trained alone is actually below the base model's Pass@256; PSR collapses diversity by locking in correct paths, while NSR preserves diversity by only clearing roadblocks rather than dictating a path; W-REINFORCE (0.1·PSR + NSR) matches PPO's Pass@1 while achieving the highest Pass@k | arXiv 2506.01347 |
| DRPO (2026.06) | An improved RLVR trust-region algorithm: diagnoses the root cause of ratio-based methods (PPO/GRPO/SPO) having unbounded gradient weights on low-frequency tokens and collapsing under FP8; replaces the ratio-based ruler with an absolute-probability-difference (Binary-TV) measure, turning DPPO's hard clipping into a bounded, continuous spring-like penalty; key finding: the corrective signal beyond the boundary matters more than the smooth decay within it; ratio-based methods degrade severely under end-to-end FP8 training on Qwen3-30B while DRPO remains stable | arXiv 2606.09821 |
| RELEX (2026.05) | RLVR's weight-update trajectory is extremely low-rank (rank-1 accounts for 80%+ per layer) with coefficients evolving nearly linearly (R²>0.98); observing just 15% of the training steps and applying SVD + linear extrapolation directly predicts the final weights, saving 85% of training; the SVD projection also denoises, so OOD benchmarks actually beat full training (AIME 2025 8.8% vs. 4.6%, HMMT 2.1% vs. 0.4%), with MATH performance on par | arXiv 2605.21468 |
Related Work: OPD Mechanism Analysis
| Paper | Summary | Link |
|---|---|---|
| OPD Geometry (2026.06) | Diagnoses OPD's parameter-space trajectory along three dimensions (stable rank / principal-component rotation / spectral drift): by 20% of training it locks into a narrow ~16-dimensional channel (subspace locking); restricting training to that subspace leaves OPD performance unchanged while SFT degrades sharply; the composition of the objective function — not token selection — is the only lever that changes the channel; OPD isn't a compromise between SFT and RLVR, but an independent third path | arXiv 2606.07082 |
| Dense Supervision Sparse Updates (2026.06) | OPD's parameter updates are coordinate-sparse (66–89% untouched) and spectrally concentrated (stable rank 8–12 vs. 83 for offline distillation), while actively avoiding the source model's principal singular directions (only about half the energy expected under random chance lands there); OPD and RLVR share the same set of plastic coordinates (3× overlap over a random baseline); it's AdamW, not the raw gradient, that produces the sparsity — the optimizer distills a sparse signal out of a dense gradient | arXiv 2606.13657 |
Related Work: Two-Stage Policy Decoupling
| Paper | Summary | Link |
|---|---|---|
| ERPD (2026.05) | A two-stage decoupling: stage one relaxes the KL constraint, letting the policy fly into "extreme regions" to extract maximal signal; stage two uses trust-region distillation to recycle the valuable parts back into the base model while filtering out invalid drift; the distilled student's KL is just 0.01 (vs. the teacher's 0.041, 4× smaller) with accuracy nearly unchanged; even a degraded teacher (trained in reverse via MSE) can still provide an effective distillation signal, addressing the case where the teacher is weaker than the student | arXiv 2605.25582 |
awesome-list
knowledge-distillation
large-language-models
llm
llm-agent
on-policy-distillation
paper-list
reinforcement-learning
Contributors
kissshhot
4 commits
Similar repos
A curated paper list on On-Policy Distillation (OPD) & On-Policy Self-Distillation (OPSD) for LLMs — student rollouts + teacher feedback, covering white-box/black-box methods, OPD-RL hybrids, agentic & multimodal applications, and industrial multi-teacher recipes.
24
4 commits
updated Aug 6, 2026
README
Awesome On-Policy Distillation (OPD)
A curated paper list on On-Policy Distillation (OPD) — the student model samples its own rollouts, and a teacher model provides feedback on them, eliminating the exposure bias of offline distillation.
- 🎯 OPD core requirements:
C1the student samples its own rollouts during training,y ~ π_student(·|x);C2the teacher provides token- or sequence-level feedback on those student samples. Both conditions must hold for a strict definition of OPD. - 🪞 OPSD (On-Policy Self-Distillation) is a special case of OPD: the teacher is the same model, but conditioned on privileged information (the correct answer / a longer context / an earlier checkpoint).
- 📚 This repo tags each paper with one or more technical directions (white-box / black-box / PI-based OPSD / OPD×RL hybrid / CoT / multimodal / agent / industrial multi-teacher), plus a one-line method summary and key experimental numbers.
- ⚠️ Content is compiled from close readings of paper PDFs. PRs to fix errors are welcome.
Table of Contents
Surveys
| Paper | Summary | Link |
|---|---|---|
| A Survey of On-Policy Distillation for LLMs (2026.04) | The first systematic survey; proposes a 3D taxonomy (feedback signal × teacher access × loss granularity) and outlines 9 open research directions | arXiv 2604.00626 |
Direction Overview
| Direction | Description |
|---|---|
| 🔬 White-Box OPD | Teacher logits are accessible; direct KL / JSD distillation |
| 🎭 Black-Box OPD | Only discrete teacher feedback available (verbal score / ranking / verification result), no logits |
| ♻️ PI-based OPSD | Teacher is the same model but conditioned on privileged information (PI) |
| 🤝 OPD × RL Hybrid | OPD signal fused with RLVR advantage/reward during training |
| 🧠 CoT OPD | Distillation targeting long chain-of-thought reasoning patterns |
| 🖼️ Multimodal OPD | Vision-language / image generation and other multimodal settings |
| 🤖 Agent / Embodied OPD | Multi-turn agent, embodied AI, and other long-horizon decision-making settings |
| 🏭 Industrial Multi-Teacher | Multi-domain / multi-teacher distillation in industrial practice |
Methods
| Paper | Direction | Summary | Link |
|---|---|---|---|
| KDRL (2025.06) | OPD × RL Hybrid | Jointly optimizes GRPO (RL direction) and KD-RKL (reverse-KL distillation direction) in a single loss, with β-annealing and reward-guided masking; +2.6% vs GRPO with 10% fewer distillation tokens, showing joint training beats sequential staging | arXiv 2506.02208 |
| SCoRe (2025.09) | White-Box OPD | Student generates rollouts, teacher only corrects the earliest error step (aligns only the first deviation); a 7B model approaches 72B-teacher performance on 12 benchmarks | arXiv 2509.14257 |
| DDT / On-Policy SFT (2026.02) | White-Box OPD | Uses signal-detection theory to derive CLL = log p + H[p] as the optimal criterion for distinguishing on-policy vs. OOD tokens; IDFT dynamically down-weights OOD-token gradients via exp(−φ); Hinted Decoding adaptively mixes distributions in log-space by imitator entropy to align training data; HD+IDFT surpasses DPO/SimPO at SFT-level GPU cost | arXiv 2602.12222 |
| SDPO (2026.01) | OPD × RL Hybrid | Uses the model itself as a self-teacher under feedback conditioning (RLVR environment feedback plus correct outputs in the same batch), providing dense token-level KL signal; beats GRPO with 4× fewer generations | arXiv 2601.20802 |
| OVD (2026.01) | Black-Box OPD · Agent OPD | Replaces token-logit matching with discrete verbal scores (0–9); on web tasks an environment agent simulates a search engine and produces the score, on math tasks the teacher model scores directly; 48,000× memory reduction, up to +26% on math reasoning | arXiv 2601.21968 |
| PID (2026.02) | PI-based OPSD · Agent OPD | Proposes a privileged-information distillation paradigm for agentic tasks with "observable behavior, hidden reasoning"; both π-Distill (joint teacher-student parameter training) and OPSD (on-policy RL + reverse-KL penalty) surpass the SFT+RL industry standard | arXiv 2602.04942 |
| SSD (2026.04) | Black-Box OPD | No verifier/teacher/RL needed — SFT on the model's own high-temperature truncated-sampling outputs; adaptively reshapes the token distribution via an "accuracy-exploration conflict" mechanism; Qwen3-30B pass@1 on LiveCodeBench v6 rises from 42.4% to 55.3% (+12.9pp) | arXiv 2604.01193 |
| Skill-SD (2026.04) | White-Box OPD · Agent OPD | Dynamically extracts a skill-conditioned teacher from the student's own historical trajectories (asymmetric prompting), combined with importance-weighted reverse KL + dynamic parameter sync; +14.0%/+10.9% over vanilla GRPO and +42.1%/+40.6% over vanilla OPD on AppWorld/Sokoban | arXiv 2604.10674 |
| SCOPE (2026.04) | White-Box OPD | Diagnoses the OPD Pass@k paradox (PSR/OPD raises Pass@1 but drops Pass@32, i.e., mode collapse) plus a prefix trap (high teacher PPL means the teacher can't parse the student's bad prefix either); routes by correctness — correct samples go through a student MLE self-bootstrap path, incorrect ones through a teacher-KL consultation path, using the same PPL in opposite directions on each path; +11.42% Avg@32, +7.30% Pass@32 — the only method to improve both simultaneously | arXiv 2604.10688 |
| Lightning OPD (2026.04) | White-Box OPD · CoT OPD | Proposes a "teacher consistency" principle (SFT and OPD must use the same teacher or gradients become biased), and accordingly moves the teacher to offline pre-computation; 4× training efficiency gain, a single node can train a 30B model to 71.0% on AIME 2024 | arXiv 2604.13010 |
| StableOPD (2026.04) | White-Box OPD | Diagnoses OPD's truncation collapse: repeated tokens receive 4–9× the reverse-KL advantage, on-policy sampling forms a self-reinforcing loop, and within ~30 steps the truncation rate goes 0.5→1.0, repetition rate 0→0.6, with accuracy crashing in sync; fix is mixture distillation (blending in gold solutions as an SFT distribution anchor) + a KL gate (limiting policy drift from the reference model); ablations show the gold anchor contributes +6.0 vs. only +1.7 from the KL gate; 1.5B student scores 36.1 vs. OPD 28.9 / SFT 31.9 / GRPO 30.1, and the 7B student's 47.6 beats all zero-style RLVR; OPD is actually worse than SFT here (collapse makes a strong signal less reliable than a stable weak one) | arXiv 2604.08527 |
| OPD Recipe (2026.04) | White-Box OPD · Agent OPD | Systematically analyzes when OPD succeeds/fails: two preconditions (compatible thinking patterns + a teacher with genuinely new capability), token-level progressive alignment signal, off-policy cold-start remedies, and the root cause of signal degradation on long sequences (>7K tokens), which stems from agent environments producing very long sequences where OPD's entropy signal from the teacher becomes unreliable deep into those sequences | arXiv 2604.13016 |
| REOPOLD (2026.03) | White-Box OPD | Reinterprets OPD as RL (log-probability ratio = token reward), fixing three issues with the RL toolbox: a mixture-distribution-derived reward-clipping floor log(λ/(1−λ)) to prevent gradient explosion, entropy-guided masking that skips 60% of near-zero-reward tokens, and a two-stage explore→refine learning-mode switch; 7B student gets 3.32× faster inference vs. a 32B teacher, and vanilla RKL collapses at matched teacher-student scale while REOPOLD remains unaffected | arXiv 2603.11137 |
| OPD Fixes (2026.03) | White-Box OPD | Identifies three engineering flaws in sampled-token OPD (imbalanced signal / unreliable teacher guidance / tokenizer mismatch), and proposes Teacher Top-K Local Support Matching (truncated reverse KL + support-set normalization + top-pp sampling + special-token masking); observes that the teacher's probability on a given token is usually lower than the student's, so the fix for the long-tail is to forcibly compute the student's probability over the teacher's top-k, effectively pulling the student toward the teacher off-policy; +19.8% in a multi-task setting | arXiv 2603.25562 |
| RLAD (2026.02) | White-Box OPD · OPD × RL Hybrid | Folds the teacher policy into the GRPO importance ratio (geometric mean) to construct a mixed anchor TRRD: r^TRRD = (r^GRPO)^α × (r^T)^(1−α), implicitly equivalent to α·KL(student‖old policy)+(1−α)·KL(student‖teacher); selective imitation automatically aligns with the advantage direction, removing objective interference; α=0.5 works well by default across tasks; +5.5pp on math reasoning, AIME24 Pass@32 77.8→85.4, with only +12% extra training overhead | arXiv 2602.22495 |
| G-OPD / ExOPD (2026.02) | White-Box OPD · OPD × RL Hybrid · Industrial Multi-Teacher | Proves OPD is a special case of dense KL-constrained RL (with an implicit reward = teacher/reference log-probability ratio); introduces a reward scale λ: λ=1.25 (ExOPD) lets the student extrapolate along the teacher's improvement direction, surpassing the teacher; supports multi-teacher merging and strong-to-weak distillation reward correction | arXiv 2602.12125 |
| EOPD (2026.03) | White-Box OPD | Identifies a dual failure of reverse KL at high-entropy positions (unstable gradients + diversity collapse to 6.8%), proposing entropy-aware switching: reverse KL at low entropy, forward-KL top-K approximation at high entropy; +5.05 Pass@8 on Qwen3-4B | arXiv 2603.07079 |
| NPO (2026.04) | OPD × RL Hybrid · Multimodal OPD | Replaces one rollout with a near-future checkpoint (Δ* steps ahead) from the same training run, optimizing S=Q/V to balance trajectory quality and variance; AutoNPO auto-detects plateaus and selects the optimal Δ*; +5.27 average across 8 benchmarks on Qwen3-VL-8B, 2.1× faster convergence | arXiv 2604.20733 |
| SRPO (2026.04) | OPD × RL Hybrid | Routes by correctness: correct samples reinforced via GRPO, incorrect samples distilled via SDPO (using correct answers from the same batch as the teacher); an entropy-aware exponential weight exp(−β·H_t) suppresses contributions at teacher-uncertain positions; the mixing ratio adapts automatically to the success rate; +3.4pp vs. GRPO and +6.3pp vs. SDPO across 5 benchmarks on Qwen3-8B, with −17.2% compute cost; key finding: SDPO applied only to correct samples is actually worse than GRPO | arXiv 2604.02288 |
| RLSD (2026.04) | OPD × RL Hybrid · Multimodal OPD | Reframes OPSD's self-distillation signal from "distribution matching" to "magnitude modulation": the environmental reward (RLVR) determines the update direction (sign(A)), while the teacher/student log-probability ratio Δ (OPD) determines the token-level update magnitude; mathematically proves OPSD suffers from an unremovable mutual-information leak causing KL stagnation; Qwen3-VL-8B averages 56.18% across five benchmarks, +2.32pp over GRPO and +3.69pp over OPSD | arXiv 2604.03128 |
| GLM-5 (2026.02) | OPD × RL Hybrid · Agent OPD · Industrial Multi-Teacher | Industrial-scale OPD deployment within multi-stage RL post-training: uses earlier-stage checkpoints as teachers, embedding the teacher-student log-probability ratio directly into the advantage function for cross-stage anti-forgetting and capability fusion; agentic RL uses an asynchronous decoupled framework (slime) + TITO gateway + two-sided importance sampling supporting 1000+ concurrent rollouts; #1 open-source on Vending-Bench 2, 50 points on the Artificial Analysis Intelligence Index v4.0 | arXiv 2602.15763 |
| MiMo-V2-Flash (2026.01) | Agent OPD · Industrial Multi-Teacher | Proposes multi-teacher on-policy distillation (MOPD): domain-specific teachers are each trained independently via RL, then distilled on-policy into a single student using token-level advantage (teacher/student log-probability ratio + ORM reward), with clipped importance weights to prevent off-policy drift, and each teacher distills only prompts from its own domain; multiple types of agentic RL (code/search/web/terminal) plus R3 routing replay resolve MoE expert-routing inconsistency; 73.4% on SWE-Bench Verified, the highest among open-source models | arXiv 2601.02780 |
| Nemotron-Cascade 2 (2026.03) | OPD × RL Hybrid · Agent OPD · Industrial Multi-Teacher | Proposes a Cascade RL + MOPD post-training framework: MOPD uses historical pipeline checkpoints as teachers (Math/RLHF/Multi-domain categories), with token-level advantage = teacher-student log-probability ratio and clipping weights ε=0.5/2.0, acting as a "stable anchor" between RL stages to recover from capability drift, converging 3× faster than GRPO; SWE RL supports both Agentless and Execution-based modes, and joint training transfers across frameworks; the 30B-A3B model achieves gold-medal level on IMO/IOI/ICPC | arXiv 2603.19220 |
| HDPO (2026.03) | OPD × RL Hybrid | Addresses GRPO's "cliff" problem (all rollouts fail, advantage is zero, no gradient): the same model is given the correct answer "open-book" to generate a solution, R=1 filters correct trajectories, and top-k=64 JSD distills them back into the "closed-book" model itself; identical teacher/student weights (differing only in input) bound the realizability gap; drifting λ=0.01 barely hurts pass@1 while pass@4 gains +1.1pp; λ=0.1 pushes pass@8 to 0.84 (+1.7pp) at a cost of −2.3pp pass@1; counter-intuitive: a frozen teacher, by preserving pre-training diversity, actually beats the drifting teacher on pass@8 | arXiv 2603.23871 |
| MAD-OPD (2026.05) | White-Box OPD · Agent OPD · Industrial Multi-Teacher | Uses multi-teacher debate (R=2 rounds) to break through the single-teacher capability ceiling: debate history is injected as privileged information via force-decoding, and confidence-based softmax weighting merges the distributions; agentic tasks use JSD, code generation uses reverse KL; paired with OPAD (on-policy sampling + environment interaction at every step) to eliminate long-trajectory error accumulation; ranks first across all 6 teacher-student configurations, and a 4B student beats its 14B teacher by +4.26% on a code benchmark | arXiv 2605.01347 |
| Uni-OPD (2026.05) | White-Box OPD · Industrial Multi-Teacher · Multimodal OPD | Fixes two major OPD pain points from dual perspectives: offline difficulty-awareness plus online correctness-balancing addresses insufficient exploration, and margin calibration (Mask/Shift) uses outcome rewards to correct ordering errors in the teacher signal; on Qwen3-4B multi-teacher settings, 48.7 vs. 47.0 on math and 63.6 vs. 60.2 on code, with consistent gains across LLM/MLLM and strong-to-weak settings | arXiv 2605.03677 |
| SDAR (2026.05) | OPD × RL Hybrid · Agent OPD | Computes the teacher-student log-probability difference per token, adaptively adjusting distillation trust via a sigmoid gate (with detach to prevent self-sabotage); remains effective even when the teacher's average level is below the student's; +10.2pp accuracy on WebShop and +4.7pp on ALFWorld with Qwen2.5-7B, and SDAR still converges stably in settings where directly combining GRPO+OPSD collapses | arXiv 2605.15155 |
| TCOD (2026.04) | Agent OPD | Identifies "trajectory-level KL instability" in multi-turn agent OPD (errors compound step by step, distorting the teacher signal), fixed via a progressive trajectory-depth curriculum: F2B expands from a short prefix to the full trajectory, B2F has the teacher hand off from the early segment to the student progressively; +15.71pp on ALFWorld with Qwen2.5-3B, a 1.7B model recovers from near-0% to usable, and training time drops 32% | arXiv 2604.24005 |
| SOD (2026.05) | White-Box OPD · Agent OPD | Addresses distribution jumps caused by a single tool-call error in tool-integrated reasoning (TIR), via step-wise divergence-adaptive weighting: the student-teacher log-probability gap is computed in real time at each step, lowering that step's KL weight when the gap is large and auto-recovering once it narrows; a 1.7B student beats vanilla OPD by an average of +6.71pp (50.83% on AIME 2024), and stronger teachers become even more effective under SOD | arXiv 2605.07725 |
| vOPD (2026.05) | White-Box OPD | Casts OPD as policy-gradient RL and stabilizes the high-variance single-sample estimator by subtracting a control variate baseline; shows the OPD value function has a closed form as the per-token negative reverse KL between student and teacher, computable directly from the existing forward pass with no extra critic; a top-k approximation of this baseline lowers cost further with no bias; matches full-vocabulary OPD while cutting wall-clock time by up to 57.7%, with up to +6.2% on MATH500 and gradient norms 1–2 orders of magnitude lower than vanilla OPD | arXiv 2605.07865 |
| EffOPD (2026.05) | White-Box OPD | The first mechanistic explanation of why OPD is 3× faster than RL: (1) RL wastes a large share of updates on ineffective layers like embeddings, while OPD naturally concentrates on reasoning layers; (2) after just 10% of OPD training, the update direction already has extremely high cosine similarity to the final direction (low-rank concentration). Based on this, proposes EffOPD: exponentially spaced snapshotting + extrapolation along the locked direction + 50-problem validation filtering; roughly 3× faster convergence on math/code tasks across 1.5B–32B models, with performance matching standard OPD | arXiv 2605.11739 |
| OpenClaw-RL (2026.03) | OPD × RL Hybrid · Black-Box OPD · Agent OPD | The first open-source RL framework unifying Terminal/GUI/SWE/Tool-call agents; extracts an evaluative signal (GRPO) and a guidance signal (OPD hint) from user interactions, with a mixed objective ℒ=w_RL·ℒ_GRPO+w_OPD·ℒ_OPD; Overlap-Guided Hint Selection picks the hint with maximal top-k overlap with the student, plus log-prob clipping for gradient stability; tool-call accuracy rises 0.19→0.25, and a personalized agent converges in just 10.3 sessions | arXiv 2603.10165 |
| TrOPD (2026.06) | White-Box OPD | Uses the teacher-student probability ratio instead of entropy as the trust criterion: reverse KL (trustworthy) where π_teacher ≥ π_student, forward KL for outlier positions where π_teacher < π_student; off-policy guidance lets the student start from the teacher's prefix; +3.44~+3.52 across domains vs. OPD, and entropy filtering (the EOPD route) is actually worse than the probability-ratio criterion under matched conditions | arXiv 2606.01249 |
| SDPG (2026.06) | OPD × RL Hybrid · PI-based OPSD | RL's sparse reward acts as a gate, OPD's dense signal does the refinement: positive-advantage gating (distillation activates only on RL-approved trajectories) + a β warm-up-then-decay schedule (actively turning off distillation late in training); full-vocabulary reverse KL is mathematically equivalent to a policy gradient with token-level advantage; Qwen3-4B AIME2025 goes 0.300→0.327 vs. RLSD, maintaining high entropy throughout to prevent mode collapse | arXiv 2606.04036 |
| OPRD (2026.06) | White-Box OPD | Moves distillation from output space (vocabulary probabilities) to hidden-state space: MSE alignment across all layers, supervising only the last 2000 tokens; proves theoretically that output-space OPD noise doesn't converge late in training (null-space masking), while hidden-state MSE has zero conditional gradient variance; a 1.5B model matches its teacher on AIME2024 (49.8% vs. teacher 50.8%), 1.44× faster and 54% less memory than top-16 OPD | arXiv 2606.06021 |
| PBSD (2026.06) | OPD × RL Hybrid · PI-based OPSD · Agent OPD | Uses Bayesian self-distillation to score every step of a long-horizon search agent: subtracting the likelihood of the same model under "answer given" vs. "answer withheld" conditions (a Bayesian flip turning an intractable posterior into a tractable likelihood ratio) yields a per-step evidence score s_t, which is used only as a weight to calibrate the magnitude of the GRPO advantage (direction still comes from the outcome reward) — never as a training target, avoiding the leakage of directly imitating a privileged teacher as in OPSD; Qwen3-30B-A3B scores 40.87 on validation vs. GRPO's 38.25 and BC(300)'s +3.50, with consistent gains across 256K-long-context benchmarks (BrowseComp 46.21 vs. GRPO's 40.05, where GRPO actually drops 4.66 below SFT); counter-intuitive: OPSD that directly imitates the privileged teacher (33.25) is worse than GRPO, showing that privileged information is safe as a weight but toxic as a target | arXiv 2606.09348 |
| RLCSD (2026.06) | OPD × RL Hybrid | Diagnoses "privilege-induced style drift" in OPSD: 3× of the teacher-student gap concentrates on stylistic words (e.g., "Wait", "Therefore") rather than task-relevant ones; uses incorrect answers as a control for common-mode suppression — subtracting the teacher-student gap under a wrong "cheat sheet" from the gap under the correct one cancels shared stylistic noise while retaining task signal; the contrastive signal only modulates (never replaces) the GRPO outcome advantage, with dual independent normalization to prevent minority signals from being drowned out; Qwen3-8B scores 74.0 on K&K vs. GRPO's 66.0 (+14.4 over base), and removing the verifier anchor drops it by 4.5 in ablation; counter-intuitive: in cross-model distillation, a weaker but stylistically closer Base teacher actually produces a stronger student than an Instruct teacher (20.7 vs. 15.7) | arXiv 2606.11709 |
| PowerOPD (2026.06) | White-Box OPD | Traces OPD's sample inefficiency, unstable generation dynamics, and gap to full-vocabulary OPD back to the log-ratio reward being unbounded by construction, which produces extremely high-variance gradients concentrated at early positions; proposes a family of natively bounded, sign-consistent rewards via the Box-Cox power transformation (parameterized by α>0, with the log-ratio reward as the degenerate α→0 limit); across six math benchmarks and four Qwen3 teacher-student pairs, gains up to +6.37/+5.71 Avg@8/Pass@8 over vanilla OPD and +2.59/+8.90 over full-vocabulary OPD, at 59.2% less wall-clock time and 23.1% less peak memory, with gradient norms over 3,000× smaller than vanilla OPD | arXiv 2606.17199 |
| IW-OPD (2026.06) | White-Box OPD | Discovers a positional bias: supervising only the first 30% of tokens matches full-token supervision, while the last 30% is nearly useless; derives the theoretically optimal weight as the teacher/student probability ratio, implemented via accumulated unsigned prefix-deviation weighting + γ-mixing with standard OPD; +6.9 points at step 10 on AIME2025, +1.7 points at convergence, +1.3~+2.2 points at extreme 235B→30B scale; side finding: KL drops only 20% after OPD convergence, revealing that OPD is fundamentally local-neighborhood optimization rather than global alignment | arXiv 2606.22600 |
| ATOD (2026.06) | OPD × RL Hybrid · Agent OPD | An annealing schedule lets OPD dominate early cold-start while RL takes over later to surpass the teacher; T-DUR uses a divergence × uncertainty soft weight to focus on key steps within multi-turn trajectories; a 0.6B student surpasses its 4B teacher on both ALFWorld and WebShop, averaging +3.03 points over OPD and +23.62 points over GRPO | arXiv 2606.27814 |
| DanceOPD (2026.06) | Image-Generation OPD · Multi-Capability Fusion | Abstracts multiple frozen image-generation experts (text-to-image / local editing / global style) each as a velocity field, using sample-level hard routing (each sample assigned to exactly one field) + single-point on-policy semantic-side queries to eliminate gradient conflicts and distribution bias; a combined text-to-image + editing model gets +8.1% on GEditBench; key finding: dense querying (K=16) is actually 16.6% worse than a single query, and the merged student surpasses any individual teacher | arXiv 2606.27377 |
| DEAR (2026.06) | White-Box OPD | Splits the reasoning chain into "decisions" (high-entropy branch points for the student) and "evidence" (positions where the hidden state resembles a decision but the student is confidently wrong, with a large teacher-student divergence); uses cosine similarity of the student's hidden states as a relevance gate combined with the teacher-student log-probability gap as a gap amplifier, multiplicatively scoring and selecting the top-q% evidence, supervising only decisions ∪ evidence (~36% of tokens); 2.2×/2.7× improvement over standard OPD on AIME 2024/2025, +5.74pp on MBPP code; counter-intuitive: decision-only supervision covers just 39.1% of the gradient mass (vs. 35.9% random), meaning evidence alone drives almost all the gain, with DEAR reaching 75.8% coverage | arXiv 2606.22830 |
| MOPD (2026.06) | Industrial Multi-Teacher · White-Box OPD | A multi-domain post-training paradigm: starting from the same SFT base, same-origin domain-expert teachers are trained in parallel (math/code/instruction each via RL), then each problem is routed to only its corresponding domain teacher for per-token reverse-KL distillation, with no signal merging whatsoever; the teacher-student log-difference is fed directly into the PPO/GRPO framework as the advantage; a 30B student scores a normalized 0.937 (vs. 0.882 for mixed training and 0.775 for sequential training), with the tightest spread across five domains (0.044), and sample efficiency of 25–30K vs. 150–180K for mixed training; already deployed in the 309B MiMo-V2-Flash; counter-intuitive: swapping in a stronger but distant 235B external teacher actually breaks the student (5× higher initial KL, entropy drops 0.30→0.21, top-k collapses at step 18) — same-origin matters more than raw capability | arXiv 2606.30406 |
| OPID (2026.06) | Agent OPD · OPD × RL Hybrid | Extracts hierarchical hindsight skills (episode-level global workflows + step-level critical decisions) from successful trajectories, uses critical-first routing for adaptive selection, and injects token-level self-distillation advantage into the advantage function; 84.3% on ALFWorld (+9.3pp), 74.2% on WebShop (+10.9pp), matches full-data GRPO with only 60% data, zero external dependencies at inference; counter-intuitive: Skill-GRPO drops −14.8~−25.0 points when inference-time skills are removed, proving skill-crutch dependency is toxic | arXiv 2606.26790 |
| SEED (2026.07) | Agent OPD · OPD × RL Hybrid | Self-evolving hindsight distillation: the model serves simultaneously as actor (collecting trajectories) and analyzer (extracting skills), with supervision signals co-evolving with the policy; two-stage training (cold-start SFT + self-evolving RL), skill-induced log-probability shift gated into advantage; 91.8% on ALFWorld (+16.8pp vs. GRPO), 78.9% on WebShop (+15.6pp), beats full-data GRPO with 60% data; counter-intuitive: static skill library drops 7.4 points, skills as prompts underperform distillation internalization | arXiv 2607.14777 |
Related Work
The following are neighboring-field papers frequently cited or borrowed from in OPD research, grouped by topic.
Related Work: Agentic RL Environments
| Paper | Summary | Link |
|---|---|---|
| R2E-Gym (2025.04) | A programmatically synthesized SWE training environment: generates 8135 problems via commit backtranslation (only 0.2pp gap between synthetic and real issues); a hybrid verifier combines execution-based and execution-free checks to cover each other's blind spots; a 32B model reaches 51% Best@26, the first open-source result approaching Claude Sonnet's commercial-level performance | arXiv 2504.07164 |
| SWE-Smith (2025.04) | Actively manufactures bugs to scale up SWE training data: four bug-injection strategies (program transformation / LM rewriting / PR reversal / multi-hole merging) generate 50K problems across 128 repos, with repo-level image reuse compressing storage 500× at $0.025/problem; a 32B model reaches 40.2% on SWE-Bench, open-source SOTA; key finding: harder problems aren't more valuable for training than easy ones — repo diversity is the deciding factor | arXiv 2504.21798 |
| SWE-Factory (2025.06) | A multilingual SWE data pipeline (Python/Java/TS/Go): a 4-agent SWE-Builder system auto-configures test environments, using exit codes instead of log parsing (F1=0.99), with an environment-image pool reusing configs across versions; GPT-4.1 mini achieves a 50.2% F2P rate at $0.047/instance; Qwen2.5-Coder-14B jumps from 5.8% to 21.0% on SWE-Bench after training | arXiv 2506.10954 |
| LLM-in-Sandbox (2026.01) | (Sandbox construction) A minimal three-tool sandbox (bash/file_editor/finish) gives rise to three general meta-abilities: external resource access, file management, and code execution; 7 frontier models gain up to +15.5% across 6 domains with no training and 8× lower token cost; LLM-in-Sandbox-RL trains weaker models on general data, raising the tool-call rate from <3% to 4–7% | arXiv 2601.16206 |
| SkillRL (2026.02) | Distills success/failure trajectories into a hierarchical skill library (general + task-specific, 10–20× compression), with recursive co-evolution during GRPO training (55→100 skills); 89.9% on ALFWorld (+12.3pp vs. GRPO), surpassing GPT-4o's 41.9%; abstraction beats memorization (replacing with raw trajectories costs −25%) | arXiv 2602.08234 |
| AgentRL (2025.10) | A fully asynchronous Rollout/Actor/Reference three-worker decoupled pipeline with a unified Function-Call API; cross-policy sampling (mixing current + historical checkpoints) resolves multi-turn exploration collapse; task-level advantage normalization stabilizes multi-task training; surpasses GPT-4/Claude-Sonnet-4/DeepSeek-R1, with multi-task performance matching single-task experts | arXiv 2510.04206 |
| AgentGym-RL (2025.09) | Proposes a ScalingInter-RL curriculum: limiting then expanding the number of interaction steps to address sparse-reward cold-start in long-horizon RL, with no SFT needed; a decoupled EnvServer/EnvClient architecture supports 14 environments and 27 tasks; pure RL from scratch matches or surpasses commercial models like GPT-4 | arXiv 2509.08755 |
| Chain-of-Agents (2025.08) | Distills three specialized multi-agent systems (Web/Code/MHQA) into a single AFM (masked fine-tuning computing loss only on action-reasoning tokens), then refines it with DAPO/PPO agentic RL; 55.3% Pass@1 / 69.9% Pass@3 on GAIA; fully open-sourced at 3B/7B/32B | arXiv 2508.13167 |
| SPA-RL (2025.05) | Trains a lightweight MLP progress estimator, using a potential function to decompose the terminal reward into per-step contributions (telescoping preserves the total sum); combines progress reward + grounding reward to replace the sparse terminal reward; +2.5% success rate across three benchmarks, +25% vs. PPO on long-horizon settings of 25–29 steps | arXiv 2505.20732 |
| GiGPO (2025.05) | (NeurIPS 2025) Adds a step-level advantage on top of GRPO's trajectory-level advantage: identifies identical environment states across rollouts (anchors) to build step-level contrastive groups, A=A^E+ω·A^S; requires no extra model, <0.002% overhead; +13.9pp on ALFWorld (72.8%→86.7%), +7.7 on WebShop vs. GRPO | arXiv 2505.10978 |
| SkyRL-Agent (2025.11) | A fully asynchronous three-stage pipeline (init/execution/reward run in parallel) reaching ~90% GPU utilization and 1.55× throughput; an AST-based code-search tool reduces code-localization failures, hitting SOTA at 125 steps; SA-SWE-32B: SWE-Bench 24.4%→39.4%, at 2× the training-cost efficiency of DeepSWE | arXiv 2511.16108 |
| VAGEN (2025.10) | (NeurIPS 2025) A POMDP framework with explicit world-model reasoning (structured output tags for state estimation + transition prediction); world-model reward provides dense intermediate signal; Bi-GAE estimates advantage at both the turn and token level; a 3B model scores 0.82 overall, surpassing GPT-5 (0.75) / Gemini 2.5 Pro (0.67) / Claude 4.5 (0.62) | arXiv 2510.16907 |
Related Work: RL Post-Training Data Engineering
| Paper | Summary | Link |
|---|---|---|
| SAERL (2026.05) | Uses SAE internal activations to extract semantic coordinates for 103K training problems, then applies quality filtering (99.92% precision), curriculum difficulty ranking (Spearman 0.749), and moderate batch mixing (mix₈ is the sweet spot; over-mixing hurts); +3.00% vs. vanilla GRPO with 20% fewer steps, at a preprocessing cost of just 0.5 GPU-hours (34× cheaper than fast-rollout approaches) | arXiv 2605.27354 |
Related Work: Multi-Reward RL
| Paper | Summary | Link |
|---|---|---|
| MVA (2025.11) | Two-stage decoupled multi-value alignment: HSIC regularization pushes mutual information between value vectors toward zero (eliminating parameter interference), while an extrapolation weight in [0,C] breaks past the limits of convex interpolation to explore the Pareto front; full MVA vs. without HSIC: helpfulness 1.66 vs. 0.66 (2.5×), harmlessness 0.61 vs. 0.33; remains effective when extended to three value dimensions | arXiv 2511.17579 |
| GDPO (2026.01) | Fixes the normalization order for multi-reward RL: normalize each reward independently within-group before summing (rather than summing then normalizing), eliminating "advantage collapse" (different reward combinations compressing to the same advantage value); a conditioning strategy addresses weight-adjustment failure when reward difficulty is highly skewed; +2.6pp on tool-calling, +6.3pp on math (AIME) at 1.5B, and bug rate on a 3-reward coding task drops from 3.9% to 2.5% | arXiv 2601.05242 |
| RVPO (2026.05) | A variance penalty for multi-reward RL: replaces "mean aggregation" with SoftMin (a soft-minimum operator), equivalent to mean − k/2 × variance, forcing the model to shore up weak dimensions rather than letting high-scoring ones carry the average; a k-annealing curriculum (0.5→2.0) ensures stability; a 14B model's HealthBench score rises from 0.215 to 0.261, while GDPO collapses to zero late in 14B training and RVPO holds at 0.236 | arXiv 2605.05750 |
| RDPO (2026.05) | Multi-objective mixed-reward RL training: MAQ quantile normalization unifies heterogeneous reward scales (handling binary/continuous/piecewise rewards), and Mahalanobis whitening removes redundancy across reward dimensions; introduces η_eff = η_proj × η_corr to diagnose the effective information content of multiple rewards; +10.91pp on IFEval vs. GRPO, and ArenaHard-Hard (large-scale) rises from 65.80% to 76.10% | arXiv 2605.13641 |
Related Work: RLVR Mechanism Analysis
| Paper | Summary | Link |
|---|---|---|
| RM Overoptimization Scaling Laws (2022.10) | The first quantitative study of RLHF reward-model overoptimization: using synthetic labels from dual gold/proxy RMs, finds that both BoN and RL gold scores first rise then fall, with the curve's coefficients scaling smoothly (log-linearly) with RM size; key finding: KL penalty is equivalent to early stopping (it doesn't improve the KL-gold frontier), and larger policy models plateau at the same KL as smaller ones | arXiv 2210.10760 |
| Sparse but Critical (2026.03) | A token-level dissection of RLVR: after DAPO training, 83%+ of positions have near-zero divergence (98%+ for SimpleRL); injecting <4% of RL tokens into the base model fully recovers the gain (AIME 2024 8%→25%), while removing the top 5% critical positions collapses everything; RL changes candidate ranking rather than the candidate set (80% of critical tokens are already in the base model's top-3), and the SFT distribution shift is far wider than RLVR's | arXiv 2603.22446 |
| Inference Manifold (2026.05) | A geometric diagnosis of reasoning quality: reasoning trajectories spontaneously compress to 5–10 intrinsic dimensions (out of 4096), but compression plus rising information volume (deeper layers have lower dimension yet more information) is the true mark of healthy reasoning; a three-factor health score H (base expressiveness × information volume ÷ dimensional-scatter penalty) correlates with all tested benchmarks at Spearman >0.9, entirely label-free | arXiv 2605.08142 |
| Negative RL (2026.06) | Splits RLVR into PSR (rewarding correct answers) and NSR (penalizing wrong ones): NSR trained alone reaches 53.3% AIME Pass@256, beating GRPO (50.0%) and PPO (43.3%), while PSR trained alone is actually below the base model's Pass@256; PSR collapses diversity by locking in correct paths, while NSR preserves diversity by only clearing roadblocks rather than dictating a path; W-REINFORCE (0.1·PSR + NSR) matches PPO's Pass@1 while achieving the highest Pass@k | arXiv 2506.01347 |
| DRPO (2026.06) | An improved RLVR trust-region algorithm: diagnoses the root cause of ratio-based methods (PPO/GRPO/SPO) having unbounded gradient weights on low-frequency tokens and collapsing under FP8; replaces the ratio-based ruler with an absolute-probability-difference (Binary-TV) measure, turning DPPO's hard clipping into a bounded, continuous spring-like penalty; key finding: the corrective signal beyond the boundary matters more than the smooth decay within it; ratio-based methods degrade severely under end-to-end FP8 training on Qwen3-30B while DRPO remains stable | arXiv 2606.09821 |
| RELEX (2026.05) | RLVR's weight-update trajectory is extremely low-rank (rank-1 accounts for 80%+ per layer) with coefficients evolving nearly linearly (R²>0.98); observing just 15% of the training steps and applying SVD + linear extrapolation directly predicts the final weights, saving 85% of training; the SVD projection also denoises, so OOD benchmarks actually beat full training (AIME 2025 8.8% vs. 4.6%, HMMT 2.1% vs. 0.4%), with MATH performance on par | arXiv 2605.21468 |
Related Work: OPD Mechanism Analysis
| Paper | Summary | Link |
|---|---|---|
| OPD Geometry (2026.06) | Diagnoses OPD's parameter-space trajectory along three dimensions (stable rank / principal-component rotation / spectral drift): by 20% of training it locks into a narrow ~16-dimensional channel (subspace locking); restricting training to that subspace leaves OPD performance unchanged while SFT degrades sharply; the composition of the objective function — not token selection — is the only lever that changes the channel; OPD isn't a compromise between SFT and RLVR, but an independent third path | arXiv 2606.07082 |
| Dense Supervision Sparse Updates (2026.06) | OPD's parameter updates are coordinate-sparse (66–89% untouched) and spectrally concentrated (stable rank 8–12 vs. 83 for offline distillation), while actively avoiding the source model's principal singular directions (only about half the energy expected under random chance lands there); OPD and RLVR share the same set of plastic coordinates (3× overlap over a random baseline); it's AdamW, not the raw gradient, that produces the sparsity — the optimizer distills a sparse signal out of a dense gradient | arXiv 2606.13657 |
Related Work: Two-Stage Policy Decoupling
| Paper | Summary | Link |
|---|---|---|
| ERPD (2026.05) | A two-stage decoupling: stage one relaxes the KL constraint, letting the policy fly into "extreme regions" to extract maximal signal; stage two uses trust-region distillation to recycle the valuable parts back into the base model while filtering out invalid drift; the distilled student's KL is just 0.01 (vs. the teacher's 0.041, 4× smaller) with accuracy nearly unchanged; even a degraded teacher (trained in reverse via MSE) can still provide an effective distillation signal, addressing the case where the teacher is weaker than the student | arXiv 2605.25582 |
awesome-list
knowledge-distillation
large-language-models
llm
llm-agent
on-policy-distillation
paper-list
reinforcement-learning
Similar repos
Contributors
kissshhot
4 commits