✨ GEPO: Group Expectation Policy Optimization for Heterogeneous RL

📅 Release Date: Aug 25, 2025 (arXiv)
📄 Paper: Group Expectation Policy Optimization for Heterogeneous Reinforcement Learning
🧑‍💻 Authors: Pengcheng Lab / Heterogeneous Large Model Research Team

⚡ Why It Matters

Training giant AI models now requires global, decentralized compute. But network delays cause “policy staleness,” making traditional RL algorithms (like GRPO, GSPO) crash due to exploding gradient variance.

GEPO solves this. By replacing unstable per-token weights with Group Expectation Importance Weighting, it exponentially reduces variance under high latency — enabling stable training even with 1800-second delays.

✅ Theoretically Proven: Exponentially reduces importance sampling variance (Theorem 1).
✅ Extremely Robust: Only 3% performance drop under extreme 1800s latency.
✅ Plug-and-Play: Easy to integrate — modifies only the importance weight calculation.
✅ Better Everywhere: Outperforms GRPO/GSPO even in zero-delay (online) settings.

📊 Key Results (Qwen3-1.7B):

• Zero-Delay: GEPO Last = 41.4 vs. GSPO Last = 24.3 (+17.1 gain).
• High-Delay (64 steps): GEPO Last = 43.5 (no drop) vs. GSPO Last = 20.9.
• Extreme Test (1800s): Performance degradation < 3%.

📋 Full Benchmark Comparison

🧠 The Core Idea: Think Groups, Not Tokens

Traditional methods use p(y|x) / q(y|x), which explodes when q(y|x) is small. GEPO’s genius is simple:

Group Expectation Weight: w_GEPO(y|x) = p(y|x) / Ê_q[q(y|x)]

Where Ê_q[q(y|x)] is estimated from a group of responses {y1...yG} for the same prompt x: Ê_q[q(y|x)] ≈ Σ(q(yi|x)²) / Σ(q(yi|x))

This group-level denominator smooths out wild fluctuations, preventing gradient explosions and keeping training stable — no matter how stale the data is.

Figure 1: GEPO improves upon GRPO and GSPO by employing group-level importance weights to enhance training stability. It demonstrates superior performance in both zero-delay (online) and high-delay (up to 1800s) heterogeneous RL scenarios.

🚀 The Future: Decentralized AI is Here

GEPO is the engine of HeteroRL, a framework that decouples sampling and learning across global nodes. This isn’t just an algorithm — it’s the foundation for community-driven, globally distributed AI training.

💡 Pro Tip:

• Use GEPO as your default RL algorithm — it’s more stable everywhere.
• For maximum robustness in production, combine it with the “Defensive Sampling” mechanism (Appendix F).

✨ Clipped IS-weight Policy Optimization (CISPO): Stable & Efficient RL without Token Clipping

📅 Release Date: Jun 16, 2025 (arXiv)
📄 Paper: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
🧑‍💻 Authors: MiniMax AI Team
🔗 Implementation: Integrated into HeteroRL; compatible with hybrid-attention and long-context models

⚡ Why It Matters

Traditional PPO/GRPO-style algorithms clip token-level policy updates to enforce trust-region constraints. However, this inadvertently discards critical low-probability tokens (e.g., reasoning “forks” like “Wait”, “Recheck”) that are essential for exploration and scalable reasoning.
CISPO resolves this by clipping the importance sampling (IS) weights instead of the token updates, preserving gradient signal from all tokens while stabilizing off-policy training.

✅ No dropped tokens – critical rare tokens remain learnable
✅ Lower training variance via bounded IS weights
✅ Faster convergence: 2× speedup over DAPO on math RL tasks
✅ Plug-and-play: drop-in replacement for GRPO/DAPO in token-level RL setups

💡 Pro Tips:

• Use asymmetric clipping: typically only upper-bound IS weights (e.g., clip(·, 0, 1+ε) with ε=0.2–0.4)
• Combine with group-relative advantage normalization (as in GRPO) for stable reward scaling
• Avoid KL penalties — CISPO is designed to be KL-free and more compute-efficient
• Ideal for long CoT and hybrid-attention architectures where token survival is crucial

✨ Tapered Off-Policy REINFORCE (TOPR): Stable & Efficient Off-Policy RL for LLMs

📅 Release Date: Mar 20, 2025 (arXiv)
📄 Paper: Tapered Off-Policy REINFORCE
🧑‍💻 Authors: Nicolas Le Roux, Marc G. Bellemare, Jonathan Lebensold, et al. (Mila & Reliant AI)
🔗 Implementation: Integrated into HeteroRL with native support for negative examples and asynchronous rollout datasets

⚡ Why It Matters

Standard REINFORCE collapses in off-policy settings due to unbounded negative updates, while many popular methods (e.g., PPO, DPO) either discard negative examples or suffer from limited off-policy capability.
TOPR resolves this with an asymmetric, tapered importance sampling scheme that:

✅ Stably leverages both positive and negative examples without KL regularization
✅ Accelerates learning on rare positives via SFT-style updates (clipping lower bound = 1)
✅ Prevents destructive updates from negatives via truncated importance ratios (clipping lower bound = 0)
✅ Matches 70B-model performance with only 8B parameters through iterative dataset curation (e.g., Anna Karenina sampling)

💡 Pro Tips:

• Always keep negative examples — they dramatically improve data efficiency and reduce “wasted inference.”
• Tune the baseline parameter not for variance reduction, but to control effective positive ratio (~10–20% is optimal).
• Combine TOPR with dynamic dataset balancing (e.g., favoring hard negatives) for multi-iteration gains.
• Avoid gradient clipping values >1.0 unless using ratio truncation — standard IS can explode on skewed datasets.

✨ Geometric-Mean Policy Optimization (GMPO): Stabilizing GRPO with Outlier-Robust Aggregation

📅 Release Date: Jul 28, 2025 (arXiv)
📄 Paper: Geometric-Mean Policy Optimization
🧑‍💻 Authors: Yuzhong Zhao, Yue Liu (UCAS), Junpeng Liu (CUHK), Jingye Chen (HKUST), and Microsoft Research Team
🔗 Implementation: Based on GMPO

⚡ Why It Matters

GRPO optimizes the arithmetic mean of token-level rewards, which is highly sensitive to outlier importance-weighted rewards, causing unstable policy updates and extreme importance sampling ratios.
GMPO addresses this by switching to the geometric mean, which is inherently robust to outliers and leads to:

✅ Stable importance sampling ratios (narrower range, fewer extremes)
✅ Lower training variance and more reliable gradients
✅ Enhanced exploration via wider clipping (e.g., (e⁻⁰·⁴, e⁰·⁴)) without sacrificing stability
✅ Consistent gains: +4.1% on math benchmarks and +1.4% on multimodal reasoning (Geometry3K)

💡 Pro Tips:

• Use token-level clipping (not sequence-level) for finer gradient control.
• Set clipping range to (e⁻⁰·⁴, e⁰·⁴) to balance exploration and stability.
• GMPO maintains higher token entropy and lower KL divergence from the pre-RL model — ideal for scalable RL training.

✨ ∆L Normalization: Rethink Loss Aggregation in RLVR

📅 Release Date: Sep 9, 2025 (arXiv)
📄 Paper: ∆L Normalization: Rethink Loss Aggregation in RLVR
🧑‍💻 Authors: Zhiyuan He, Xufang Luo (Microsoft Research), Yike Zhang (Tsinghua), et al.
🔗 Implementation: Based on Delta-L-Normalization

⚡ Why It Matters

In RLVR, response lengths vary dramatically — leading to high gradient variance and biased updates in existing methods (GRPO, DAPO, Dr. GRPO).
∆L Normalization solves both:

✅ Unbiased estimator of true policy gradient
✅ Theoretically minimal variance (when α=1)
✅ Plug-and-play — <10 lines to integrate

💡 Pro Tip:

• Use α=1 for minimum variance (default, stable training).
• Use α=0.75 for Math tasks — better utilization of long, informative responses.

✨ GSPO: Group Sequence Policy Optimization for Scalable RL

📅 Release Date: July 28, 2025 (arXiv v2)
📄 Paper: Group Sequence Policy Optimization
🧑‍💻 Authors: Chujie Zheng, Shixuan Liu, Mingze Li, Bowen Yu, et al. (Qwen Team, Alibaba)

⚡ Why It Matters

Existing methods like GRPO suffer from catastrophic instability when scaling to large models — especially MoE architectures — due to noisy token-level importance ratios.
GSPO fixes this at the root:

✅ Sequence-level importance weights — Matches reward granularity & reduces variance
✅ Stable MoE training — No “Routing Replay” hacks needed 🚫
✅ Higher training efficiency — Achieves better performance with same compute
✅ Simpler infrastructure — Compatible with inference-engine likelihoods (no recompute needed)

💡 Pro Tip:

• Use clip_range=(3e-4, 4e-4) for optimal stability (default in Qwen3 RL training).
• For multi-turn RL, try GSPO-token variant — enables per-token advantage while preserving sequence-level stability.

✨ Dr. GRPO: Group Relative Policy Optimization Done Right

📅 Release Date: March 26, 2025 (arXiv)
📄 Paper: Understanding R1-Zero-Like Training: A Critical Perspective
🧑‍💻 Authors: Zichen Liu, Changyu Chen, Wenjun Li, et al. (Sea AI Lab, NUS, SMU)

⚡ Why It Matters

Original GRPO introduces length bias and difficulty bias — artificially inflating response lengths (especially for incorrect answers) and skewing updates toward “easier” questions.
Dr. GRPO removes these biases at the source:

✅ Unbiased gradient estimator — Faithfully implements true policy gradient objective
✅ Token-efficient training — Prevents wasteful generation of long, incorrect responses 🚫📏
✅ Plug-and-play replacement — Drop-in substitute for GRPO with minimal code change
✅ Preserves reasoning performance — Matches or exceeds GRPO’s final accuracy with less compute

💡 Pro Tip:

• Use Dr. GRPO when you want stable length growth (only for correct reasoning, not noise).
• Combine with ∆L Normalization for double variance reduction + unbiasedness.

✨ BNPO: Beta Normalization Policy Optimization for Stable RL Training

📅 Release Date: June 3, 2025 (arXiv)
📄 Paper: BNPO: Beta Normalization Policy Optimization
🧑‍💻 Authors: Changyi Xiao, Mengdi Zhang, Yixin Cao (Fudan University, Meituan)

⚡ Why It Matters

Current RL methods like GRPO and REINFORCE use static reward normalization — fixed throughout training — which fails to adapt to the evolving policy distribution, leading to unstable gradients and suboptimal convergence.
BNPO solves this with dynamic, theoretically grounded normalization:

✅ Adaptive Beta normalization — Parameters (α, β) update dynamically with policy evolution
✅ Proven variance reduction — Theoretically minimizes gradient variance under binary rewards
✅ Generalizes GRPO & REINFORCE — Reduces to them under specific (α, β) settings
✅ Handles complex rewards — Via novel Advantage Decomposition mechanism

💡 Pro Tip:

• BNPO automatically sets α = (1+a)/3, β = (1+b)/3 — no manual tuning needed.
• Use Advantage Decomposition when combining multiple reward signals (e.g., accuracy + format).