Prime Values

arXiv Paper · Project Page

@misc{venkatraman2026lecritiqueprivilegedvalue,
      title={Le Critique: Privileged Value Functions for LLM Reinforcement Learning},
      author={Siddarth Venkatraman and Matthieu Dinot and Laurence Aitchison},
      year={2026},
      eprint={2608.16739},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.16739},
}

Prime Values extends PRIME-RL with asynchronous value functions and rollout replay. It keeps critic learning outside the policy trainer: a value evaluator scores each rollout's causal states, the orchestrator turns those predictions into per-token GAE and value targets, and the policy receives the pure value advantage without owning any critic model or loss.

Completed rollouts enter a bounded, nonblocking queue and a rollout-granular replay buffer. The value trainer samples critic batches independently of policy updates, publishes monotonically versioned weights, and lets the evaluator adopt those weights without coupling critic progress to policy versions. Value evaluation can run on dedicated GPUs for independent throughput or queue on the value trainer's GPUs when a smaller deployment matters more.

See Value Functions for topology, replay semantics, configuration, and monitoring.

Architecture at a glance

PRIME-RL: Async RL Training at Scale

Overview

PRIME-RL is a framework for large-scale reinforcement learning. It is designed to be easy to use and hackable, yet capable of scaling to 1000+ GPUs. Here is what we think sets it apart:

1. Fully asynchronous RL for high-throughput agentic training at scale.
2. Performant: built to train 1T+ MoE models on 1000+ GPUs with FSDP2 for training and vLLM for inference, with FP8 inference, PD disaggregation, EP and CP parallelism, and more.
3. Native integration with verifiers environments through the Environments Hub, including built-in support for SWE and agentic environments.
4. End-to-end post-training: SFT, RL training, and evals.
5. Multi-node deployment with Slurm and Kubernetes support.
6. Multimodal support for VLMs such as Qwen3-VL.
7. Hackable, modular, and extensible by design.
8. One-line SLURM deployment for frontier models — e.g. GLM-5 FP8 with P/D disaggregation, the llm-d router, and Mooncake KV offload.

Models support

The trainer works with both Hugging Face and Prime custom ModelForCausalLM out of the box. For selected families (especially large MoE) we also ship highly optimized training code under src/prime_rl/trainer/models/, including expert parallelism (EP) for MoE layers and context parallelism (CP) for long sequences (see the table), and additional kernels like quack-kernels.

With [model] impl = "auto" (the default), the trainer selects that custom stack when the Hugging Face config type is registered.

Setup

We develop and test on NVIDIA RTX 3090/4090/5090, A100, H100, H200, and B200. If your setup fails, please create an issue.

Prerequisites

Currently, you need at least one NVIDIA GPU to use PRIME-RL. If you don't already have access to one, we recommend our compute platform for everything from renting on-demand single GPUs for developing, debugging and small ablations, to reserving 1000+ GPU clusters for production-scale training.

Quick Setup

Set up PRIME-RL in a single command.

curl -sSL https://raw.githubusercontent.com/PrimeIntellect-ai/prime-rl/main/scripts/install.sh | bash

git clone https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl

git submodule update --init -- deps/verifiers deps/renderers deps/research-environments deps/pydantic-config

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv sync --all-extras

bash scripts/docker-arm64-post-install.sh

uv pip install "flash-attn-3 @ git+https://github.com/Dao-AILab/flash-attention.git@main#subdirectory=hopper" --no-build-isolation

uv run python -V

uv run python -c "import flash_attn"

uv run sft @ configs/debug/sft/train.toml

uv run trainer @ configs/debug/rl/train.toml

uv run inference @ configs/debug/infer.toml

uv run orchestrator @ configs/debug/orch.toml

uv run eval @ configs/debug/eval.toml

Additional Setup

1. If you want to log your runs to W&B, log in

uv run wandb login
# Or set `export WANDB_API_KEY=...`

2. If you require gated/ private models or datasets from HuggingFace, log in

uv run hf auth login
# Or set `export HF_TOKEN=...`

Training Examples

We provide end-to-end training examples in the examples directory to highlight features of the framework and guide you through the process of training your own models.

Basic Training: 1 to 8 GPUs

Follow this guide to learn the basics of Prime-RL. You can train your own models on 1 to 8 GPUs. Ideal for getting started and exploring the capabilities of the framework. These guides cover most use cases -- single-turn, multi-turn, tool calling, etc. -- on toy environments and small models.

1. Reverse Text: Train Qwen3-0.6B to reverse a small chunk of text. Demonstrates tiny-scale single-turn SFT and RL training. Can be trained on a single consumer GPU in a few minutes, and is ideal for getting started.
2. Wordle: Train Qwen3-1.7B to play Wordle. A fun example of multi-turn SFT and RL training. Can be trained on a 2-4 H100 GPUs in a few hours. Ideal for exploring the multi-turn training capabilities of the framework.
3. Alphabet Sort: Train Qwen3-4B-Instruct-2507 to sort names alphabetically. Demonstrates multi-turn RL training via LoRA without SFT warmup. Can be trained on a single H100 GPU in just over an hour. Ideal for exploring LoRA-based training.
4. Wiki Search: Train Qwen3-4B-Instruct-2507 to answer trivia questions by searching through a Wikipedia. Demonstrates multi-turn with web search tool use.
5. Hendrycks Sanity: Run a sanity check experiment on DeepSeek-R1-Distill-Qwen-1.5B using a filtered subset of MATH where the model already partially solves 20-80% of problems. Useful for algorithm ablations.

Advanced Training: 32 - 2048 GPUs:

Follow this guide to train large models on hard reasoning and agentic / swe environments. These guides are designed to be run from a Slurm cluster but can also be adapted to k8s deployments.

1. Qwen 3 30B - A3B Math: Train Qwen3-30B-A3B to solve hard math problems.
2. Qwen 3 30B - A3B SWE: Train Qwen3-30B-A3B to solve hard SWE problems.
3. Intellect-3.1: Reproduce our INTELLECT-3.1 training run.
4. MiniMax-M2.5 SWE: Train MiniMax-M2.5 on agentic SWE tasks.
5. High-throughput GLM-5: Train GLM-5 with PD disaggregation and FP8 inference on SWE.
6. High-throughput GLM-5 (llm-d): One-line SLURM deployment for GLM-5 FP8 with P/D disaggregation, the llm-d router, and Mooncake KV offload — the faster way to run GLM-5.

Docs

Check out the docs directory for in-depth guides on how to use PRIME-RL.

• Overview - Architecture, install, and a copy-pasteable end-to-end RL run
• Configuration - TOML composition, CLI overrides, env vars, validation
• Training - RL, SFT, evals, checkpointing, observability, rules of thumb
• Value Functions - Async critic topology, GAE targets, baselines, warmup, and checkpoints
• Scaling - Single-GPU through multi-node, FSDP/EP/CP, SLURM, benchmarking
• Algorithms - Async/off-policy training, the AIPO loss, advantage and filter plugins, trajectory merging
• Advanced - Custom modeling, multimodal training, LoRA, multi-tenant training
• Development - Test suite, pre-commit hooks, adding a new model

Contributing

We warmly welcome community contributions! We use issues to track bugs, feature requests, and share our internal roadmap. If you encounter bugs, have pain points during development, or have ideas for new features, please open an issue.

Contributions are welcome via PR. Please follow these guidelines:

1. Install the pre-commit hooks to ensure your code is formatted correctly.
2. Please keep your PR in "Draft" until it is ready for review.
3. If your PR resolves an issue, please link the issue in the PR description
4. If you can, try running the test suite locally to ensure your changes are working as expected.

Pre-Commit Hooks

Please install the pre-commit hooks to ensure your code is formatted correctly.

uv run pre-commit install

Tests

uv run pytest -v                    # everything
uv run pytest tests/unit -v         # unit only
uv run pytest tests/integration -v  # integration only
uv run pytest -v -m "not gpu"       # CPU-only (inverse of the gpu marker)

License

This project is licensed under the Apache 2.0 license, as found in the License file.

Upstream PRIME-RL Citation

Prime Values extends PRIME-RL. To cite the upstream project, use:

@misc{primeintellect2025prime-rl,
  author = {Prime Intellect},
  title = {PRIME-RL},
  url = {https://github.com/PrimeIntellect-ai/prime-rl},
  year = {2025}
}