DiffusionGemma-26B-A4B-it-ORL-NVFP4

DiffusionGemma + ORL, NVFP4: DiffusionGemma-26B-A4B-it-ORL with its MoE experts quantized to NVFP4 for NVIDIA Blackwell GPUs, at comparable accuracy

Highlights

• 4,214 tokens/s for a single request on GSM8K on one B200, 2.7–2.9× the BF16 checkpoint on one H100 on every benchmark
• 1.14–1.30× faster than NVIDIA's NVFP4 DiffusionGemma (nvidia/diffusiongemma-26B-A4B-it-NVFP4) for a single request on the same B200, at comparable accuracy
• 7,708 tokens/s on eight B200s with tensor parallelism (GSM8K)
• Fast on a consumer GPU: 1,540 tokens/s on GSM8K on one 32 GB GeForce RTX 5090, matching or exceeding BF16 on an H100 on every benchmark

Results

Accuracy (%) / TPF with the ORL decoder and the paper's scorers:

With tensor parallelism across eight B200s, accuracy is 96.1 / 91.4 / 92.7 / 86.0 on GSM8K / MATH-500 / HumanEval / MBPP; tensor parallelism does not change the tokens per forward.

Speed

Tokens per second for a single request with dlmrt (the paper's NVFP4 tables):

The 8× B200 and RTX 5090 rows use generation seed 1234.

Comparison with NVIDIA's NVFP4 DiffusionGemma

NVIDIA's nvidia/diffusiongemma-26B-A4B-it-NVFP4 quantizes the DiffusionGemma base model to NVFP4. With ORL, this checkpoint decodes more tokens per forward at comparable accuracy (within 0.8 points) and is 1.14–1.30× faster for a single request, both measured back to back with dlmrt on the same B200. Accuracy (%) / TPF, and the speedup:

Quantization

Exported with NVIDIA ModelOpt (hf_quant_config.json), experts-only recipe with Diagonal-Hessian calibration. The export also keeps BF16 copies of the expert weights, which NVFP4 inference does not read; this is why the checkpoint is larger than the BF16 one.

Usage

Native FP4 needs an NVIDIA Blackwell GPU: a B200 or B300, or a GeForce RTX 50-series card such as the RTX 5090. Stock vLLM and SGLang do not run DiffusionGemma's block-diffusion decoding; the ORL repository's patched vLLM serves the BF16 checkpoint.

dlmrt

dlmrt, our inference engine, decodes one request at a time with the paper's decoder (role="orl") and measures the paper's throughput; see its README for installation, tensor parallelism and the commands of every paper cell.

from dlmrt import LLM

llm = LLM("yifanyu/DiffusionGemma-26B-A4B-it-ORL-NVFP4", role="orl", precision="nvfp4")
out = llm.generate("Write a Python function that reverses a linked list.", max_new_tokens=2048)
print(out.text)
print(f"{out.request_tps:.0f} tokens/s")

python -m dlmrt serve --model yifanyu/DiffusionGemma-26B-A4B-it-ORL-NVFP4 --role orl --precision nvfp4 --port 18080
curl -N localhost:18080/generate -d '{"messages": [{"role": "user", "content": "Hi!"}], "max_new_tokens": 512}'

For tensor parallelism, run the same program or dlmrt serve under torchrun --nproc-per-node 8.

Checkpoint identity

Paper role dg-orl, training step 900, quantized from yifanyu/DiffusionGemma-26B-A4B-it-ORL. The tokenizer and chat template are the BF16 checkpoint's. Decoding settings are the same as for the BF16 checkpoint (paper_decoding.json there).

Related Models

Citation

@misc{yu2026orl,
  title         = {On-Policy Residual Learning for Diffusion Language Models},
  author        = {Yu, Yifan and Jian, Yuqing and Li, Jisen and Hu, Yuezhou and Zhang, Tianyi and Wu, Xiaoxia and
                   Zhu, Rui-Jie and Wang, Junxiong and Zhou, Zhongzhu and Lai, Fan and Zou, James and Xu, Chenfeng},
  year          = {2026},
  eprint        = {2610.20687},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.20687}
}