LLaDA2.1-mini-ORL-NVFP4

LLaDA-2.1-mini + ORL, NVFP4: LLaDA2.1-mini-ORL with its routed experts in NVFP4 (W4A4) and an FP8 KV cache, for NVIDIA Blackwell GPUs

Highlights

• 5,046 tokens/s for a single request on HumanEval on one B200, 2.3–2.4× the BF16 checkpoint on one H100 on every benchmark
• Fast on a consumer GPU: 2,788 tokens/s on HumanEval on one 32 GB GeForce RTX 5090, faster than BF16 on an H100 on every benchmark
• 10.6 GB instead of 32.6 GB at comparable accuracy (NVIDIA Model Optimizer, local_hessian calibration)

Results

Accuracy (%) / TPF with the ORL decoder (threshold 0.85, edit threshold 0.5, 4096 new tokens) and the paper's scorers:

On the same B200 with native FP4 experts, against the BF16 checkpoint:

Weights only, in the paper's SGLang pipeline on H100 (NVFP4 expert weights dequantized to BF16; BF16 activations and KV cache):

Accuracy stays within run-to-run noise: thresholded parallel decoding turns any numeric difference into a different trajectory, one HumanEval task is 0.6 points, and BF16 itself moves by 5 HumanEval points between the two engines. FP4 rounding costs 2-12% of the tokens per forward.

Speed

Tokens per second for a single request with dlmrt (the paper's NVFP4 tables):

A denoising round takes 2.0 ms on one B200, against 2.8 ms in BF16. With 8-way tensor parallelism a rank holds 32 experts and a round is latency-bound: 1.69 ms against 1.78 ms in BF16, and 6,036 against 5,844 tokens/s on full HumanEval.

Quantization

Weight block scales come from a search over FP8 scales that minimizes each block's error weighted by a local 16 × 16 Hessian of that expert's calibration inputs; gate and up share their global scale, and the experts of a layer share their input scale. Calibration uses 128 rows of 16,256 chat-templated tokens from nvidia/Nemotron-Post-Training-Dataset-v2 (chat, math, code, STEM, multilingual), each continued by 256 tokens that the model decodes with its own block decoder (threshold 0.85, edit threshold 0.5, 16 refine steps, self-conditioning on), so the experts are calibrated on the denoising canvas they see at inference. quant_provenance.json records the recipe and per-layer routing coverage (no expert left unrouted).

Usage

The checkpoint is a ModelOpt Hugging Face export (hf_quant_config.json, quantization_config in config.json; per expert weight, weight_scale, weight_scale_2 and input_scale) and keeps the custom model code, tokenizer, chat template and paper_decoding.json of LLaDA2.1-mini-ORL. Plain transformers generation supports neither the FP4 experts nor the paper's decoder. Two engines run it: dlmrt with native FP4 experts on Blackwell, and the paper's SGLang pipeline of the ORL repository with the expert weights dequantized to BF16 (the weights-only table above).

dlmrt

dlmrt, our inference engine, decodes one request at a time with the paper's decoder (role="orl") and measures the paper's throughput; see its README for installation, tensor parallelism and the commands of every paper cell.

from dlmrt import LLM

llm = LLM("yifanyu/LLaDA2.1-mini-ORL-NVFP4", role="orl", precision="nvfp4")
out = llm.generate("Write a Python function that reverses a linked list.", max_new_tokens=4096)
print(out.text)
print(f"{out.request_tps:.0f} tokens/s")

python -m dlmrt serve --model yifanyu/LLaDA2.1-mini-ORL-NVFP4 --role orl --precision nvfp4 --port 18080
curl -N localhost:18080/generate -d '{"messages": [{"role": "user", "content": "Hi!"}], "max_new_tokens": 512}'

On GPUs without FP4 tensor cores, precision="bf16" runs the dequantized experts in BF16. The paper's runs, with its problem files:

hf download yifanyu/LLaDA2.1-mini-ORL-NVFP4 --local-dir LLaDA2.1-mini-ORL-NVFP4
python -m dlmrt bench suite --model LLaDA2.1-mini-ORL-NVFP4 --role orl --precision nvfp4 --threshold 0.85 \
    --edit-threshold 0.5 --max-new-tokens 4096 --problems gsm8k.jsonl math500.jsonl humaneval.jsonl mbpp.jsonl --out DIR

Related Models

Citation

@misc{yu2026orl,
  title         = {On-Policy Residual Learning for Diffusion Language Models},
  author        = {Yu, Yifan and Jian, Yuqing and Li, Jisen and Hu, Yuezhou and Zhang, Tianyi and Wu, Xiaoxia and
                   Zhu, Rui-Jie and Wang, Junxiong and Zhou, Zhongzhu and Lai, Fan and Zou, James and Xu, Chenfeng},
  year          = {2026},
  eprint        = {2610.20687},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.20687}
}