🎯 Overview

We introduce DARE (dLLM Alignment and Reinforcement Executor), a flexible and efficient supervised-finetuning (SFT) and reinforcement learning (RL) training framework designed specifically for diffusion large language models (dLLMs). DARE also integrates dLLMs into a comprehensive evaluation platform. It aims to be both flexible and user-friendly to use with:

• Easy extension of diverse RL algorithms for dLLMs
• Easy extension of extra benchmark evaluations for dLLMs
• Easy integration of popular and upcoming dLLM infras and HuggingFace weights

DARE is a work in progress, we plan to support more models and algorithm for training and evaluation. We warmly welcome the research community to collaborations, give feedback and share suggestions. Let's advance the diffusion large language models together !!!👊

📢 News

• [2026-07-28]: Extend the SGLang PR branch to preserve complete terminal-block token and step-map replay after a mid-block stop, enabling exact CJ-GRPO actor updates while keeping normal decoded responses stop-truncated.
• [2026-07-18]: Support CJ-GRPO for SDAR and LLaDA2.0-mini; update the SGLang PR branch with rollout diversity and step-map trajectory replay support.
• [2026-07-16]: Support ESPO for LLaDA.
• [2026-07-15]: Support JustGRPO for LLaDA.
• [2026-06-12]: Fix DLLM sampling params for rollout diversity SGLang branch
• [2026-04-19]: Support ebpo for SDAR.
• [2026-04-18]: Add an example of multi nodes rl training for LLaDA d1.
• [2026-04-05]: We release our report "DARE: Diffusion Large Language Models Alignment and Reinforcement Executor" on arXiv.
• [2026-03-23]: Support d-treerpo algorithm for LLaDA and Dream.
• [2026-03-16]: Support bgpo and ebpo for LLaDA2.X.
• [2026-03-13]: Fix bugs in SDAR SGLang rollout and dp actor.
• [2026-03-12]: Support sp for SDAR family, LLaDA2.0 and 2.1.
• [2026-03-10]: Support evaluation of SDAR-30B-A3B-Chat with SGLang and lmdeploy.
• [2026-03-09]: Support evaluation of LLaDA2.1-mini with SGLang.
• [2026-03-03]: Support mdpo for LLaDA and Dream.
• [2026-03-02]: Support SGLang for SDAR rl rollout.
• [2026-02-28]: Fix bugs and update features for LLaDA and Dream sequence parallel.
• [2026-02-27]: Support evaluation of SDAR with SGLang.
• [2026-02-26]: Update LLaDA cj-grpo and add Dream cj-grpo.

2025

• [2025-12-28]: Fix bugs and update features in dp_actor_algorithm.
• [2025-12-24]: Support online rl (online weight update of rollout) for SDAR.
• [2025-12-23]: Support vrpo (preference optimization) for Dream.
• [2025-12-16]: Support vrpo (preference optimization) for LLaDA.
• [2025-12-12]: Support sft and peft of SDAR.
• [2025-12-11]: Support evaluation of LLaDAMoE and LLaDA2.0-mini with SGLang.
• [2025-12-08]: Support coupled-grpo, cj-grpo and spg algorithm.
• [2025-12-03]: Support sequence parallel to enable longer generation ability for dLLMs.
• [2025-12-01]: We initialize the codebase of DARE (dLLM Alignment and Reinforcement Executor), including training (sft, peft, and rl) and faster evaluation.

🔍 Catalogue

• 🎯 Overview
• 🏆 Key Features
• 🛠️ Installation and Setup
• 🏋️ Training
• 📊 Evaluation
• 📈 Performance
• 📦 Supported Models
• 🌱 Supported RL Algorithms
• 📧 Contact
• 📚 Citation
• 🙏 Acknowledgments

🏆 Key Features

• Algorithm Zoo
  • Support d1, Coupled-GRPO, VRPO, MDPO, CJ-GRPO, BGPO, SPG, EBPO, d-TreeRPO, JustGRPO, ESPO
• Model Diversity
  • Masked diffusion language models (e.g., LLaDA and Dream)
  • Block diffusion language model (e.g., SDAR and LLaDA2.X)
• Comprehensive Evaluation for dLLMs
  • Integrate faster dLLM evaluation in opencompass
• Upcoming Features
  • Support multi-modal, omni, more algorithms

🛠️ Installation and Setup

Our training framework is built on top of verl, providing a robust foundation for supervised finetuning and reinforcement learning experiments, and our evaluation framework is built on the top of opencompass, providing a comprehensive and fast evaluations.

[!NOTE] Due to some irreconcilable dependency conflicts between packages, we strongly recommend using two separate virtual environments, for training and evaluation, respectively.

🚀 Quick Installation

Clone the DARE repo:

git clone https://github.com/yjyddq/DARE

Build training vitual environment:

# Create and activate environment
conda create -n DARE python=3.10 -y
conda activate DARE

# Install dependencies
cd DARE
pip install -r requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation
# or (Recommend)
# install from whl
# wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
# pip install flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

# You can go to https://flashattn.dev/ to find the compiled FlashAttention .whl that corresponds to your Python, PyTorch, and CUDA versions.
# After download the .whl, install it
pip install flash_attn-2.8.x+cu12xtorch2.x-cp3xx-cp3xx-linux_x86_64.whl # For example

Build evaluation vitual environment:

# Create and activate environment
conda create --name opencompass python=3.10 -y
conda activate opencompass

# Install dependencies
cd DARE/opencompass
pip install -e .

# For HumanEval evaluation, install the additional dependency:
git clone https://github.com/open-compass/human-eval.git
cd human-eval && pip install -e .
cd ..

# For Math evaluation, pip install the additional dependency:
pip install math_verify latex2sympy2_extended

## Full installation (with support for more datasets)
# pip install "opencompass[full]"

## Environment with model acceleration frameworks
# pip install "opencompass[lmdeploy]"
# or
# pip install lmdeploy==0.10.1

Install SGLang (required for SDAR and LLaDA2.x rollout and evaluation acceleration) from source. We recommend using the SGLang PR branch before it is merged: it fixes DLLM sampling parameters for rollout diversity and exposes opt-in decoder step maps for CJ-GRPO trajectory replay.

# Recommended: rollout diversity plus optional decoder step maps
git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/27943/head:dllm-sampling-params
git checkout dllm-sampling-params

# Fallback: use the previous compatible release only without these PR features
# git clone -b v0.5.9 https://github.com/sgl-project/sglang.git
# cd sglang

# Install the python packages
pip install --upgrade pip
pip install -e "python"

🔧 Model Setup

After downloading LLaDA-8B-Instruct, replace the source files with our modified versions to enable several key features:

# Copy modified files to your LLaDA model directory
cp models/xxx/* <path_to_llada_model>/

Or you can move the model weights (.safetensors) to model/xxx/*

# Copy weights to models/xxx/ directory
cp <path_to_llada_model>/*.safetensors models/xxx/

Also for Dream, SDAR, etc.

[!NOTE] Since optimization plan in RL pipeline (various attention-computation backend), this step is indispensable.

🗄️ Dataset Setup

Preprocessed datasets is under data/preprocessed. Please refer verl.utils.preprocess to organize datasets.

🏋️ Training

🚀 SFT Quick Start

bash scripts/run_sft.sh # | scripts/run_sft_peft.sh

Alternatively, use/write scripts in recipe/xxx/run_xxx.sh

# peft for llada_8b_instruct
bash recipe/run_sft_peft_llada_8b_instruct.sh 

# sft for dream_7b_instruct
bash recipe/run_sft_dream_7b_instruct.sh 

# peft for sdar_8b_chat
bash recipe/run_sft_peft_sdar_8b_chat.sh 

🚀 RL Quick Start

# online rl for llada_8b_instruct
bash recipe/run_d1_llada_8b_instruct.sh --task math # use Fast-dLLM for rollout acceleration

# online rl for dream_7b_instruct
bash recipe/run_coupled_grpo_dream_7b_instruct.sh --task math # use Fast-dLLM for rollout acceleration

# online rl for sdar_8b_chat
bash recipe/run_bgpo_sdar_8b_chat.sh --task math # use lmdeploy engine for rollout acceleration

# d-TreeRPO for llada_8b_instruct
bash recipe/llada/run_dtreerpo_llada_8b_instruct.sh --task math

# d-TreeRPO for dream_7b_instruct
bash recipe/dream/run_dtreerpo_dream_7b_instruct.sh --task math

🚀 DPO/VRPO Quick Start

Run an example for preference optimization. First download argilla/ultrafeedback-binarized-preferences-cleaned, then run scripts/preprocess_dpo_dataset.sh to save ultrafeedback.parquet under data/preprocessed/dpo/train and data/preprocessed/dpo/test

# preference optimization for llada_8b_instruct
bash recipe/run_vrpo_llada_8b_instruct.sh --task ultrafeedback

# preference optimization for dream_7b_instruct
bash recipe/run_vrpo_dream_7b_instruct.sh --task ultrafeedback

🚀 Convert Checkpoints Weights to HuggingFace Safetensors

# convert FSDP sharded checkpoints to HuggingFace safetensors format
bash scripts/convert_ckpt_to_hf.sh \
    --model_path models/LLaDA-8B-Instruct \
    --ckpt_path ./ckpts/<project>/<exp_name>/global_step_xxx/actor \
    --output_path ./converted_models/llada_8b_stepxxx \
    --model_name llada \
    --fsdp_strategy fsdp2 \
    --n_gpus 8

📊 Evaluation

🚀 Eval on OpenCompass Quick Start

First, please follow opencompass for benchmark dataset preparation. Then, you need to specify the model path in opencompass/opencompass/configs/models/dllm/*. For example llada_instruct_8b.py:

from opencompass.models import LLaDAModel

models = [
    dict(
        type=LLaDAModel,
        abbr='llada-8b-instruct',
        path='/TO/YOUR/PATH', # Need to modify
        max_out_len=1024,
        batch_size=1,
        run_cfg=dict(num_gpus=1),
    )
]

Evaluation of LLaDA-8B-Instruct on mmlu with hf backend:

bash scripts/eval_llada.sh --task mmlu

Evaluation of SDAR-8B-Chat on mmlu with lmdeploy backend:

bash scripts/eval_sdar_8b_chat.sh --task mmlu --engine lmdeploy

🚀 Eval on Local Bench Quick Start

If you want to add more benchmarks, models, or custom datasets, please refer to the Evaluation Guideline.

📦 Supported Models

🌱 Supported RL Algorithms

📈 Performance

Evaluation Result Reproduction

Algorithm Comparison (LLaDA-8B-Instruct)

Algorithm Comparison (Dream-7B-Instruct)

📧 Contact

For any questions or collaboration inquiries, feel free to reach out Jingyi Yang at: yangjingyi946@gmail.com.

👷‍♂️ Contributor

Waiting for your joining and contribution.

📚 Citation

If you find our work useful, please consider citing:

@article{yang2026dare,
  title={DARE: Diffusion Large Language Models Alignment and Reinforcement Executor},
  author={Yang, Jingyi and Jiang, Yuxian and Hu, Xuhao and Cheng, Shuang and Qi, Biqing and Shao, Jing},
  journal={arXiv preprint arXiv:2604.04215},
  year={2026}
}

@article{yang2025taming,
  title={Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step},
  author={Yang, Jingyi and Chen, Guanxu and Hu, Xuhao and Shao, Jing},
  journal={arXiv preprint arXiv:2509.23924},
  year={2025}
}

🙏 Acknowledgments

We thank the open-source community for their wonderful work and valuable contributions:

• Models Source: GSAI-ML, inclusionAI, Dream-org, JetLM, huggingface for model supply or hosting
• Algorithm: d1, CJ-GRPO, MDPO, Coupled-GRPO, SPG, BGPO, VRPO, EBPO, d-TreeRPO
• Training Framework: verl, BGPO, Dream, dFactory
• Inference Acceleration/Engine: Fast-dLLM, lmdeploy, SGLang
• Evaluation Framework: opencompass, LLaDA
• Environment Dependencies: verl, opencompass, flash-attn