Introduction

dInfer is an efficient and extensible inference framework for dLLMs. As illustrated in the following architecture, it modularizes inference into four components: model, diffusion iteration manager, decoder and KV-cache manager. It provides well-designed APIs for flexible algorithms combinations in each component. Now supports batched inference for improved throughput.

Figure: Overall Architecture of dInfer

dInfer supports multiple dLLM variants, including LLaDA and LLaDA-MoE.

Algorithmic improvements:

• Soft diffusion iteration for smoother denoising
• Hierarchical and credit decoding for enhanced parallel decoding
• Vicinity refresh strategy for KV-cache management to mitigate cache staleness

System-level optimizations:

• Tensor Parallelism (TP) and Expert Parallelism (EP) to maximize GPU utilization across batch sizes
• Dynamic batching support for improved throughput on multi-request workloads
• PyTorch compilation and NVIDIA CUDA Graphs for efficient kernel execution
• Loop unrolling mechanism to eliminate CUDA stream bubbles across diffusion iterations

Contents

• Supported Models
• Benchmark Results
• Getting Started

Supported Models

dInfer supports multiple diffusion language model variants with different architectures and sizes. Below are the HuggingFace model links and their corresponding implementation files:

LLaDA2.0

Implementation: modeling_llada2_moe.py

Features:

• Trained using Block Diffusion to improve throughput and stability
• Supports tool calling and complex agent-based task execution
• Excels at complex mathematical reasoning and code generation
• Supports both Expert Parallelism (EP) and Tensor Parallelism (TP)
• Decoding algorithms: Hierarchical, Credit, Threshold

LLaDA-MoE Models (Mixture-of-Experts)

Implementation: modeling_fused_olmoe.py

Features:

• Sparse Mixture-of-Experts with 64 experts
• FusedMoE optimization for efficient inference
• Support both Expert Parallelism (EP) and Tensor Parallelism (TP)
• Decoding algorithms: Hierarchical, Credit, Threshold

LLaDA Models (Dense)

Implementation: modeling_llada.py

Features:

• Dense transformer architecture
• Optimized for single-GPU and multi-GPU inference
• Decoding algorithms: Hierarchical, Credit, Threshold

Benchmark Results

Figure: Benchmark results

Performance on HumanEval:

• Over 1,100 TPS at batch size 1
• Averages 800+ TPS across six benchmarks on a single node with 8× H800 GPUs

Speedup comparisons:

• 10× faster than Fast-dLLM while maintaining accuracy
• 2-3× faster than Qwen2.5-3B on vLLM (LLaDA-MoE) with comparable quality

Getting Started

Please follow the instruction below to install dInfer.

git clone https://github.com/inclusionAI/dInfer.git
cd dInfer
pip install .

Download from HuggingFace and Convert to FusedMoE Format

This project supports using LLaDA and LLaDA-MoE checkpoints from HuggingFace. After downloading a model, run the conversion script to fuse MoE experts into FusedMoE format for local loading.

1) Download checkpoints

pip install -U huggingface_hub hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1

# Example: Instruct checkpoint
hf download inclusionAI/LLaDA-MoE-7B-A1B-Instruct \
  --repo-type model \
  --local-dir /path/to/LLaDA-MoE-7B-A1B-Instruct

2) Convert to FusedMoE format

(For LLada2.0-mini/flash, this step should be skipped.)

Use the conversion tool to fuse MoE experts.

# From repo root
python -m tools.transfer.py \
  --input  /path/to/LLaDA-MoE-7B-A1B-Instruct \
  --output /path/to/LLaDA-MoE-7B-A1B-Instruct-fused

After conversion, the output directory contains:

• modeling_fused_olmoe.py
• config.json with:
  • architectures: [FusedOlmoeForCausalLM]
  • auto_map.AutoModelForCausalLM: modeling_fused_olmoe.FusedOlmoeForCausalLM

3) Load the model

• Load via Auto classes:

from dinfer.model import AutoModelForCausalLM
from transformers import AutoTokenizer
m = "/path/to/LLaDA-MoE-7B-A1B-Instruct-fused"
tok = AutoTokenizer.from_pretrained(m, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(m, trust_remote_code=True, torch_dtype="bfloat16")

Tutorial

• Benchmark-only (speed) — scripts in benchmarks/

  • Measure throughput (TPS) only; predictions are saved under --output_dir; no automatic scoring.
  • Example 1 (LLaDA-MoE, threshold decoder, TP across 4 GPUs):

  python benchmarks/benchmark_dataset.py \
    --model_name inclusionAI/LLaDA-MoE-7B-A1B-Instruct \
    --model_type llada_moe \
    --dataset dataset_path \
    --gen_len 1024 \
    --block_length 64 \
    --gpu 0,1,2,3 \
    --output_dir runs/llada_moe_threshold \
    --use_tp \
    --parallel_decoding threshold \
    --threshold 0.8 \
    --cache dual \
    --prefix_look 16 \
    --after_look 16 \
    --warmup_times 4 \
    --cont_weight 0.3
  

  • Example 2 (threshold decoder, TP across 4 GPUs, LLaDA2-mini):
  • Currently, we only support up to 4-way parallelism for LLaDA2 since the number of heads is 4.

  python benchmarks/benchmark_dataset.py \
    --model_name inclusionAI/LLaDA2.0-mini-preview \
    --model_type llada2 \
    --dataset dataset_path \
    --gen_len 2048 \
    --block_length 32 \
    --gpu 0,1,2,3 \
    --output_dir runs/llada2_mini \
    --use_tp \
    --parallel_decoding threshold \
    --threshold 0.9 \
    --cache prefix \
    --use_bd
  

  • Example 3 (threshold decoder, TP across 4 GPUs, LLaDA2-flash):
  • Currently, we only support up to 4-way parallelism for LLaDA2 since the number of heads is 4.

    python benchmarks/benchmark_dataset.py \
      --model_name inclusionAI/LLaDA2.0-flash-preview \
      --model_type llada2 \
      --dataset dataset_path \
      --gen_len 2048 \
      --block_length 32 \
      --gpu 0,1,2,3 \
      --output_dir runs/llada2_flash \
      --use_tp \
      --parallel_decoding threshold \
      --threshold 0.9 \
      --cache prefix \
      --use_bd
  

  • Other entry points:
    • benchmark.py — Single-sample profiling.
    • Example 1 (threshold decoder, TP across 4 GPUs, LLaDA2-mini))

    python benchmarks/benchmark.py \
      --model_name inclusionAI/LLaDA2.0-mini-preview \
      --model_type llada2 \
      --gen_len 2048 \
      --block_length 32 \
      --gpu 0,1,2,3 \
      --use_tp \
      --parallel_decoding threshold \
      --threshold 0.9 \
      --cache prefix \
      --use_bd
    

• End-to-end evaluation (speed + accuracy) — scripts in evaluations/

  • Built on HuggingFace lm-eval-harness; computes both TPS and benchmark scores.
  • Tasks provided:
    • gsm8k_llada: math reasoning.
    • mbpp_sanitized_llada: sanitized Python code generation.
  • For more examples and comprehensive instructions, see our quickstart guide.
  • Currently, the evaluation configuration is only aligned with LLaDA-MoE; lm-eval evaluations for llada2-mini/flash and other models will be updated later.

Contact us

• Wechat Group

Citation

@article{dinfer,
    title={dInfer: An Efficient Inference Framework for Diffusion Language Models},
    author={Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, Xinyuan Zhang, Zhen Tao, Haibo Feng, Ziyun Jiang, Ying Xu, Zenan Huang, Yihong Zhuang, Haokai Xu, Jiaqi Hu, Zhenzhong Lan, Junbo Zhao, Jianguo Li, Da Zheng},
    year={2025},
    journal={arXiv preprint arXiv:2510.08666}
}