iLLaDA-8B-Instruct

iLLaDA is an 8B fully bidirectional masked diffusion language model trained from scratch with 12T pre-training tokens, an 8192-token context length, variable-length generation, and confidence-based scoring for multiple-choice evaluation.

For more details, please refer to the paper: Improved Large Language Diffusion Models.

Inference and evaluation codes can be found in the LLaDA GitHub Repository.

How to Use

You can load the model and tokenizer using the transformers library:

import torch
from transformers import AutoModel, AutoTokenizer

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained('GSAI-ML/iLLaDA-8B-Instruct', trust_remote_code=True)
model = AutoModel.from_pretrained('GSAI-ML/iLLaDA-8B-Instruct', trust_remote_code=True, torch_dtype=torch.bfloat16)

For customized generation and evaluation scripts (such as generate.py and chat.py), please visit the official GitHub repository.

Architecture

Benchmark Results of Instruct Models

Citation

@article{nie2025large,
  title={Large Language Diffusion Models},
  author={Nie, Shen and Zhu, Fengqi and You, Zebin and Zhang, Xiaolu and Ou, Jingyang and Hu, Jun and Zhou, Jun and Lin, Yankai and Wen, Ji-Rong and Li, Chongxuan},
  journal={arXiv preprint arXiv:2502.09992},
  year={2025}
}