iLLaDA-8B-Base

iLLaDA is an 8B fully bidirectional masked diffusion language model trained from scratch with 12T pre-training tokens, an 8192-token context length, variable-length generation, and confidence-based scoring for multiple-choice evaluation. It was introduced in the paper Improved Large Language Diffusion Models.

Inference and evaluation codes: https://github.com/ML-GSAI/LLaDA.

Architecture

Benchmark Results of Base Models

How to use

You can load and use the model with transformers as follows:

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('GSAI-ML/iLLaDA-8B-Base', trust_remote_code=True)
model = AutoModel.from_pretrained('GSAI-ML/iLLaDA-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16)

Refer to the GitHub repository for generation scripts such as generate.py.

Citation

@article{nie2026illada,
  title={Improved Large Language Diffusion Models},
  author={Nie, Shen and Min, Qiyang and Xu, Shaoxuan and Huang, Zihao and Song, Yuxuan and Shan, Yong and Lin, Yankai and Zhao, Wayne Xin and Li, Chongxuan and Wen, Ji-Rong},
  journal={arXiv preprint arXiv:2606.25331},
  year={2026}
}