d3LLM Trajectory Dataset

Project Page | Paper | GitHub | Blog

This repository contains the pseudo-trajectory distillation data used for training d3LLM (pseuDo-Distilled Diffusion Large Language Model), as introduced in the paper "d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation".

Introduction

d3LLM is a framework designed to strike a balance between accuracy and parallelism in diffusion-based large language models (dLLMs). This dataset consists of pseudo-trajectories extracted from teacher models (such as Dream or LLaDA).

During training, these trajectories are used in Pseudo-Trajectory Distillation to teach the student model which tokens can be decoded confidently at early steps. This process guides the model to learn efficient generation patterns, significantly improving generation speed and parallelism with negligible accuracy loss. The distillation data primarily focuses on tasks involving coding and mathematical reasoning.

Citation

If you find this dataset or the d3LLM framework useful for your research, please cite:

@inproceedings{ICML'26:d3llm,
    title = {d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation},
    author = {Yu-Yang Qian and Junda Su and Lanxiang Hu and Peiyuan Zhang and Zhijie Deng and Peng Zhao and Hao Zhang},
    booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
    pages = {to appear},
    year = {2026}
}