English | 简体中文

LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences

Ao Liang     Youquan Liu     Yu Yang     Dongyue Lu     Linfeng Li
Lingdong Kong     Huaici Zhao     Wei Tsang Ooi

     

In this work, we introduce LiDARCrafter, a unified framework for 4D LiDAR generation and editing. We contribute:

• The first 4D generative world model dedicated to LiDAR data, with superior controllability and spatiotemporal consistency.
• We introduce a tri-branch 4D layout conditioned pipeline that turns language into an editable 4D layout and uses it to guide temporally stable LiDAR synthesis.
• We propose a comprehensive evaluation suite for LiDAR sequence generation, encompassing scene-level, object-level, and sequence-level metrics.
• We demonstrate best single-frame and sequence-level LiDAR point cloud generation performance on nuScenes, with improved foreground quality over existing methods.

📚 Citation

If you find this work helpful for your research, please kindly consider citing our paper:

@inproceedings{liang2026lidarcrafter,
    title     = {{LiDARCrafter}: Dynamic {4D} World Modeling from {LiDAR} Sequences},
    author    = {Ao Liang and Youquan Liu and Yu Yang and Dongyue Lu and Linfeng Li and Lingdong Kong and Huaici Zhao and Wei Tsang Ooi},
    booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
    volume    = {40},
    year      = {2026},
}

Updates

• [11.2025] - LiDARCrafter has been accepted to AAAI 2026 for Oral Presentation. 🎉
• [10.2025] - We will soon start organizing the code. All pretrained weights for evaluation can be found at Hugging Face.
• [08.2025] - The technical report of LiDARCrafter is available on arXiv.

Outline

• Updates
• Outline
• ⚙️ Installation
• ♨️ Data Preparation
• 🚀 Getting Started
  • Evaluation
• 🔧 Generation Framework
  • Overall Framework
  • 4D Layout Generation
  • Single-Frame Generation
• 🐍 Model Zoo
• 📝 TODO List
• License
• Acknowledgements

⚙️ Installation

Please configure your environment according to the version information in environment.yml.

♨️ Data Preparation

• Create dataset: same as DrivingDiffusion

ln -s ${ROOT_DATA_PATH} ./data/nuscenes

Run bash scripts/create_data.sh for generate:

• info with track and state

• Updated pkl with scene graph

• CLIP feature of scene graph

The file-tree of data is like:

data
├── clips
│   └── nuscenes
│       ├── obj_text_feat.pkl
│       ├── train
│       └── val
├── infos
│   ├── needed_5_framed_token.pkl
│   ├── nuscenes_dbinfos_10sweeps_withvelo.pkl
│   ├── nuscenes_infos_10sweeps_train.pkl
│   ├── nuscenes_infos_10sweeps_val.pkl
│   ├── nuscenes_infos_lidargen_train.pkl
│   ├── nuscenes_infos_lidargen_val.pkl
│   ├── nuscenes_infos_train.pkl
│   ├── nuscenes_infos_val.pkl
│   ├── nuscenes_object_classification_train.pkl
│   └── nuscenes_object_classification_val.pkl
└── nuscenes

🚀 Getting Started

Evaluation

• Train classification model
  • python train/train_classification_pointmlp.py
• Train uncertainty model
  • python train/train_uncertainty_glenet.py

For each generated 1w model

• Extract foreground samples
  • python evaluation/extract_foreground_samples.py --model ori

🔧 Generation Framework

Overall Framework

4D Layout Generation

Single-Frame Generation

🐍 Model Zoo

To be updated.

📝 TODO List

• Initial release. 🚀
• Release the training code.
• Release the inference code.
• Release the evaluation code.
• Refine the Readme.md

License

This work is under the Apache License Version 2.0, while some specific implementations in this codebase might be under other licenses. Kindly refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.

Acknowledgements

This work is developed based on the MMDetection3D codebase.

MMDetection3D is an open-source toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

Part of the benchmarked models are from the OpenPCDet and 3DTrans projects.

Related Projects