DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes

Hengwei Bian^1,2,*    Lingdong Kong^1,3    Haozhe Xie⁴    Liang Pan^1,†,‡ Yu Qiao¹    Ziwei Liu^4
1Shanghai AI Laboratory    ²Carnegie Mellon University   
³National University of Singapore    ⁴S-Lab, Nanyang Technological University
^*Work done during an internship at Shanghai AI Laboratory    ^†Corresponding author    ^‡Project lead

Overview

Our DynamicCity framework consists of two key procedures: (a) Encoding HexPlane with an VAE architecture, and (b) 4D Scene Generation with HexPlane DiT.

Updates

• [February 2025]: Code released.
• [October 2024]: Project page released.

Outline

• Installation
• Data Preparation
• Getting Started
• Dynamic Scene Generation

⚙️ Installation

conda create -n dyncity python=3.10 -y
conda activate dyncity
conda install pytorch==2.4.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install einops hydra-core matplotlib numpy omegaconf timm tqdm wandb -c conda-forge -y
pip install flash-attn --no-build-isolation

♨️ Data Preparation

Download the CarlaSC dataset from here and extract it into the ./carlasc directory. Your repository should look like this:

DynamicCity
├── carlasc/
│   ├── Cartesian/
│   │   ├── Train/
│   │   │   ├── Town01_Heavy
│   │   │   ├── ...
│   │   ├── Test/
├── ...

🚀 Getting Started

You can obtain our checkpoints from here. To use the pretrained models, simply unzip ckpts.zip as ./ckpts and run infer_dit.py.

To train VAE on CarlaSC dataset, run the following command:

torchrun --nproc-per-node 8 train.py VAE carlasc name=DYNAMIC_CITY_VAE

After VAE is trained, save hexplane rollouts using:

torchrun --nproc-per-node 8 infer_vae.py -n DYNAMIC_CITY_VAE --save_rollout --best

Then, you can train your DiT using this command:

torchrun --nproc-per-node 8 train.py DiT carlasc name=DYNAMIC_CITY_DIT vae_name=DYNAMIC_CITY_VAE

Finally, use DiT to sample novel city scenes:

torchrun --nproc-per-node 8 infer_dit.py -d DYNAMIC_CITY_DIT --best_vae

🏙️ Dynamic Scene Generation

Unconditional Generation

HexPlane Conditional Generation

Command & Trajectory-Driven Generation

Layout-Conditioned Generation

Dynamic Scene Inpainting

Citation

If you find this work helpful for your research, please kindly consider citing our papers:

@inproceedings{bian2025dynamiccity,
  title={DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes},
  author={Bian, Hengwei and Kong, Lingdong and Xie, Haozhe and Pan, Liang and Qiao, Yu and Liu, Ziwei},
  booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},
  year={2025},
}