In end-to-end autonomous driving systems, two main approaches have emerged:

• Vision-Action (VA) models take visual information (LiDAR point clouds, images) as inputs and output action signals through a modular framework. While these methods achieve promising results through accurate spatial perception, they are limited in language-based scene interaction.
• Vision-Language-Action (VLA) approaches leverage the reasoning capabilities of multimodal large language models (MLLMs). These methods achieve superior interaction capabilities but often struggle due to the absence of fine-grained intermediate 3D perception and prediction.

DrivePI bridges this gap by combining the strengths of both approaches, serving as a unified Vision-Language-Action framework that is also compatible with vision-action models. Our method jointly performs spatial understanding, 3D perception (i.e., 3D occupancy), prediction (i.e., occupancy flow), and planning (i.e., action outputs) in parallel through end-to-end optimization. To obtain both precise geometric information and rich visual appearance, our approach integrates point clouds, multi-view images, and language instructions within a unified MLLM architecture.

📊 Data Engine

Our multi-stage data pipeline consists of:

1. Caption Annotation: We use InternVL3-78B to generate captions of front and back views separately, then merge and polish them to create comprehensive scene descriptions.
2. 4D Spatial Understanding Annotation: We leverage ground-truth occupancy and flow data to generate diverse text-occupancy and text-flow QA pairs through multi-turn conversations, enabling fine-grained 3D understanding.
3. Planning Reasoning Annotation: We create planning QA pairs based on future trajectory annotations to enhance planning interpretability, enabling the MLLM to predict future actions of the ego-vehicle.

📈 Results

Remarkably, with only a 0.5B Qwen2.5 model as MLLM backbone, DrivePI as a single unified model matches or exceeds both existing VLA models and specialized VA models:

• Compared to VLA models, DrivePI outperforms OpenDriveVLA-7B by 2.5% mean accuracy on nuScenes-QA and reduces collision rate by 70% over ORION (from 0.37% to 0.11%) on nuScenes.
• Against specialized VA models, DrivePI surpasses FB-OCC by 10.3 RayIoU for 3D occupancy on OpenOcc, reduces the mAVE from 0.591 to 0.509 for occupancy flow on OpenOcc, and achieves 32% lower L2 error than VAD (from 0.72m to 0.49m) for planning on nuScenes.

3D Occupancy and Occupancy Flow on OpenOcc

3D Occupancy on Occ3D-nuScenes

*DrivePI trained exclusively on the 3D occupancy task of Occ3D-nuScenes.

Planning on nuScenes

Text Understanding on nuScenes-QA

Component Ablation Study

🛠️ Installation and Running

Environment Setup

# Create conda environment
conda create -n drivepi python==3.10.18

# Install requirements
pip install -r requirements.txt

# Install EMOVA
# Reference: https://github.com/emova-ollm/EMOVA
pip install -e .
pip install flash-attn --no-build-isolation

Data Preparation

1. BEV features generation:

   • Use UniLION to generate and save BEV features
   • Save path: /path/DrivePI_Data/unilion_bev_feats_train/
   • Name the features using token names

2. QA datasets:

   • Save at: /path/DrivePI_Data/drivepi_captions/

Training

bash run_train_occ_action_llm_8_gpus_final.sh

Testing

# For occupancy and action testing
bash run_test_occ_llm_occ_action.sh

# For text understanding testing
bash run_test_occ_llm__text.sh

📝 Citation

@article{liu2025drivepi,
  title={DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning},
  author={Liu, Zhe and Huang, Runhui and Yang, Rui and Yan, Siming and Wang, Zining and Hou, Lu and Lin, Di and Bai, Xiang and Zhao, Hengshuang},
  journal={CVPR},
  year={2026}
}

Acknowledgements

We thank these great works and open-source repositories: UniLION, MMDectection3D, InternVL3, LLaVA, and EMOVA.