LMDrive Model Card

Model details

Model type: LMDrive is an end-to-end, closed-loop, language-based autonomous driving framework, which interacts with the dynamic environment via multi-modal multi-view sensor data and natural language instructions. The model weight is for the vision encoder used in the LMDrive.

Model date: LMDrive-1.0 was trained in November 2023.

Paper or resources for more information:

Github: https://github.com/opendilab/LMDrive/README.md

Paper: https://arxiv.org/abs/2312.07488

Where to send questions or comments about the model:

https://github.com/opendilab/LMDrive/issues

Intended use

Primary intended uses:

The primary use of LMDrive is research on large multimodal models for autonomous driving.

Primary intended users:

The primary intended users of the model are researchers and hobbyists in computer vision, large multimodal model, autonomous driving, and artificial intelligence.

Training dataset

• 64K instruction-sensor-control data clips collected in the CARLA simulator. dataset_webpage
  • where each clip includes one navigation instruction, several notice instructions, a sequence of multi-modal multi-view sensor data, and control signals. The duration of the clip spans from 2 to 20 seconds

Evaluation benchmark

LangAuto, LangAuto-short, LangAuto-tiny, LangAuto-notice