Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis :car:

This repository will collect research, implementations, and resources related to Foundation Models for Scenario Generation and Analysis in autonomous driving. The repository will be maintained by TUM-AVS (Professorship of Autonomous Vehicle Systems at Technical University of Munich) and will be continuously updated to track the latest work in the community.

:fire: Updates

• Aug. 2026 – Added 9 new papers on scenario generation* and 11 new papers on scenario analysis*.
• Jul. 2026 – Added 4 new papers on scenario generation* and 14 new papers on scenario analysis*.
• Jun. 2026 – Added 33 new papers on scenario analysis* (monthly backlog catch-up spanning Dec 2023 – Jun 2026).
• May 2026 – Added 7 new papers on scenario generation* and 19 new papers on scenario analysis*.
• Apr. 2026 – Added 5 new papers on scenario generation* and 13 new papers on scenario analysis*.
• Mar. 2026 – Added 10 new papers on scenario generation* and 11 new papers on scenario analysis*.
• Feb. 2026 – Added 4 new papers on scenario generation* and 19 new papers on scenario analysis*.
• Jan. 2026 – 🎉 Our survey paper is accepted by IEEE Open Journal of Intelligent Transportation Systems (OJ-ITS) and uploaded a new version to arXiv.
• Dec. 2025 – Added 3 new papers on scenario generation* and 14 new papers on scenario analysis*. Added new columns: Hardware and Citation.
• Nov. 2025 – Added 2 new papers on scenario analysis*. Added new section: Useful Resources and Links.
• Uploaded new version to arXiv. Repository now categorizes 348 papers:
  • 93 on scenario generation
  • 56 on scenario analysis
  • 58 on datasets
  • 21 on simulators
  • 25 on benchmark challenges
  • 95 on other related topics (e.g., FMs' implementation)
• Oct. 2025 – Added 17 new papers on scenario generation* and 2 new papers on scenario analysis*.
• Sep. 2025 – Added 3 new papers on scenario generation* and 14 new papers on scenario analysis*.
• Aug. 2025 – Added 4 new papers on scenario generation* and 4 new papers on scenario analysis*.
• Jul. 2025 – Added 9 new papers on scenario generation* and 8 new papers on scenario analysis*.
• Jun. 2025 – Released our paper on arXiv. Repository now categorizes 342 papers:
  • 93 on scenario generation
  • 54 on scenario analysis
  • 55 on datasets
  • 21 on simulators
  • 25 on benchmark challenges
  • 94 on other related topics
• May 2025 – Repository initialized.

🤝   Citation

Please visit Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis for more details and comprehensive information. If you find our paper and repo helpful, please consider citing it as follows:

@ARTICLE{11370877,
  author={Gao, Yuan and Piccinini, Mattia and Zhang, Yuchen and Wang, Dingrui and Moller, Korbinian and Brusnicki, Roberto and Zarrouki, Baha and Gambi, Alessio and Totz, Jan Frederik and Storms, Kai and Peters, Steven and Stocco, Andrea and Alrifaee, Bassam and Pavone, Marco and Betz, Johannes},
  journal={IEEE Open Journal of Intelligent Transportation Systems}, 
  title={Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis}, 
  year={2026},
  volume={},
  number={},
  pages={1-1},
  keywords={Autonomous vehicles;Surveys;Foundation models;Scenario generation;Frequency modulation;Visualization;Reviews;Diffusion models;Adaptation models;Cognition;Autonomous vehicles;foundation model;scenario generation;scenario analysis;scenario based testing},
  doi={10.1109/OJITS.2026.3660686}}

:page_with_curl: Introduction

Foundation models are large-scale, pre-trained models that can be adapted to a wide range of downstream tasks. In the context of autonomous driving, foundation models offer a powerful approach to scenario generation and analysis, enabling more comprehensive and realistic testing, validation, and verification of autonomous driving systems. This repository aims to collect and organize research, tools, and resources in this important field.

:chart_with_upwards_trend: Publication Timeline

The following figure shows the evolution of foundation model research in autonomous driving scenario generation and analysis over time:

:mag: Search Methodology

The following list of keywords was used to search this survey's papers in the Google Scholar database. The keywords were entered either individually or in combination with other keywords in the list. The search was conducted until May 2025.

Keywords:

• Foundation Model Types: Foundation Models, Large Language Models (LLMs), Vision-Language Models (VLMs), Multimodal Large Language Models (MLLMs), Diffusion Models (DMs), World Models (WMs), Generative Models (GMs)
• Scenario Generation & Analysis: Scenario Generation, Scenario Simulation, Traffic Simulation, Scenario Testing, Scenario Understanding, Driving Scene Generation, Scene Reasoning, Risk Assessment, Safety-Critical Scenarios, Accident Prediction
• Application Context: Autonomous Driving, Self-Driving Vehicles, AV Simulation, Driving Video Generation, Traffic Datasets, Closed-Loop Simulation, Safety Assurance

🕒 Timeline of the Development of Foundation Models

The figure illustrates the evolution of foundation models. LLMs (e.g., BERT, GPT) appear at the bottom, followed by VLMs built on visual FMs (e.g., ViT, CLIP) and instruction-tuned VLMs for interactive vision–language reasoning. MLLMs are shown at the top. In parallel, the progression of visual FMs is traced through Diffusion Models (DMs) and World Models (WMs). Highlighted entries mark key conceptual milestones.

🌟 Large Language Models for Autonomous Driving

Scenario Generation (LLM)

Paper	Date	Venue	Code	Hardware	Citation
TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles	2023-05	IEEE Transactions on Software Engineering	-	API&RTX4090	24
Language Conditioned Traffic Generation	2023-07	CoRL 2023	GitHub	API	98
A Generative AI-driven Application: Use of Large Language Models for Traffic Scenario Generation	2023-11	ELECO 2023	-	API	15
ChatGPT-Based Scenario Engineer: A New Framework on Scenario Generation for Trajectory Prediction	2024-02	IEEE Transactions on Intelligent Vehicles	-	-	48
Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation	2024-04	arXiv	GitHub	A6000	24
LLMScenario: Large Language Model Driven Scenario Generation	2024-05	IEEE Transactions on Systems, Man, and Cybernetics: Systems	-	API	69
Automatic Generation Method for Autonomous Driving Simulation Scenarios Based on Large Language Model	2024-05	AIAT 2024	-	API	2
ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles	2024-05	CVPR 2024	GitHub	API	113
Editable scene simulation for autonomous driving via collaborative llm-agents	2024-06	CVPR 2024	GitHub	API	130
Chat2Scenario: Scenario Extraction From Dataset Through Utilization of Large Language Model	2024-06	IV 2024	GitHub	-	15
SoVAR: Building Generalizable Scenarios from Accident Reports for Autonomous Driving Testing	2024-09	ASE 2024	-	RTX4070	29
LeGEND: A Top-Down Approach to Scenario Generation of Autonomous Driving Systems Assisted by Large Language Models	2024-09	ASE 2024	GitHub	RTX3090	32
Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model	2024-09	arXiv	GitHub	API	18
Promptable Closed-loop Traffic Simulation	2024-09	CoRL 2024	GitHub	A100	16
Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles	2024-09	Automotive Innovation	-	-	26
LLM-Driven Testing for Autonomous Driving Scenarios	2024-11	FLLM 2024	-	API& T4	15
ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObility	2024-11	IEEE Transactions on Intelligent Vehicles	-	RTX4090	54
Generating Out-Of-Distribution Scenarios Using Language Models	2024-11	arXiv	-	API	13
Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner	2024-12	AAAI 2025 Oral	GitHub	RTX 3090	3
LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Models	2025-01	TITS 2025	-	RTX 4000	25
ML-SceGen: A Multi-level Scenario Generation Framework	2025-01	arXiv	-	RTX 4090	0
From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios	2025-02	ITSC 2025	GitHub	API	3
CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Models	2025-02	arXiv	GitHub	-	10
Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test	2025-03	arXiv	GitHub	-	14
Enhancing Autonomous Driving Safety with Collision Scenario Integration	2025-03	arXiv	-	8xV100	6
Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Models	2025-05	ITSC 2025	-	API	5
From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving	2025-05	arXiv	-	RTX4090	0
AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Framework	2025-07	arXiv	-	API	1
LLM-based Realistic Safety-Critical Driving Video Generation	2025-07	arXiv	-	RTX4090	1
Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles	2025-08	arXiv	GitHub	2xRTX4090	0
LLM-based Human-like Traffic Simulation for Self-driving Tests	2025-08	arXiv	-	API& A800	0
Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehicles	2025-09	GeCoIn 2025	-	A6000	3
Txt2Sce: Scenario Generation for Autonomous Driving System Testing Based on Textual Reports	2025-09	arXiv	-	RTX3070Ti	1
LLM‑Based Semantic Modeling & Cooperative Evolutionary Fuzzing	2025-09	APSEC 2025	-	-	0
LinguaSim: Interactive Multi-Vehicle Testing Scenario Generation via Natural Language Instruction Based on Large Language Models	2025-10	IEEE ITSC 2025	-	-	0
Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge	2025-11	arXiv	GitHub	-	0
AnchorDrive: LLM Scenario Rollout with Anchor-Guided Diffusion Regeneration for Safety-Critical Scenario Generation	2026-03	arXiv	-	-	0
TRACE: Topology-aware Reconstruction of Accidents in CARLA for AV Evaluation	2026-04	FSE 2026	GitHub	API	0
Traffic Scenario Orchestration from Language via Constraint Satisfaction	2026-05	arXiv	-	-	0
PCASim: Promptable Closed-loop Adversarial Simulation for Urban Traffic Environment	2026-05	arXiv	Project	-	0
TrafficAlign: Aligning Large Language Models for Traffic Scenario Generation	2026-06	CVPR 2026	GitHub	-	0
REARL: A Closed-loop Autonomous Driving Simulation Enhancement Framework with Real Traffic Data and Large Language Models	2026-09	arXiv	-	-	0
LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality	2026-09	arXiv	-	-	0
SimSkill: A Self-Evolving LLM Agent for Skill and Knowledge Accumulation in Traffic Simulation	2026-09	arXiv	GitHub	M1 Max & API	0
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving	2026-09	EMNLP 2026	GitHub	RTX5090 & API	0

Scenario Analysis (LLM)

Paper	Date	Venue	Code	Hardware	Citation
Semantic Anomaly Detection with Large Language Models	2023-09	Autonomous Robots	-	-	131
LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs	2023-12	CVPR 2024	GitHub	-	0
Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving	2024-02	arXiv	GitHub	-	0
Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models	2024-03	IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (Forge)	GitHub	API	22
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving	2024-05	ICRA 2024	GitHub	API	340
Bench4Merge: A Comprehensive Benchmark for Merging in Realistic Dense Traffic with Micro-Interactive Vehicles	2024-10	arXiv	GitHub	-	0
SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving	2025-01	2025 WACVW	-	API	9
From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios	2025-02	ITSC 2025	GitHub	API	3
A Comprehensive LLM-powered Framework for Driving Intelligence Evaluation	2025-03	ICRA 2025	GitHub	API	7
Understanding Driving Risks using Large Language Models: Toward Elderly Driver Assessment	2025-07	arXiv	-	-	0
Collision risk prediction and takeover requirements assessment based on radar-video integrated sensors data: A system framework based on LLM	2025-08	Accident Analysis & Prevention	-	API&RTX4090	7
Evaluation of Large Language Models for Anomaly Detection in Autonomous Vehicles	2025-09	arXiv	-	-	0
AgentDrive: An open benchmark suite for agentic AI reasoning in autonomous systems	2026-01	arXiv	GitHub	-	0
LLM-MLFFN: Multi-Level Autonomous Driving Behavior Feature Fusion via Large Language Model	2025-03	arXiv	-	-	0
Towards Lawful Autonomous Driving: Deriving Scenario-Aware Driving Requirements from Traffic Laws and Regulations	2026-04	arXiv	-	-	0
SwarmDrive: Semantic V2V Coordination for Latency-Constrained Cooperative Autonomous Driving	2026-04	arXiv	-	-	0
Pedestrian-Aware LLM-Driven Behavioral Planning for Autonomous Vehicles	2026-05	IEEE ITSC 2026	-	-	0
AutoMine Solution for AV2 2026 Scenario Mining Challenge	2026-06	arXiv	-	-	0
A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving	2026-07	arXiv	GitHub	-	0
SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment	2026-09	arXiv	GitHub	API	0

🌟 Vision-Language Models for Autonomous Driving

Scenario Generation (VLM)

Paper	Date	Venue	Code	Hardware	Citation
WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models	2023-05	CVPR workshop 2023	GitHub	API	63
DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving	2024-08	IAVVC 2024	-	L4	13
From Dashcam Videos to Driving Simulations: Stress Testing Automated Vehicles against Rare Events	2024-11	arXiv	-	API	7
From Accidents to Insights: Leveraging Multimodal Data for Scenario-Driven ADS Testing	2025-02	arXiv	-	API	4
CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Models	2025-02	arXiv	GitHub	-	8
CrashAgent: Crash Scenario Generation via Multi-modal Reasoning	2025-05	arXiv	-	API	2
SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling	2025-07	arXiv	-	-	0
BENCH2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving	2025-08	arXiv	-	8×A800	2
Vision Language Model-based Testing of Industrial Autonomous Mobile Robots	2025-08	arXiv	-	-	1
VLMasStrategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion	2025-12	arXiv	-	-	0
Any2Critical: Safety-Critical Scenario Generation from Arbitrary Real-World Driving Contexts	2026-03	AAAI 2026	GitHub	-	0
CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation	2026-04	arXiv	GitHub	-	0
Agent-driven Long-tail Simulation for Autonomous Driving	2026-07	arXiv	Project	-	0

Scenario Analysis (VLM)

Paper	Date	Venue	Code	Hardware	Citation
Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving	2023-09	ICCV 2023	-	-	49
OpenAnnotate3D: Open-Vocabulary Auto-Labeling System for Multi-modal 3D Data	2023-10	ICRA 2024	GitHub	API&RTX4090	19
On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving	2023-11	ICIL 2024 Workshop on Large Language Models for Agents	GitHub	-	105
Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving	2023-11	ICRA 2024	GitHub	A100	114
LLM Multimodal Traffic Accident Forecasting	2023-11	Sensors 2023 MDPI	-	-	98
NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations	2024-01	WACVW LLVM-AD 2024	GitHub	8xA100	35
Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing	2024-02	UR 2024	-	-	22
Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving	2024-03	VLADR 2024	GitHub	RTX 3090Ti&V100	55
Embodied Understanding of Driving Scenarios (ELM)	2024-03	ECCV 2024	GitHub	-	0
LATTE: A Real-time Lightweight Attention-based Traffic Accident Anticipation Engine	2024-04	Information Fusion (Elsevier)	-	RTX4080	3
OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning	2024-05	CVPR 2025	GitHub	-	47
Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models	2024-05	arXiv	GitHub	-	0
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving	2024-06	ECCV 2024	GitHub	8xV100	118
ConnectGPT: Connect Large Language Models with Connected and Automated Vehicles	2024-06	IV 2024	-	-	21
SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving	2024-07	arXiv	-	-	14
Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving	2024-07	IROS 2024	GitHub	API	13
DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving	2024-08	IAVVC 2024	-	L4	13
V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models	2024-08	arXiv	-	RTX4090	46
Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis	2024-08	arXiv	-	-	0
Think-Driver: From Driving-Scene Understanding to Decision-Making with Vision Language Models	2024-09	ECCV 2024 Workshop	-	4xRTX4090	4
Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving	2024-09	AAAI 2025	Project	-	0
ContextVLM: Zero-Shot and Few-Shot Context Understanding for Autonomous Driving using Vision Language Models	2024-09	arXiv	GitHub	-	0
Efficient Driving Behavior Narration and Reasoning on Edge Device Using Large Language Models	2024-09	arXiv	-	-	0
VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes	2024-10	FLLM 2024	GitHub	RTX4090	41
Visual Adversarial Attack on Vision-Language Models for Autonomous Driving	2024-11	arXiv	-	A800	19
Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases	2024-12	WACV 2025	GitHub	8xA800	33
SFF Rendering-Based Uncertainty Prediction using VisionLLM	2024-12	AAAI 2025 Workshop LM4Plan	-	A100	3
AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving	2024-12	arXiv	GitHub	-	0
Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives	2025-01	ICCV 2025	GitHub	8xA800	28
Enhancing Large Vision Model in Street Scene Semantic Understanding through Leveraging Posterior Optimization Trajectory	2025-01	arXiv	-	2xRTX4090	7
Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding	2025-01	IAVVC 2024	-	-	15
DriveLM: Driving with Graph Visual Question Answering	2025-01	ECCV 2024	GitHub	8xV100	473
Scenario Understanding of Traffic Scenes Through Large Visual Language Models	2025-01	WACV 2025	-	A100	7
Application of Vision-Language Model to Pedestrian Behavior and Scene Understanding in Autonomous Driving	2025-01	arXiv	-	-	0
INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluation	2025-02	arXiv	-	A6000	8
Evaluating Multimodal Vision-Language Model Prompting Strategies for Visual Question Answering in Road Scene Understanding	2025-02	WACV workshop 2025	-	RTX4090	14
Vision-Integrated LLMs for Autonomous Driving Assistance: Human Performance Comparison and Trust Evaluation	2025-02	arXiv	-	-	0
NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Driving	2025-03	arXiv	-	-	4
Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving	2025-03	ICCV 2025	GitHub	8xV100	10
AutoDrive-QA- Automated Generation of Multiple-Choice Questions for Autonomous Driving Datasets Using Large Vision-Language Models	2025-03	arXiv	GitHub	3xA6000	4
DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding	2025-03	arXiv	GitHub	4xA6000	20
ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation	2025-03	arXiv	GitHub	32xA800	50
ChatBEV: A Visual Language Model that Understands BEV Maps	2025-03	arXiv	-	-	2
Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru	2025-03	arXiv	GitHub	A100&API	2
Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving	2025-03	arXiv	-	-	0
ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models	2025-03	arXiv	Project	-	0
Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios	2025-04	arXiv	GitHub	4xA100	6
ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models	2025-04	arXiv	GitHub	-	0
Vision Foundation Model Embedding-Based Semantic Anomaly Detection	2025-05	ICRA 2025 Workshop	-	-	3
OpenLKA: An Open Dataset of Lane Keeping Assist from Recent Car Models under Real-world Driving Conditions	2025-05	arXiv	GitHub	-	3
SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models	2025-05	NeurIPS 2025	GitHub	8xA800	6
Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving	2025-05	arXiv	GitHub	8xA800	5
Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist systems	2025-05	arXiv	-	-	3
AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving	2025-05	EMNLP Findings 2025	GitHub	-	0
FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving	2025-06	NeurIPS 2025	-	8xA6000	65
Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios	2025-06	arXiv	-	API	1
Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving	2025-06	arXiv	-	8xRTX4090	2
DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models	2025-06	arXiv	Dataset	-	0
Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving	2025-07	arXiv	GitHub	-	0
DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction	2025-07	arXiv	GitHub	8xH100	0
SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation	2025-07	ACMMM 2025	GitHub	-	3
DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving	2025-08	arXiv	GitHub	-	2
NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving	2025-09	ICRA	-	API&RTX5090	1
DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking	2025-09	arXiv	-	API&8xH20	1
Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning	2025-09	IROS 2025 RoboSense	GitHub	API	0
Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?	2025-09	arXiv	-	-	0
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models	2025-10	arXiv	-	-	0
STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes	2025-10	AAAI 2026	GitHub	16xH100	3
VLR-Driver: Large Vision-Language-Reasoning Models for Embodied Autonomous Driving	2025-10	ICCV 2025	-	-	0
Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos	2025-10	arXiv	GitHub	-	0
WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving	2025-11	arXiv	-	8xA6000	0
A TOOL FOR BENCHMARKING LARGE LANGUAGE MODELS’ ROBUSTNESS IN ASSESSING THE REALISM OF DRIVING SCENARIOS	2025-11	arXiv	-	API	0
V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models	2025-11	arXiv	-	RTX 4090	49
RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios	2025-11	arXiv	GitHub	-	0
Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks	2025-11	arXiv	-	-	0
Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach	2025-11	arXiv	-	-	0
RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System	2025-12	arXiv	GitHub	4xA100	0
OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving	2025-12	arXiv	-	16xA800	0
Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning	2025-12	CVPR 2026	-	-	0
Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensus	2025-12	arXiv	GitHub	-	0
Vision-Based Natural Language Scene Understanding for Autonomous Driving: An Extended Dataset and a New Model for Traffic Scene Description Generation	2026-01	arXiv	-	A100	0
AutoDriDM: An Explainable Benchmark for Decision-Making of Vision-Language Models in Autonomous Driving	2026-01	arXiv	-	A100	0
ScenePilot-Bench: A Large-Scale Dataset and Benchmark for Evaluation of Vision-Language Models in Autonomous Driving	2026-01	arXiv	GitHub	4xA800	0
Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning	2026-02	arXiv	-	-	0
DRIV-EX: Counterfactual Explanations for Driving LLMs	2026-02	arXiv	-	20xA100	0
DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving	2026-03	arXiv	-	-	0
DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving	2026-03	arXiv	-	8xH100	0
Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving	2026-03	arXiv	GitHub	-	0
BEVLM: Distilling Semantic Knowledge from LLMs into Bird’s-Eye View Representations	2026-03	arXiv	-	8xH100	0
Perception-Aware Multimodal Spatial Reasoning from Monocular Images	2026-03	arXiv	-	8xH100	0
Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures	2026-03	arXiv	IV 2025	-	0
More than the Sum: Panorama-Language Models for Adverse Omni-Scenes	2026-03	CVPR2026	-	https://github.com/InSAI-Lab/PanoVQA	0
Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning	2026-03	arXiv	-	4xA100	0
Are Video Reasoning Models Ready to Go Outside?	2026-03	arXiv	GitHub	4xA100	0
WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning	2026-03	arXiv	-	4xA40	0
VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events	2026-03	arXiv	-	32xH100	0
KLDrive: Fine-Grained 3D Scene Reasoning for Autonomous Driving based on Knowledge Graph	2026-03	arXiv	-	A6000	0
DRIVINGVQA: A Dataset for Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios	2026-03	ECAL 2026	GitHub	4xA100	0
3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding	2026-03	arXiv	GitHub	A100	0
Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets	2026-03	arXiv	-	-	0
Dual-Stage LLM Framework for Scenario-Centric Semantic Interpretation in Driving Assistance	2026-03	arXiv	-	-	0
A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection	2026-03	arXiv	-	5090	0
How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study (VENUSS)	2026-04	arXiv	-	API	0
BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving	2026-04	arXiv	-	API	0
CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning	2026-04	arXiv	-	8xA100	0
VAGNet: Vision-based Accident Anticipation with Global Features	2026-04	arXiv	-	RTX4090	0
SignReasoner: Compositional Reasoning for Complex Traffic Sign Understanding via Functional Structure Units	2026-04	arXiv	-	4xH100	0
CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs	2026-04	arXiv	-	API	0
Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Dataset (UniVLT / LTD)	2026-04	arXiv	-	8xH100	0
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving	2026-04	arXiv	-	API	0
Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference (VIBES)	2026-04	arXiv	-	RTX4090	0
VLM-VPI: A Vision-Language Reasoning Framework for Improving Automated Vehicle-Pedestrian Interactions	2026-04	arXiv	-	API	0
HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving	2026-05	arXiv	GitHub	-	0
Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models	2026-04	arXiv	GitHub	-	0
Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding	2026-05	arXiv	-	-	0
C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving	2026-05	arXiv	-	-	0
D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving	2026-05	arXiv	GitHub	-	0
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving	2026-05	NeurIPS 2026	Project	-	0
PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction	2026-05	arXiv	GitHub	-	0
ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving	2026-05	arXiv	-	-	0
Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video	2026-05	arXiv	-	-	0
nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving	2026-05	arXiv	Project	-	0
Vision-Language Work Zone Intelligence for Safety-Critical Speed Regulation of Mixed-Autonomy Vehicles in Dynamic Environments	2026-06	arXiv	GitHub	-	0
Multi-Stage VLM Pipeline for Zero-Shot Traffic Accident Understanding	2026-05	arXiv	GitHub	-	0
DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models	2026-06	arXiv	-	-	0
GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving	2026-06	arXiv	GitHub	-	0
Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding	2026-06	arXiv	-	-	0
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City	2026-06	arXiv	-	-	0
AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding	2026-07	arXiv	-	-	0
DispatchRAG: Grounding Emergency Dispatch Decisions in Real-World Protocols from Traffic Accident Video	2026-07	arXiv	-	-	0
Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding	2026-07	arXiv	-	-	0
CAViAR: A Causal Video Dataset for Fine-Grained Accident Reasoning in Real-World Scenarios	2026-08	arXiv	GitHub	-	0
Observe Before You Alert: Adaptive Driver Alerting with Vision-Language Models	2026-09	arXiv	-	RTX5090	0
CrossView: Can Vision-Language Models Reason Across Cameras?	2026-08	arXiv	GitHub	API	0
SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions	2026-08	arXiv	-	V100S	0

🌟 Multimodal Large Language Models for Autonomous Driving

Scenario Generation (MLLM)

Paper	Date	Venue	Code	Hardware	Citation
Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model	2024-11	Tsinghua Science and Technology	-	API&RTX3090	12
LMM-enhanced Safety-Critical Scenario Generation for Autonomous Driving System Testing From Non-Accident Traffic Videos	2025-01	arXiv	GitHub	RTX2080	6
Multi-modal Traffic Scenario Generation for Autonomous Driving System Testing	2025-06	FSE 2025	-	API	1
Talk2Traffic: Interactive and Editable Traffic Scenario Generation for Autonomous Driving with Multimodal Large Language Model	2025-06	CVPR 2025 WDFM-AD	GitHub	-	3

| OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning | 2026-06 | arXiv | - | - | 0 |

Scenario Analysis (MLLM)

Paper	Date	Venue	Code	Hardware	Citation
DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model	2023-10	IEEE Robotics and Automation Letters 2024	GitHub	-	615
Dolphins: Multimodal Language Model for Driving	2023-12	ECCV 2024	GitHub	4xA100	139
AccidentGPT: Accident analysis and prevention from V2X Environmental Perception with Multi-modal Large Model	2023-12	IV 2024	GitHub	-	45
Lidar-llm: Exploring the potential of large language models for 3d lidar understanding	2023-12	AAAI 2025	GitHub	A100	125
LingoQA: Visual Question Answering for Autonomous Driving	2023-12	ECCV 2024	GitHub	8xA100	143
Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models	2024-01	CVPR 2024	GitHub	-	106
MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding	2024-01	CVPR 2024	GitHub	8xV100&2xA100	65
Probing Multimodal LLMs as World Models for Driving	2024-05	arXiv	-	-	0
WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-Grained Spatial-Temporal Understanding	2024-06	ECCV 2024	GitHub	-	19
Semantic Understanding of Traffic Scenes with Large Vision Language Models	2024-06	IV 2024	GitHub	API	27
VLAAD: Vision and Language Assistant for Autonomous Driving	2024-06	WACVW 2024	GitHub	-	52
InternDrive: A Multimodal Large Language Model for Autonomous Driving Scenario Understanding	2024-07	AIAHPC 2024	-	API	4
WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving	2024-07	ICML 2025	GitHub	-	0
Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events	2024-09	Vehicles 2024 MDPI	-	API	10
MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios	2024-12	arXiv	GitHub	A800	7
Application of Multimodal Large Language Models in Autonomous Driving	2024-12	arXiv	-	-	0
Distilling Multi-modal Large Language Models for Autonomous Driving	2025-01	CVPR 2025	-	-	23
TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos	2025-01	arXiv	GitHub	-	0
TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes	2025-02	ICML 2025	GitHub	4xA100	13
ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding	2025-02	WACV Workshop 2025	GitHub	2xA100	8
Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning	2025-02	IEEE RAL	-	8xL20	18
A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving	2025-03	arXiv	-	-	1
HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving	2025-03	International Journal of Computer Vision	-	8xV100	68
NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models	2025-03	ICCV 2025	GitHub	-	8
Tracking Meets Large Multimodal Models for Driving Scenario Understanding	2025-03	arXiv	GitHub	2xA6000	4
SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment	2025-03	CVPR 2025	GitHub	8xA100	34
V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving	2025-04	CVPR DriveX 2025	GitHub	H100	3
Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding	2025-04	arXiv	GitHub	-	7
V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models	2025-04	arXiv	GitHub	8xA100	22
ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Driving	2025-05	arXiv	-	2xH200	3
DriveSOTIF: Advancing SOTIF Through Multimodal Large Language Models	2025-05	arXiv	GitHub	-	0
X-Driver: Explainable Autonomous Driving with Vision-Language Models	2025-06	arXiv	-	-	5
STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving	2025-06	NeurIPS 2025	GitHub	-	2
SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding	2025-08	arXiv	-	8xA100	2
RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving	2025-08	ICCV2025	GitHub	32xA100	9
Passing the Driving Knowledge Test (DriveQA)	2025-08	ICCV 2025	-	-	0
EMMA: End-to-End Multimodal Model for Autonomous Driving	2025-09	TMLR	-	-	161
Investigating Traffic Accident Detection Using Multimodal Large Language Models	2025-09	IAVVC 2025	-	-	1
AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond	2025-09	arXiv	GitHub	-	0
Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMs	2025-10	Transportation Research Part C: Emerging Technologies	-	4090	0
BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving	2025-12	arXiv	-	4xH100	0
MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion	2025-12	arXiv	-	-	0
Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Model	2026-02	arXiv	-	4090	0
DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding	2026-03	arXiv	GitHub	4xA100	0
Interpretable Traffic Responsibility from Dashcam Video via Legal Multi-Agent Reasoning	2026-03	arXiv	-	-	0
ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation	2026-03	arXiv	GitHub	8xH20	0
AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models	2026-04	CVPR 2026 Findings	-	8xA100	0
XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments	2026-04	arXiv	-	-	0
V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for MLLMs in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views	2026-04	arXiv	GitHub	-	0
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios	2026-05	arXiv	-	-	0
Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis	2026-05	arXiv	-	-	0
GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logic	2026-05	arXiv	-	-	0
Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving	2026-06	arXiv	-	-	0
OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs	2026-06	arXiv	-	-	0
Beyond Scene Priors: Fine-Grained Traffic Scene Reasoning with Benchmarking and Query-Guided Small-Object Focus	2026-07	arXiv	-	-	0
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models	2026-07	arXiv	Project	-	0
Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA	2026-09	arXiv	GitHub	RTX6000	0
Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering	2026-08	arXiv	GitHub	4×RTX Pro 6000	0
UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations	2026-08	arXiv	GitHub	API	0
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving	2026-09	arXiv	-	-	0
From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning	2026-08	arXiv	-	8×A100	0
Drive the Thoughts: Runtime Monitoring of VLA Reasoning-Trajectory Consistency	2026-08	arXiv	GitHub	A100 & API	0

🌟 Diffusion Models for Autonomous Driving

Scenario Generation (Diffusion Models)

Paper	Date	Venue	Code	Hardware	Citation
Guided Conditional Diffusion for Controllable Traffic Simulation	2022-10	ICRA 2023	GitHub	-	223
Generating Driving Scenes with Diffusion	2023-05	ICRA Workshop	-	-	25
DiffScene: Guided Diffusion Models for Safety-Critical Scenario Generation	2023-06	AdvML-Frontiers 2023	-	RTX3090	76
BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout	2023-09	arXiv	-	-	45
DriveSceneGen: Generating Diverse and Realistic Driving Scenarios From Scratch	2023-09	IEEE Robotics and Automation Letters 2024	GitHub	4xV100	22
MagicDrive: Street View Generation with Diverse 3D Geometry Control	2023-10	ICLR 2024	GitHub	V100	219
DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model	2023-10	ECCV 2024	GitHub	8xA100	80
Language-guided traffic simulation via scene-level diffusion	2023-11	CoRL 2023	-	RTX 3090	121
Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion	2023-11	NeurIPS 2023	-	2xA6000	64
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving	2023-11	CVPR 2024	GitHub	-	127
SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries	2023-12	ECCV 2024	GitHub	4xA6000	28
Text2Street: Controllable Text-to-image Generation for Street Views	2024-02	ICPR 2024	-	-	12
ChatTraffic: Text-to-Traffic Generation via Diffusion Model	2024-02	IEEE Transactions on Intelligent Transportation Systems	GitHub	RTX4090	19
GEODIFFUSION: Text-Prompted Geometric Control for Object Detection Data Generation	2024-02	LCLR 2024	GitHub	-	56
GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model	2024-04	ITSC 2024	-	RTX3090Ti	6
Versatile Behavior Diffusion for Generalized Traffic Agent Simulation	2024-04	RSS 2024	GitHub	8xL40	13
SceneControl: Diffusion for Controllable Traffic Scene Generation	2024-05	ICRA 2024	-	-	33
SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic	2024-07	ECCV 2024	GitHub	-	24
DrivingGen: Efficient Safety-Critical Driving Video Generation with Latent Diffusion Models	2024-07	ICME 2024	-	-	9
Controllable Traffic Simulation through LLM-Guided Hierarchical Chain-of-Thought Reasoning	2024-09	IROS 2025	-	-	7
AdvDiffuser: Generating Adversarial Safety-Critical Driving Scenarios via Guided Diffusion	2024-10	IROS 2024	-	4xRTX3090	21
Data-driven Diffusion Models for Enhancing Safety in Autonomous Vehicle Traffic Simulations	2024-10	arXiv	-	-	4
DiffRoad: Realistic and Diverse Road Scenario Generation for Autonomous Vehicle Testing	2024-11	arXiv	-	-	7
SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout	2024-12	NeurIPS 2024	-	-	48
Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation	2025-02	arXiv	-	-	2
Causal Composition Diffusion Model for Closed-loop Traffic Generation	2025-02	CVPR 2025	GitHub	4xV100	11
Rolling Ahead Diffusion for Traffic Scene Simulation	2025-02	AAAI 2025 Workshop	-	V100	2
AVD2: Accident Video Diffusion for Accident Video Description	2025-03	ICRA 2025	GitHub	-	16
DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance	2025-03	arXiv	GitHub	8xA800	4
Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments	2025-03	CVPR 2025	GitHub	8xA100	16
DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Models	2025-03	arXiv	-	-	4
DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer	2025-04	arXiv	-	8xA800	4
Decoupled Diffusion Sparks Adaptive Scene Generation	2025-04	ICCV 2025	GitHub	A100	10
DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion	2025-05	ICRA 2025	GitHub	-	3
LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios	2025-05	arXiv	-	RTX4090	7
Dual-Conditioned Temporal Diffusion Modeling for Driving Scene Generation	2025-05	ICAR 2025	GitHub	V100	1
Diffusion Models for Safety Validation of Autonomous Driving Systems	2025-06	arXiv	-	GTX1080Ti	1
Diffusion-Based Generation and Imputation of Driving Scenarios from Limited Vehicle CAN Data	2025-09	ITSC 2025	-	A100	0
Path Diffuser: Diffusion Model for Data-Driven Traffic Simulator	2025-09	-	GitHub	-	0
3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion	2025-10	NeurIPS 2025 Workshop	-	H20	3
VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion	2025-12	arXiv	-	8x4090	0
FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving	2026-03	arXiv	-	-	0
Controllable Latent Diffusion for Traffic Simulation	2026-03	arXiv	GitHub	-	0
ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation	2026-04	arXiv	Project	-	0
DriveCtrl: Conditioned Sim-to-Real Driving Video Generation	2026-05	arXiv	-	L40	0
AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond	2026-05	arXiv	Project	8xA100	0
SafeGen: Goal-Conditioned Video Diffusion of Safety-Critical Scenarios for VLM-Based Autonomous Driving	2026-07	arXiv	GitHub	-	0
One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation	2026-09	arXiv	-	-	0
CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation	2026-09	arXiv	-	H200	0
Safety-Critical Scenanrio Emerges from Initial Scene	2026-09	arXiv	-	-	0

Scenario Analysis (Diffusion Models)

Paper	Date	Venue	Code	Hardware	Citation
AVD2: Accident Video Diffusion for Accident Video Description	2025-03	ICRA 2025	GitHub	-	16

🌟 World Models for Autonomous Driving

World Models for Autonomous Driving

Paper	Date	Venue	Code	Hardware	Citation
DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving	2023-09	ECCV 2024	GitHub	A800	376
GAIA-1: A Generative World Model for Autonomous Driving	2023-09	arXiv Wayve	-	32xA100	428
TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction	2023-09	ICRA 2023	GitHub	6xRTX2080Ti	70
Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion	2023-11	ICLR 2024	-	-	102
MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations	2023-11	IV 2025	-	-	4
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving	2023-11	CVPR 2024	GitHub	A40	280
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability	2024-03	NeurIPS 2024	GitHub	8xA100	181
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation	2024-05	AAAI 2025	GitHub	A800	162
UniScene: Multi-Camera Unified Pre-training via 3D Scene Reconstruction for Autonomous Driving	2024-08	RAL 2024	GitHub	8xA40	3
WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation	2024-08	ECCV 2024	GitHub	8xA6000	84
Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving	2024-08	CVPR 2024	GitHub	-	127
DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving	2024-08	arXiv	GitHub	8xA100	58
DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation	2024-11	CVPR 2025	GitHub	H20	66
ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration	2024-11	CVPR 2025	GitHub	-	55
MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes	2024-11	arXiv	GitHub	-	56
MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control	2024-11	ICCV 2025	GitHub	8xA800	18
ACT-Bench: Towards Action Controllable World Models for Autonomous Driving	2024-12	arXiv	-	8xH100	7
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control	2024-12	CVPR 2025	GitHub	H100	37
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT	2024-12	arXiv	GitHub	32xRTX4090&64xA100	41
Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving	2025-01	AAAI 2025	GitHub	8xA100	25
MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction	2025-02	arXiv	GitHub	32xA800	20
Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning	2025-03	arXiv	GitHub	-	57
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving	2025-03	arXiv	-	256xH100	66
Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latent Space	2025-03	arXiv	-	64xA800	5
Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception	2025-03	arXiv	GitHub	-	5
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control	2025-04	arXiv	GitHub	1024xH100	33
DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment	2025-04	ACMMM 2025	GitHub	-	6
OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving	2025-05	arXiv	GitHub	8xA100	64
PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth	2025-05	IROS 2025	-	8xA100	2
ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos	2025-05	arXiv	-	4xA100	4
ReSim: Reliable World Simulation for Autonomous Driving	2025-06	arXiv	GitHub	40xA100	10
SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model	2025-06	CVPR 2025	-	-	6
Epona: Autoregressive Diffusion World Model for Autonomous Driving	2025-06	ICCV 2025	GitHub	48xA100	22
STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation	2025-06	IROS 2025	-	-	3
DeepVerse: 4D Autoregressive Video Generation as a World Model	2025-06	arXiv	GitHub	A100	11
World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving	2025-07	Commun Eng	GitHub	RTX4090	2
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation	2025-08	ICCV 2025	GitHub	32xH20	27
Driving scenario generation and evaluation using a structured layer representation and foundational models	2025-11	arXiv	GitHub	API	0
GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation	2025-12	CVPR 2026	GitHub	16xA100	0
VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning	2026-03	arXiv			0
X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving	2026-03	arXiv	GitHub		0
Toward Physically Consistent Driving Video World Models under Challenging Trajectories	2026-03	arXiv	GitHub	48xH20	0
DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving	2026-03	ICLR2026	GitHub		0
Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation	2026-03	arXiv	GitHub	16xH100	0
Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation	2026-03	arXiv	GitHub	16xA100	0
LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving	2026-04	arXiv	-	-	0
HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation	2026-04	arXiv	GitHub	-	0
Is Your Driving World Model an All-Around Player? (WorldLens)	2026-05	arXiv	-	-	0
Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving	2026-05	arXiv	Project	-	0
Diffusion Transformer World-Action Model for AV Scene Prediction	2026-06	arXiv	GitHub	-	0
ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving	2026-06	arXiv	GitHub	-	0
GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation	2026-08	arXiv	GitHub	16×A100	0
4D-WAM: 4D Consistent World Modeling for Autonomous Driving	2026-08	arXiv	-	16×H200	0

📊 Datasets Comparison

The following figure shows the usage distribution of different foundation model types across autonomous driving datasets:

Datasets Comparison

Dataset	Year	Img	View	Real	Lidar	Radar	Traj	3D	2D	Lane	Weather	Time	Region	Company
CamVid	2009	RGB	FPV	✔	✖️	✖️	✖️	✖️	✔	✔	✔	D	U	-
KITTI	2013	RGB/S	FPV	✔	✔	✖️	✔	✔	✔	✔	✔	D	U/R/H	-
Cyclists	2016	RGB	FPV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	U	-
Cityscapes	2016	RGB/S	FPV	✔	✖️	✖️	✖️	✔	✔	✔	✖️	D	U	-
SYNTHIA	2016	RGB	FPV	✖️	✖️	✖️	✖️	✔	✔	✔	✔	D/N	U	-
Campus	2016	RGB	BEV	✖️	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	C	-
RobotCar	2016	RGB	FPV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D/N	U	-
Mapillary	2017	RGB	FPV	✔	✖️	✖️	✖️	✖️	✔	✔	✔	D/N	U	-
P.F.B.	2017	RGB	FPV	✔	✖️	✖️	✖️	✖️	✔	✔	✔	D/N	U	-
BDD100K	2018	RGB	FPV	✔	✖️	✖️	✖️	✔	✔	✔	✔	D	U/H	-
HighD	2018	RGB	BEV	✔	✖️	✖️	✖️	✖️	✔	✔	✖️	D	H	-
Udacity	2018	RGB	FPV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	U	-
KAIST	2018	RGB/S	FPV	✔	✔	✖️	✖️	✖️	✔	✔	✔	D/N	U	-
Argoverse	2019	RGB/S	FPV	✔	✔	✖️	✖️	✖️	✔	✔	✔	D/N	U	-
TRAF	2019	RGB	FPV	✔	✖️	✖️	✖️	✖️	✔	✔	✔	D	U	-
ApolloScape	2019	RGB/S	FPV	✔	✖️	✖️	✖️	✔	✔	✔	✔	D	U	-
ACFR	2019	RGB	BEV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	RA	-
H3D	2019	RGB	FPV	✔	✖️	✖️	✖️	✖️	✔	✔	✔	D	U	-
INTERACTION	2019	RGB	BEV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	I/RA	-
Comma2k19	2019	RGB	FPV	✔	✖️	✖️	✔	✔	✖️	✖️	✖️	D/N	U/S/R/H	-
InD	2020	RGB	BEV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	I	-
RounD	2020	RGB	BEV	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	D	RA	-
nuScenes	2020	RGB	FPV	✔	✔	✔	✖️	✔	✔	✔	✔	D/N	U	-
Lyft Level 5	2020	RGB	FPV	✔	✔	✔	✖️	✔	✔	✔	✔	D/N	U/S	-
Waymo Open	2020	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U	-
A*3D	2020	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U	-
RobotCar Radar	2020	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U	-
Toronto3D	2020	RGB	BEV	✔	✔	✖️	✔	✔	✖️	✔	✖️	D/N	U	University of Waterloo
A2D2	2020	RGB	FPV	✔	✔	✔	✔	✔	✔	✖️	✔	✔	D	U/H/S/R
WADS	2020	RGB	FPV	✔	✔	✔	✔	✔	✖️	✖️	✔	D/N	U/S/R	Michigan Technological University
Argoverse 2	2021	RGB/S	FPV	✔	✔	✖️	✖️	✔	✔	✔	✔	D/N	U	-
PandaSet	2021	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U	-
ONCE	2021	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U	-
Leddar PixSet	2021	RGB	FPV	✔	✔	✖️	✔	✔	✔	✖️	✔	D/N	U/S/R	Leddar
ZOD	2022	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U/R/S/H	Zenseact
IDD-3D	2022	RGB	FPV	✔	✔	✖️	✖️	✔	✔	✖️	✖️	-	R	INAI
CODA	2022	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U/S/R	Huawei
SHIFT	2022	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U/S/R/H	ETH Zürich
DeepAccident	2023	RGB/S	FPV/BEV	✖️	✔	✖️	✖️	✔	✔	✔	✔	D/N	U/S/R/H	HKU, Huawei, CARLA
Dual_Radar	2023	RGB	FPV	✔	✔	✔	✔	✔	✖️	✔	✔	D/N	U	Tsinghua University
V2V4Real	2023	RGB	FPV	✔	✔	✖️	✔	✔	✖️	✔	✖️	-	U/H/S	UCLA Mobility Lab
SCaRL	2024	RGB/S	FPV/BEV	✖️	✔	✔	✔	✔	✔	✔	✔	D/N	U/S/R/H	Fraunhofer CARLA
MARS	2024	RGB	FPV	✔	✔	✔	✔	✔	✔	✔	✔	D/N	U/S/H	NYU, MAY Mobility
Scenes101	2024	RGB	FPV	✔	✖️	✖️	✔	✖️	✖️	✔	✔	D/N	U/S/R/H	Wayve
TruckScenes	2025	RGB	FPV	✔	✔	✔	✔	✔	✖️	✔	✔	D/N	H/U	MAN

Notes: View: FPV=First-Person, BEV=Bird's-Eye; Time: D=Day, N=Night; Region: U=Urban, R=Rural, H=Highway, S=Suburban, C=Campus, I=Intersection, RA=Road Area; Img: RGB/S=RGB+Stereo

🎮 Simulators

The following figure shows the usage distribution of different foundation model types across autonomous driving simulators:

Simulators

Simulator	Year	Back-end	Open Source	Realistic Perception	Custom Scenario	Real World Map	Human Design Map	Python API	C++ API	ROS API	Company
TORCS	2000	None	✔	✔	✔	✖️	✖️	✖️	✖️	✖️	-
Webots	2004	ODE	✔	✔	✔	✔	✖️	✔	✔	✖️	-
CarRacing	2017	None	✔	✖️	✖️	✖️	✔	✔	✖️	✖️	-
CARLA	2017	UE4	✔	✔	✔	✖️	✔	✔	✔	✔	-
SimMobilityST	2017	None	✔	✖️	✖️	✖️	✖️	✖️	✖️	✖️	-
GTA-V	2017	RAGE	✖️	✔	✖️	✖️	✖️	✖️	✖️	✖️	-
highway-env	2018	None	✔	✖️	✔	✖️	✔	✔	✖️	✖️	-
Deepdrive	2018	UE4	✔	✔	✔	✖️	✔	✔	✔	✖️	-
esmini	2018	Unity	✔	✖️	✖️	✖️	✖️	✔	✖️	✖️	-
AutonoViSim	2018	PhysX	✖️	✔	✔	✖️	✖️	✔	✖️	✖️	-
AirSim	2018	UE4	✔	✔	✔	✖️	✔	✔	✔	✖️	-
SUMO	2018	None	✔	✖️	✔	✔	✔	✖️	✔	✖️	-
Apollo	2018	Unity	✔	✔	✔	✔	✔	✔	✔	✖️	-
Sim4CV	2018	UE4	✔	✔	✔	✖️	✔	✔	✖️	✖️	-
MATLAB	2018	MATLAB	✖️	✔	✔	✔	✔	✔	✔	✔	Mathworks
Scenic	2019	None	✔	✔	✔	✔	✔	✔	✖️	✖️	Toyota Research Institute, UC Berkeley
SUMMIT	2020	UE4	✔	✔	✔	✖️	✔	✔	✔	✖️	-
MultiCarRacing	2020	None	✔	✖️	✔	✖️	✔	✔	✖️	✖️	-
SMARTS	2020	None	✔	✔	✔	✔	✔	✔	✖️	✖️	-
LGSVL	2020	Unity	✔	✔	✔	✔	✔	✔	✔	✔	-
CausalCity	2020	UE4	✔	✔	✔	✔	✔	✔	✖️	✖️	-
Vista	2020	None	✔	✔	✔	✔	✖️	✔	✖️	✖️	MIT
MetaDrive	2021	Panda3D	✔	✔	✔	✔	✔	✔	✔	✖️	-
L2R	2021	UE4	✔	✔	✔	✔	✔	✔	✔	✖️	-
AutoDRIVE	2021	Unity	✔	✔	✔	✔	✔	✔	✔	✔	-
Nuplan	2021	None	✔	✔	✔	✔	✔	✔	✖️	✖️	Motional
AWSIM	2021	Unity	✔	✔	✔	✔	✔	✖️	✖️	✔	Autoware
InterSim	2022	None	✔	✔	✔	✔	✖️	✔	✖️	✖️	Tsinghua
Nocturne	2022	None	✔	✔	✔	✔	✔	✔	✔	✖️	Facebook
BeamNG.tech	2022	Soft-body physics	✖️	✔	✔	✖️	✔	✔	✖️	✔	BeamNG GmbH
Waymax	2023	JAX	✔	✔	✔	✖️	✔	✔	✖️	✖️	Waymo
UNISim	2023	None	✖️	✔	✔	✔	✖️	✖️	✔	✖️	Waabi
TBSim	2023	None	✔	✔	✔	✔	✔	✔	✖️	✖️	NVIDIA
Nvidia DriveWorks	2024	Nvidia GPU	✖️	✔	✔	✔	✖️	✔	✔	✖️	NVIDIA

🏆 Foundation Model Benchmark Challenges (2022–2025)

Benchmark Challenges

Autonomous Driving

Name	Host
CARLA AD Challenge	CARLA
DRL4Real	ICCV
Waymo Open Dataset Challenge	Waymo / CVPR WAD
Argoverse 2: Scenario Mining	ArgoAI
Roboflow-20VL	Roboflow-VL / CVPR
AVA Challenge	AVA Challenge Team

Other Fields Related to Generation and Analysis

Name	Host
IGLU Challenge	NeurIPS / IGLU Team
LLM Efficiency Challenge	NeurIPS
Trojan Detection	NeurIPS / CAIS
SMART-101	CVPR
NICE Challenge	CVPR / LG Research
SyntaGen	CVPR
Habitat Challenge	CVPR / FAIR
BIG-bench	Google Research
BIG-bench Hard (BBH)	Google Research
HELM	Stanford CRFM
MMBench	OpenCompass
MMMU	CVPR / U-Waterloo / OSU
Open LLM Leaderboard	VILA-Lab
Text-to-Image Leaderboard	Artificial Analysis
Ego4D	FAIR
VizWiz Grand Challenge	CVPR VizWiz Workshop
MedFM	NeurIPS / Shanghai AI Laboratory
3D Scene Understanding	CVPR

🔗 Useful Resources and Links

Common Tools and Frameworks

This section provides links to commonly used tools, frameworks, and resources for working with foundation models in autonomous driving.

Model Repositories and Leaderboards

• Hugging Face Models - Large collection of pre-trained foundation models
• Hugging Face Vision-Language Models Leaderboard - Benchmark and leaderboard for VLMs
• Open LLM Leaderboard - Comprehensive LLM benchmarking
• Awesome LLM - Curated list of Large Language Models resources

Model Inference Frameworks

• vLLM - High-throughput and memory-efficient inference engine for LLMs
• LMDeploy - Toolkit for compressing, deploying, and serving LLMs
• Ollama - Run large language models locally
• Text Generation Inference - Production-ready inference container by Hugging Face
• TensorRT-LLM - High-performance inference library by NVIDIA

Training and Fine-tuning

• Hugging Face Transformers - State-of-the-art ML for PyTorch, TensorFlow, and JAX
• LLaMA Factory - Unified framework for fine-tuning LLMs
• Axolotl - Tool for fine-tuning large language models
• PEFT - Parameter-Efficient Fine-Tuning methods

Contributing

We welcome contributions from the community! If you have research papers, tools, or resources to add, please create a pull request or open an issue.

License

This repository is released under the Apache 2.0 license.