Lan Feng^1,†, Yang Gao^1,†, Éloi Zablocki^2,‡, Quanyi Li, Wuyang Li^1,†, Sichao Liu^1,†, Matthieu Cord^2,3,‡, Alexandre Alahi^1,†

¹ EPFL, Switzerland ² Valeo.ai, France ³ Sorbonne Université, France

🏆 1st Place – Waymo Open Dataset Vision-based E2E Driving Challenge (UniPlan entry)
🏆 #1 on Leaderboards – Waymo Open Dataset Vision-based E2E Driving & NAVSIM v1/v2 (RAP entry)
🏆 State-of-the-art – Bench2Drive benchmark

---

🚗 RAP (Rasterization Augmented Planning) is a scalable data augmentation pipeline for end-to-end autonomous driving.
It leverages lightweight 3D rasterization to generate counterfactual recovery maneuvers and cross-agent views and Raster-to-Real feature alignment to bridge the sim-to-real gap in feature space, achieving state-of-the-art performance on multiple benchmarks.

---

News

• Oct. 6th, 2025: Code released🔥!

---

Getting Started

Environment Setup

conda create -n rap python=3.9 
conda activate rap
# please adjust the torch version according to your cuda version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
pip install -e ./nuplan-devkit
pip install -e .

Set environment variable

set the environment variable based on where you place the PAD directory.

export NUPLAN_MAP_VERSION="nuplan-maps-v1.0"
export NUPLAN_MAPS_ROOT="$HOME/rap_workspace/dataset/maps"
export NAVSIM_EXP_ROOT="$HOME/rap_workspace/exp"
export NAVSIM_DEVKIT_ROOT="$HOME/rap_workspace/navsim"
export OPENSCENE_DATA_ROOT="$HOME/rap_workspace/dataset"
export Bench2Drive_ROOT="$HOME/rap_workspace/Bench2Drive"

Data Processing

Note: This step generate data that shares the same format (with additional rasterized camera views) as Navsim.

• Data Processing

Please organize the generated data in the same way as HERE.

Navsim training and evaluation

1. cache training data and metric

# data caching ego vehicle
export OPENSCENE_DATA_ROOT="$HOME/rap_workspace/dataset"
python navsim/planning/script/run_dataset_caching.py \
agent=rap_agent \
dataset=navsim_dataset \
agent.config.trajectory_sampling.time_horizon=5 \
agent.config.cache_data=True \
train_test_split=navtrain \
train_test_split.scene_filter.has_route=false \
experiment_name=trainval_test \
worker.threads_per_node=64 \
cache_path=./cache/rap_ego \

# data caching cross-agent
export OPENSCENE_DATA_ROOT="$HOME/rap_workspace/dataset_aug"
python navsim/planning/script/run_dataset_caching.py \
agent=rap_agent \
dataset=navsim_dataset \
agent.config.trajectory_sampling.time_horizon=5 \
agent.config.cache_data=True \
train_test_split=navtrain \
train_test_split.scene_filter.has_route=false \
experiment_name=trainval_test \
worker.threads_per_node=64 \
cache_path=./cache/rap_aug \


# data caching recovery-oriented perturbation
export OPENSCENE_DATA_ROOT="$HOME/rap_workspace/dataset_perturbed"
python navsim/planning/script/run_dataset_caching.py \
agent=rap_agent \
dataset=navsim_dataset \
agent.config.trajectory_sampling.time_horizon=5 \
agent.config.cache_data=True \
train_test_split=navtrain \
train_test_split.scene_filter.has_route=false \
experiment_name=trainval_test \
worker.threads_per_node=64 \
cache_path=./cache/rap_perturbed \

# train metric caching
python navsim/planning/script/run_training_metric_caching.py \
train_test_split=navtrain \
cache.cache_path=./train_metric_cache \
worker.threads_per_node=32 \

Set ray to True. https://github.com/vita-epfl/RAP/blob/1d9a886e00a202021e0378b96f19438035cec75d/navsim/agents/rap_dino/rap_agent.py#L46 And run the following before starting training for accelerated pdm score calculation.

ulimit -u 65535
redis_pw=$(openssl rand -hex 16)            
ray start --head --port 6396 --num-gpus 8 \
          --dashboard-host 0.0.0.0 \
          --redis-password "$redis_pw"


export ip_head="127.0.0.1:6396"             
export redis_password="$redis_pw"
export num_nodes=1

3. train navsim model

python navsim/planing/script/run_training.py \
agent=rap_agent \
agent.config.pdm_scorer=True \
agent.config.distill_feature=True \
experiment_name=test \
train_test_split=navtrain \
split=trainval \
cache_path=./cache/rap_ego \
cache_path_perturbed=./cache/rap_perturbed \
cache_path_others=./cache/rap_aug \
use_cache_without_dataset=True \
force_cache_computation=False \
dataloader.params.batch_size=64 \
dataset=navsim_dataset \
agent.config.trajectory_sampling.time_horizon=5

4. test navsim model

Please refer to Navsim for more details.

Waymo Finetuning

1. Download Waymo E2E Driving Dataset: https://waymo.com/open/download/
2. Dataset caching:

python navsim/planning/script/run_waymo_dataset_caching.py \
agent=rap_agent \
dataset=waymo_dataset \
dataset.include_val=False \
experiment_name=trainval_test \
train_test_split=navtrain \
worker.threads_per_node=64 \
cache_path=./cache/rap_waymo \
waymo_raw_path=

3. finetune pretrained model on Waymo

python navsim/planing/script/run_training.py \
agent=rap_agent \
agent.config.pdm_scorer=False \
agent.config.distill_feature=False \
experiment_name=waymo_finetune \
train_test_split=navtrain  \
train_test_split.scene_filter=navtrain \
split=trainval   \
trainer.params.max_epochs=20 \
cache_path=./cache/rap_waymo \
use_cache_without_dataset=True  \
force_cache_computation=False \
dataloader.params.batch_size=16 \
agent.config.trajectory_sampling.time_horizon=5 \
dataset=waymo_dataset \
agent.checkpoint_path=$CHECKPOINT \
agent.lr=1e-5 \

4. Leaderboard submission

mkdir waymo_submission
#Put all the ckpt files in the same directory /waymo_submission so that it can run ensembling

Python $NAVSIM_DEVKIT_ROOT/navsim/planning/script/run_waymo_submission.py \
        agent=navsim_agent \
        agent.config.pdm_scorer=False \
        agent.config.distill_feature=False \
        experiment_name=ldb \
        train_test_split=navtrain  \
        train_test_split.scene_filter=navtrain \
        split=trainval \
        trainer.params.max_epochs=20 \
        cache_path="./cache/rap_waymo" \
        use_cache_without_dataset=True  \
        force_cache_computation=False \
        dataloader.params.batch_size=8 \
        agent.config.trajectory_sampling.time_horizon=5 \
        dataset=waymo_dataset \
        agent.checkpoint_path=./waymo_submission/1.ckpt

Checkpoints

Results on NAVSIM v1

Results on NAVSIM v2

Results on Waymo

Citation

@misc{feng2025rap3drasterizationaugmented,
      title={RAP: 3D Rasterization Augmented End-to-End Planning}, 
      author={Lan Feng and Yang Gao and Eloi Zablocki and Quanyi Li and Wuyang Li and Sichao Liu and Matthieu Cord and Alexandre Alahi},
      year={2025},
      eprint={2510.04333},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.04333}, 
}