A counterfactual long-tail benchmark for driving decisions. The driving context (road geometry, ego pose, navigation command, motion history) is held fixed and only the object in the scene changes. The benchmark asks whether a model's decision updates when it should and holds when it should not.
Given a front-camera image and the driving context, output one longitudinal and one lateral action:
| Axis | Actions |
|---|---|
| Longitudinal | keep_speed, slow_down, yield, creep, stop, speed_up |
| Lateral | keep_lane, nudge_left, nudge_right |
Each sample has an accepted set of action pairs (1 to 14 pairs, 3.5 on average). A prediction is correct if it is in the set.
Both splits share identical target objects and road geometry and differ only in background traffic.
| Split | Samples | Background traffic |
|---|---|---|
vfull | 1,768 | naturalistic |
vclean | 1,768 | background vehicles removed |
50 obstacle types in five categories, grouped by the response they require:
| Category | Samples per split | Expected behaviour |
|---|---|---|
false_positive | 435 | preserve the nominal action |
entity_living | 348 | yield or stop |
road_hazard | 348 | adapt to the hazard |
entity_nonliving | 347 | slow down or steer around |
full_block | 290 | stop |
Object position is controlled separately: center (1,159), full_lane_blockage
(290), left_biased (145), right_biased (145), airborne (29).
vfull/
├── manifest.json sample ids in order
└── <sample_id>/
├── front_camera.jpg
├── prompt.txt driving context shown to the model
├── gt.json accepted action pairs
└── meta.json obstacle type, category, position
gt.json is the only ground truth needed for scoring:
{"acceptable_pairs": [{"lon_action": "slow_down", "lat_action": "nudge_left"}]}
meta.json, abridged:
{"sample_id": "NP01_O10_center_v_full", "obstacle_type": "Pothole",
"obstacle_category": "road_hazard", "position": "center", "navigation": "GO STRAIGHT"}
hf download tangzx2024/CoLT-Drive --repo-type dataset --local-dir data
Inference and scoring instructions are in the code repository. Scoring uses an LLM judge; fix one judge model across every system you compare.
This is an image-level, meta-action benchmark: no physical rollout, no trajectory, no closed-loop feedback. Accepted-set sizes vary by category, so compare within a category rather than across.
Apache License 2.0. The evaluation code is released under the same license.
@inproceedings{tang2026coltdrive,
title = {CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction},
author = {Tang, Zhengxu and Cui, Guofeng and Gong, Ziyu and Zhang, Xiaozhou and Deng, Ruifeng and Qi, Chengzhi and Chen, Ke and Patil, Sachin and Xiao, Tianjun and Liu, Langechuan and Wang, Pichao},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026},
url = {https://arxiv.org/abs/2609.00242}
}
A counterfactual long-tail benchmark for driving decisions. The driving context (road geometry, ego pose, navigation command, motion history) is held fixed and only the object in the scene changes. The benchmark asks whether a model's decision updates when it should and holds when it should not.
Given a front-camera image and the driving context, output one longitudinal and one lateral action:
| Axis | Actions |
|---|---|
| Longitudinal | keep_speed, slow_down, yield, creep, stop, speed_up |
| Lateral | keep_lane, nudge_left, nudge_right |
Each sample has an accepted set of action pairs (1 to 14 pairs, 3.5 on average). A prediction is correct if it is in the set.
Both splits share identical target objects and road geometry and differ only in background traffic.
| Split | Samples | Background traffic |
|---|---|---|
vfull | 1,768 | naturalistic |
vclean | 1,768 | background vehicles removed |
50 obstacle types in five categories, grouped by the response they require:
| Category | Samples per split | Expected behaviour |
|---|---|---|
false_positive | 435 | preserve the nominal action |
entity_living | 348 | yield or stop |
road_hazard | 348 | adapt to the hazard |
entity_nonliving | 347 | slow down or steer around |
full_block | 290 | stop |
Object position is controlled separately: center (1,159), full_lane_blockage
(290), left_biased (145), right_biased (145), airborne (29).
vfull/
├── manifest.json sample ids in order
└── <sample_id>/
├── front_camera.jpg
├── prompt.txt driving context shown to the model
├── gt.json accepted action pairs
└── meta.json obstacle type, category, position
gt.json is the only ground truth needed for scoring:
{"acceptable_pairs": [{"lon_action": "slow_down", "lat_action": "nudge_left"}]}
meta.json, abridged:
{"sample_id": "NP01_O10_center_v_full", "obstacle_type": "Pothole",
"obstacle_category": "road_hazard", "position": "center", "navigation": "GO STRAIGHT"}
hf download tangzx2024/CoLT-Drive --repo-type dataset --local-dir data
Inference and scoring instructions are in the code repository. Scoring uses an LLM judge; fix one judge model across every system you compare.
This is an image-level, meta-action benchmark: no physical rollout, no trajectory, no closed-loop feedback. Accepted-set sizes vary by category, so compare within a category rather than across.
Apache License 2.0. The evaluation code is released under the same license.
@inproceedings{tang2026coltdrive,
title = {CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction},
author = {Tang, Zhengxu and Cui, Guofeng and Gong, Ziyu and Zhang, Xiaozhou and Deng, Ruifeng and Qi, Chengzhi and Chen, Ke and Patil, Sachin and Xiao, Tianjun and Liu, Langechuan and Wang, Pichao},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026},
url = {https://arxiv.org/abs/2609.00242}
}