LocLLM: Exploiting Road Lane Keypoints Localization via Large Language Model

---

Overview

LocLLM is a permutation-invariant lane detection system that fuses a DINOv3 vision encoder with a Qwen large language model backbone. A DETR-style decoder head predicts lane keypoints as normalized coordinates, enabling end-to-end training without relying on fixed lane ordering.

Key design choices:

• Permutation-invariant training via Hungarian matching — GT lanes are shuffled each epoch so the model cannot memorize slot order.
• Unified coordinate space — all coordinates are normalized to [0, 1] using original CULane dimensions (1640 × 590 px) as the single source of truth.
• Stability-first architecture — DINOv3's RoPE outputs are wrapped in a DINOv3Wrapper that clamps, NaN-replaces, and LayerNorms features before they reach the LLM.
• QLoRA + LoRA — supports 4-bit quantization of the LLM alongside LoRA adapters for both the LLM and vision encoder.
• Adaptive continuity loss — penalises only excess curvature in lane predictions beyond the GT's natural curvature, preventing over-smoothing of edge lanes.

---

Hardware & Training Environment

---

Architecture

Fixed Y-Samples

Lane keypoints are predicted at 43 fixed Y positions (CULane's h_samples):

h_samples = list(range(160, 590, 10))  # y = 160, 170, …, 580

Only the X coordinate is learned; Y is always fixed and known.

---

Repository Structure

LocLLM/
├── datasets/
│   ├── __init__.py
│   ├── constants.py          # Keypoint name/description tables
│   ├── conversation.py       # Chat templates (Simple, Qwen, LLaMA-2)
│   └── culane.py             # CULane dataset + CoordinateConverter + TokenizingCollator
│
├── models/
│   ├── __init__.py
│   ├── locllm.py             # Main model: LocLLMKeypointModel + DETRLaneDetector
│   ├── cross_attention.py    # GatedCrossAttention + CrossAttentionInjector
│   ├── dino.py               # DinoVisionTransformer (ViT-B/16 with RoPE)
│   ├── lora.py               # LoRA adapters for DINOv3 / LLaMA / CLIP
│   └── dino_layers/          # Building blocks: attention, block, mlp, rope, …
│
├── utils/
│   ├── train2d.py            # Training script (Trainer class)
│   ├── test2d.py             # Evaluation script (CULaneEvaluator, F1/Precision/Recall)
│   ├── inference.py          # Inference + visualization (LaneInference)
│   ├── evaluate_model.py     # Thin wrapper around test2d for quick eval
│   ├── culane_metric.py      # Official CULane IoU metric (discrete/continuous)
│   ├── diagnose_eval.py      # Per-sample evaluation diagnosis tool
│   ├── diagnose_vision.py    # Vision feature diversity checker
│   └── distributed/          # Distributed training helpers
│
├── scripts/
│   ├── train_culane.sh       # End-to-end training launcher
│   └── test_culane.sh        # Evaluation launcher (all 9 CULane categories)
│
└── debug_coordinates.py      # Standalone coordinate space debug script

---

Installation

# 1. Clone the repository
git clone <repo_url>
cd LocLLM

# 2. Create environment
conda create -n locllm python=3.10 -y
conda activate locllm

# 3. Install PyTorch (adjust CUDA version as needed)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# 4. Install dependencies
pip install transformers peft accelerate bitsandbytes
pip install scipy opencv-python Pillow shapely
pip install tqdm tensorboard

---

Dataset: CULane

Download CULane and organise as:

culane_dataset/
├── driver_23_30frame/
├── driver_100_30frame/
├── driver_161_90frame/
├── driver_182_30frame/
├── driver_193_90frame/
└── list/
    ├── train_gt.txt
    ├── val_gt.txt
    └── test_gt.txt

Each annotation .lines.txt file contains one lane per line with space-separated x y pairs in pixel coordinates.

---

Model Weights

Place DINOv3 weights at:

checkpoints/model_weights/dinov3_vitb16_pretrain_lvd1689m.pth

---

Training

Quick Start

bash scripts/train_culane.sh

Manual Launch

python utils/train2d.py \
    --llm_model /path/to/Qwen3-1.7B-Instruct \
    --vision_model_path /path/to/dinov3_vitb16_pretrain_lvd1689m.pth \
    --data_root /path/to/culane_dataset \
    --output_dir ./checkpoints/culane \
    --hidden_dim 2048 \
    --max_lanes 4 \
    --num_points_per_lane 43 \
    --num_queries 8 \
    --num_decoder_layers 6 \
    --use_lora \
    --lora_r 8 \
    --lora_alpha 16 \
    --use_float32 \
    --no_lm_loss \
    --batch_size 2 \
    --gradient_accumulation_steps 8 \
    --num_epochs 10 \
    --learning_rate 3e-4 \
    --freeze_vision_encoder \
    --shuffle_gt_lanes \
    --loss_weight_class 2.0 \
    --loss_weight_discrete 5.0 \
    --loss_weight_bezier 3.0

Key Training Flags

Memory Budget (RTX 5000, 16 GB)

---

Evaluation

Run Evaluation

python utils/test2d.py \
    --checkpoint ./checkpoints/culane/best \
    --data_root /path/to/culane_dataset \
    --output_dir ./eval_results \
    --split test \
    --batch_size 4 \
    --confidence_threshold 0.5 \
    --iou_threshold 0.5

All 9 CULane Categories

bash scripts/test_culane.sh ./checkpoints/culane/best

Categories: normal, crowd, hlight, shadow, noline, arrow, curve, cross, night.

Metrics

The evaluator follows the official CULane protocol:

• A predicted lane is a True Positive if its IoU with a GT lane exceeds 0.5.
• Hungarian matching ensures each GT lane is matched at most once.
• F1 Score is the primary reported metric.

---

Inference & Visualization

python utils/inference.py \
    --checkpoint_path ./checkpoints/culane/best \
    --image_folder /path/to/culane_dataset \
    --test_list /path/to/culane_dataset/list/test_gt.txt \
    --output_dir ./visualizations \
    --num_samples 100 \
    --iou_threshold 0.5

Green polylines = ground truth lanes. Red polylines = model predictions.

---

Coordinate System

All coordinates flow through a single CoordinateConverter instance.

Rule: GT annotations are parsed in original space, out-of-bounds points are dropped, then everything is normalized before being fed to the model.

---

Loss Components

---

Diagnostic Tools

# Check coordinate spaces for a single image
python debug_coordinates.py

# Evaluate prediction quality in detail (IoU per lane, template collapse detection)
python utils/diagnose_eval.py \
    --checkpoint ./checkpoints/culane/best \
    --data_root /path/to/culane_dataset \
    --output_dir ./diagnosis \
    --num_samples 30

# Verify vision features are diverse across images
python utils/diagnose_vision.py \
    --checkpoint_path ./checkpoints/culane/best \
    --image_folder /path/to/culane_dataset \
    --test_list /path/to/culane_dataset/list/test_gt.txt \
    --num_images 5

---

TODO

Handle illusions — improve robustness to optical illusions and ambiguous lane markings (e.g. shadow patterns, road artifacts, and painted markings that visually resemble lane lines but are not). This may require dedicated augmentation strategies, hard-negative mining, or a post-processing rejection module.

---

Acknowledgements

This codebase builds upon:

• Pink — referential comprehension for general multi-modal LLMs
• LLaVA — visual instruction tuning framework
• DINOv3 — self-supervised vision transformer backbone

---

License

MIT License — see LICENSE for full terms.