OptiCarVis

Explainable automated-vehicle visualisations, rendered onto real dashcam footage.

OptiCarVis takes an ordinary dashcam clip and overlays what an automated vehicle would show a passenger to explain its behaviour: a future-path ribbon painted onto the road surface, and highlights on the road users it is reacting to, with distances. A vision-language model decides when an explanation is actually warranted, so the overlay appears only while it earns its place on screen.

---

Contents

• What it produces
• How it looks
• How it works
• What shapes the ribbon (the contract)
• Models used
• Requirements
• Getting started
• Running the pipeline
• Configuration
• Outputs
• Known limitations
• Further documentation

---

What it produces

Each render writes two videos from one perception pass — one with pedestrians and cyclists highlighted, one that additionally highlights vehicles — so the two conditions can be compared in a study without re-running the models.

The overlay is composed of:

How it looks

The ribbon is centred in the ego lane, not in the image

Assuming the vehicle sits at the image centre is wrong on a multi-lane road, and whenever it rides off-centre in its lane. Lane instances are detected and the ribbon is anchored between the two boundaries that actually straddle the vehicle.

Lane detection feeding the ego-lane centre

Detected lane polylines (green) and the ego-lane centre derived from the pair straddling the vehicle (orange). At intersections there are no longitudinal markings to follow, confidence collapses, and the ribbon falls back to straight — by design.

The ribbon follows a real turn

Through an intersection there are no lane lines to follow, and for a pre-recorded clip there is no plan to read. The turn is instead reconstructed from the clip's own future frames by visual odometry, and blended in only while a genuine turn is detected.

The look-ahead walks the reconstructed path by distance (30 m), not time — a 90° turn spans the same metres at any speed, so even a slow intersection turn taken at walking pace is anticipated and drawn. Below, a real ~80° left turn through an intersection at ~8 km/h: the bend appears as the turn enters the horizon and the ribbon sweeps left with the vehicle's actual path.

The overlay only appears when it is warranted

A vision-language model gates the explanation. Self-evident situations get a clean pass-through with no overlay at all; the visualisation is reserved for moments a passenger would actually want explained.

Occlusion and depth

Road segmentation clips the ribbon to the drivable surface, so it disappears behind pedestrians instead of being painted over them, and distance labels come from monocular depth aligned to the road plane.

---

How it works

flowchart TD
    A[Dashcam clip] --> B[src/alpamayo_stream.py<br/>per-timestep planner output]
    B --> C[src/gemma_gate_timeline.py<br/>VLM: explain, or stay clean?]
    A --> D[src/ego_trajectory.py<br/>future-frame visual odometry<br/>optional, for turns]
    D --> G[src/future_anchor.py<br/>homography chain to future frames<br/>anchors the path on the road]
    A --> E[final_preview_renderer.py]
    C -->|gate timeline| E
    D -->|future path| E
    G -->|street-pixel anchors| E
    E --> F[Two MP4s +<br/>workflow state record]

    subgraph P [per frame, inside the renderer]
        direction LR
        P1[YOLO26 + ByteTrack<br/>detect and track] --> P2[SegFormer<br/>road mask]
        P2 --> P3[UFLDv2<br/>lane instances]
        P3 --> P4[Depth Anything V2<br/>distances]
    end

    E -.-> P

Stages

1. Planner stream (alpamayo_stream.py) — produces a per-timestep action and reasoning trace from perception, standing in for a live planner.
2. Explanation gate (gemma_gate_timeline.py) — runs a multimodal VLM over sliding windows and decides, per window, whether an explanation is warranted. Deliberately conservative: the default answer is do not explain.
3. Ego trajectory (ego_trajectory.py, optional) — planar visual odometry that reconstructs the vehicle's real path from the clip's future frames, so the ribbon can follow a genuine turn.
4. Future anchors (future_anchor.py, optional) — the clip is pre-recorded, so the ground the car will occupy is visible in its later frames. This chains ground-plane homographies between frames and maps the ego's near-future position back into every earlier frame, yielding per-frame polylines of the street pixels the car actually drives over. See Anchoring the path.
5. Render (render_timeline_clip.py → final_preview_renderer.py) — builds the ribbon, highlights, occlusion and distance labels, animates the overlay on and off per the gate timeline, transcodes to H.264, and records the run.

What shapes the ribbon (the contract)

Worth reading before changing anything here, because a path overlay that points where the vehicle is not going is worse than no overlay at all.

How a frame becomes a ribbon — lane polylines, the straddle pick with its width gate, the tracked anchor projected as a ground line converging on the vanishing point, and the final composite:

A deliberate consequence: gentle curves render as near-straight. That is the chosen trade — never confidently point somewhere the vehicle is not going.

Anchoring the path to the road

The table above describes the projected sources: each reconstructs the future in world coordinates and pushes it through the flat-ground camera model. That works on lane-following roads and fails on real turns, for a reason no amount of tuning fixes. A ribbon expressed as a lateral offset over a fixed forward window, y(x_forward), cannot represent an arc that folds back — a 96-degree intersection turn reaches only ~12 m of forward distance and then curls sideways — and every modelling error (heading noise at pull-away, calibration drift, a road that is not flat) lands on top of that.

future_anchor.py removes the model from the placement entirely. The ground point ANCHOR_REF_AHEAD_M ahead of the camera sits at one fixed pixel in every frame, so the patch of road the car occupies at t+k can be carried back into frame t purely by image registration:

1. Per consecutive pair, features in a road band are tracked (LK optical flow) and a ground-plane homography is fitted with RANSAC — moving traffic does not move like the road plane, so it is rejected as outliers.
2. Chains reach across time by composing hops. Keyframe hops of OPTICARVIS_ANCHOR_KEYFRAME_STRIDE frames cut a long chain to ~n/stride matrices, which is what keeps drift small through fast yaw; the near field keeps dense per-frame hops so the band stays smooth.
3. The back-projected pixels become the frame's anchor polyline, each tagged with its arclength ahead of the ego (from pose deltas, which carry no heading bias).

The result is that the band's vertices are street pixels: heading error, calibration error and ground slope cancel by construction. A hop that loses RANSAC consensus (a truck sweeping across the camera does this) truncates the chain rather than inventing anchors, and the renderer falls back a rung.

Two implementation notes that are easy to get wrong. Rails must be offset perpendicular to the ground tangent and then reprojected — offsetting perpendicular in pixel space fans the band open into a wedge wherever the path runs sideways across the image, because on screen "perpendicular" is partly depth and has to foreshorten. And in planner mode the plan is drawn as a lateral offset from the driven path at matched arclength, applied along the anchors, so the planner ribbon inherits the anchors' correctness and its divergence from the human driver reads as sideways displacement on real asphalt.

Per-frame fallback ladder, top wins:

Choosing the cities

Which cities get filmed is a sampling decision, not a curation one, and src/city_sampler.py makes it one: cities are drawn with the Local Pivotal Method at probability proportional to population, so the sample spreads over the inhabited world, large countries get several cities without a one-per-country rule, and every selected city carries a design weight that makes country-level estimates defensible.

The sampler itself lives in its own package, lpm-sampling — it was written for this study, then extracted so it is useful and citable on its own. It is a normal pinned dependency here.

.venv/bin/python src/city_sampler.py --frame cities.csv --n 150 --seed 20260818 \
    --alpha 0.75 --footage-column footage_hours --min-footage-hours 1.0

Full rationale, references and the traps (eligibility must filter the frame before the draw; coordinates must be unit-sphere, never raw degrees) are in docs/CITY_SAMPLING.md.

Models used

Every checkpoint is named in one place — the Models block of src/pipeline_common.py — and every one is environment-overridable, so swapping a model never means editing the module that loads it.

Hugging Face weights are loaded with local_files_only=True so a long batch is never stalled by Hub metadata calls. On a machine that has not cached them yet, set OPTICARVIS_HF_LOCAL_FILES_ONLY=0 for the first run.

Swapping OPTICARVIS_ROAD_SEG_MODEL for a checkpoint trained on a different label map also means retuning ROAD_LABEL_IDS in src/scene_models.py — the default (0,) is the Cityscapes road class.

Swapping the planner model

The planner is not imported; it runs as a subprocess, so its checkpoint, CLI and Python environment are each swappable without touching this repo. That matters because a larger planner generally needs its own torch/transformers, which will not be the environment the rest of the pipeline runs in.

For example, pointing at nvidia/Alpamayo2-Super running from its own environment:

export OPTICARVIS_ALPAMAYO_PYTHON=/opt/alpamayo2/.venv/bin/python
export OPTICARVIS_ALPAMAYO_REPO=/opt/alpamayo2
export OPTICARVIS_ALPAMAYO_SCRIPT=/opt/alpamayo2/scripts/infer.py
export OPTICARVIS_ALPAMAYO_MODEL=nvidia/Alpamayo2-Super
export OPTICARVIS_ALPAMAYO_EXTRA_ARGS="--dtype bf16 --diffusion-steps 10"

The interpreter path is passed through verbatim, so a POSIX path works when the batch runner itself is driven from Windows.

Note — the wrapper controls how the planner is invoked, not what its CLI accepts. A backend whose flags differ from --clips / --output-dir / --config needs its own wrapper script; point OPTICARVIS_ALPAMAYO_SCRIPT at that. Check the backend's own hardware requirements before committing to it: Alpamayo2-Super is 34B and was profiled at ~72 GiB peak device memory on an H100 80 GB.

Requirements

• Python 3.12, uv
• An NVIDIA GPU is strongly recommended — the renderer runs four models per frame. Developed on an RTX 5080 (Blackwell, sm_120), which needs a CUDA 12.8 build of PyTorch. uv sync handles this; see below.
• ffmpeg on PATH for the H.264 delivery encode. Without it the render still completes and keeps the mp4v master, with a warning.

How PyTorch is resolved

PyPI's Windows torch wheel is CPU-only — its CUDA dependencies are gated on sys_platform == 'linux', so a plain PyPI install silently yields a CPU build and every CUDA kernel launch fails. pyproject.toml therefore routes Windows to the CUDA 12.8 index while leaving Linux on PyPI, whose wheels do bundle CUDA for both x86_64 and aarch64:

Nothing extra to install — uv sync --frozen is enough on both.

Linux is left on PyPI on purpose, and not because the CUDA 12.8 index lacks ARM builds — it does publish manylinux_2_28_aarch64 wheels. That is exactly what makes the alternative dangerous: pinning aarch64 to cu128 would resolve cleanly and look correct, while installing a CUDA 12.8 build. From torch 2.11.0 onward the PyPI Linux wheels are CUDA 13.0 builds, which is what an sm_121 part such as DGX Spark's GB10 wants. Sending Spark to the cu128 index would silently hand it kernels built for the wrong architecture generation.

Getting started

1. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

irm https://astral.sh/uv/install.ps1 | iex

2. Clone and sync

git clone https://github.com/Shaadalam9/opticarvis
cd opticarvis
uv python install 3.12.13
uv sync --frozen

3. Activate the environment

source .venv/bin/activate

.\.venv\Scripts\Activate.ps1

This installs the full pipeline, PyTorch included — see How PyTorch is resolved for what each platform gets.

Directory layout

Every path is derived from the repository root, so a fresh clone needs no path edits. Three directories are gitignored and not present after cloning:

opticarvis/
    src/                  pipeline code (tracked)
    videos/               source dashcam videos          <- you provide
    mapping.csv           clip index (tracked)
    external/             upstream checkouts             <- you clone, see below
        alpamayo/
        oom-free-alpamayo/
        UFLDv2/
    alpamayo_outputs/     extracted clips + planner JSON <- created by the pipeline
    workflow_outputs/     renders, timelines, state      <- created by the pipeline

alpamayo_outputs/ and workflow_outputs/ are created on demand — nothing to do. They are ignored because they hold multi-gigabyte video artefacts. videos/ is ignored for the same reason; the batch runner will fetch missing source videos over FTP when credentials are configured.

Fetching every model at once

.venv/bin/python scripts/setup_assets.py                 # download what is missing
.venv/bin/python scripts/setup_assets.py --check-only    # report, change nothing
.venv/bin/python scripts/setup_assets.py --with-planner  # also the ~67 GB planner checkpoint

This exists because the worst missing-asset failure is not a crash: without the UFLDv2 lane model the renderer silently falls back to a straight in-lane ribbon and a whole batch completes looking plausibly wrong. scripts/run_100_cities.sh runs the check before starting and refuses to launch with required assets missing. The sections below describe the same assets for manual setup.

External repositories

The pipeline shells out to two upstream checkouts and loads a third as a library. None are vendored — clone them into external/, which is where every default path looks:

mkdir -p external
git clone --depth 1 https://github.com/NVlabs/alpamayo.git external/alpamayo
git clone --depth 1 https://github.com/aveeslab/oom-free-alpamayo.git external/oom-free-alpamayo
git clone --depth 1 https://github.com/cfzd/Ultra-Fast-Lane-Detection-v2.git external/UFLDv2

Only batch_corrected_pipeline.py needs the two Alpamayo checkouts, and only to generate planner JSON; rendering from existing JSON does not. Each location is overridable — see Configuration.

Lane detection weights (optional). Ego-lane centring needs the UFLDv2 CULane checkpoint, which is ~865 MB and gitignored:

uv pip install gdown addict pathspec
python -c "import gdown; gdown.download(id='1AjnvAD3qmqt_dGPveZJsLZ1bOyWv62Yj', output='external/UFLDv2/culane_res34.pth')"

Without it the renderer still runs: it reports that the lane model is unavailable and keeps the ribbon straight ahead in the lane.

Running the pipeline

Build the explanation-gate timeline

python src/gemma_gate_timeline.py <clip.mp4> gate_timeline.json 6.0

Render — the tag becomes part of the output filename:

python src/render_timeline_clip.py <clip.mp4> gate_timeline.json lanecenter

Render with turn following — reconstruct the path first, then enable it:

python src/ego_trajectory.py <clip.mp4> vo_traj.json 4.0

OPTICARVIS_VO_TRAJECTORY=1 python src/render_timeline_clip.py <clip.mp4> gate_timeline.json hybrid "" vo_traj.json

Pass "" to skip an optional argument slot. Supplying a track file while its flag is unset is ignored with a warning, rather than silently.

Tune the camera calibration on a single still, with no video or models loaded:

python src/final_preview_renderer.py --calibrate frame.jpg calib.png

This draws the horizon, the vanishing point and metre distance ticks so HORIZON_V, VANISH_U, CAM_FOCAL_PX and CAM_HEIGHT_M can be set by eye.

Restyling a render without re-rendering

The cost of a render is the four models per frame, not the drawing — so the renderer dumps the geometry those models produced (per-frame ribbon centreline, selected detections, animation ramp, occlusion map) to workflow_outputs/overlay_geometry/ while it renders (OPTICARVIS_DUMP_GEOMETRY, on by default). Any overlay style can then be re-composited from that in seconds on CPU:

python src/restyle_render.py \
    --geometry workflow_outputs/overlay_geometry/<tag>_geometry.jsonl.gz \
    --style styles/default.json

Styles are per-element JSON (styles/default.json reproduces the built-in look): colour, opacity, blur/feather, stroke widths and visibility for the ribbon, chevrons, pedestrians, vehicles, distance labels and the explanation panel — plus parameters that shape geometry derived from the centreline, such as the ribbon width and chevron spacing/speed. Pixel values are at the 1280x720 calibration reference and scale with the clip exactly as the renderer's do.

The compositor calls the renderer's own drawing functions with the style values substituted, so the default style reproduces the shipped look by construction. What it cannot change is anything the models or temporal trackers decided — which objects are highlighted, where the ribbon goes, when the overlay is on. That is the point: style variants from identical geometry are clean experimental conditions.

Configuration

Every path and clip selection is environment-overridable, so rendering a second clip needs no code edits.

Model selection is listed separately under Models; the planner backend under Swapping the planner model.

Outputs

Written to <PROJECT_ROOT>/workflow_outputs/final_renders/, named after the clip that was rendered so clips cannot overwrite one another:

<clip>_roadline_v3_<tag>_h264.mp4            # pedestrians + cyclists highlighted
<clip>_roadline_v3_<tag>_vehicles_h264.mp4   # + vehicles highlighted

Each render also appends a record to the clip's workflow-state JSON with the inputs, frame counts, and the effective configuration (lane source, VO and look-ahead flags, thresholds, OPTICARVIS_* values) — so a shipped video can be traced back to the settings that produced it. Output paths that no longer exist are pruned from the state on each run.

Known limitations

• Curves need evidence. Gentle bends are followed only while both lane boundaries are visible (the curve fit decays to straight otherwise), and sharp turns only while visual odometry confidently detects them — see the contract. A bend with no visible paint and no VO confidence renders as straight, by design.
• Turn following is validated on real data at two speeds: a moderate curve at ~26 km/h and a sharp (~80°) intersection turn taken at ~8 km/h. The look-ahead is distance-based (30 m) precisely so that slow turns are not missed — a time-based window is blind to them.
• Visual odometry is monocular and planar. It assumes flat ground and the configured calibration; it estimates path shape well, not survey-grade motion.
• Calibration is per-camera. The defaults are tuned for one 1280×720 dashcam; re-tune with --calibrate for other footage or the ribbon will sit wrong. In particular the ribbon's direction is only correct if VANISH_U / HORIZON_V really are the road's vanishing point for your camera.

Further documentation

• docs/ENGINEERING.md — the measured evidence behind every geometry/tracking decision (with figures): why the ribbon is a straight ground line, why the anchor coasts through dropouts, the visual-odometry sign and look-ahead lessons, the jitter budget, and the measurement methods to reuse. Read it before changing the renderer.
• AGENTS.md — operating manual for coding agents (and new contributors): environment facts, repo map, render commands, and the invariants that must not be broken.

Licence

See LICENSE.