ECE9660 — Alpamayo + vLLM inference (single vs multi-agent)

Author: Haoran Wang

This project runs batched inference over a CSV of clip_id values. Each SLURM array task starts a local vLLM server for Qwen3-VL (vision-language calls), then runs either:

• multiagent_alpamayo/single_agent_infer.py — per-clip pipeline: VLM agent counting and maneuver tagging via vLLM, plus Alpamayo-1.5 trajectory scoring (min_ade).
• multiagent_alpamayo/multi_agent_infer.py — multi-agent / Level-K reasoning over the same clips, using vLLM for long-form reasoning outputs.

Prerequisites

• Digital Alliance of Canada (or similar) cluster with SLURM and a GPU queue compatible.

• The minimum GPU VRAM requirement of a single inference is around 64GB

• A Hugging Face account and a read token with access to the models you use (e.g. nvidia/Alpamayo-1.5-10B, Qwen/Qwen3-VL-8B-Instruct). Export it before submitting jobs:

  export HF_TOKEN="<your_hf_read_token>"
  

• Dataset access: physical_ai_av and valid clip IDs (see multiagent_alpamayo/data/ and the default --clip-csv paths in the Python entrypoints).

Environment setup

Use two separate Python virtual environments: one for the vLLM server, one for the Alpamayo / dataset stack. The SLURM scripts assume:

1. vllm_infer/venv — created under vllm_infer/, used to run serve_vllm.py.
2. multiagent_alpamayo/venv — created under multiagent_alpamayo/, used to run the inference scripts.

1. Cluster modules

The job scripts load:

module purge
module load StdEnv/2023 python/3.13 cuda/12.9 opencv/4.13.0 arrow/22.0.0 ffmpeg/7.1.1

Create each venv with the same interpreter (example):

cd /path/to/ece9660/vllm_infer
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
# Such requirement has been carefully finteuned to run on Digital Canada
pip install -r DigitalCA_vllm_requirements.txt
deactivate

cd /path/to/ece9660/multiagent_alpamayo
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r digitalCA_requirements.txt
deactivate

2. Alpamayo source on PYTHONPATH

The inference scripts import alpamayo1_5 from multiagent_alpamayo/src. The SLURM jobs set:

export PYTHONPATH=$PWD/src:$PYTHONPATH

Run the same when testing interactively from multiagent_alpamayo/.

3. Logs and model cache directories

Before the first run, ensure log directories exist next to serve_vllm.py (paths used in the SLURM scripts):

mkdir -p vllm_infer/slurm_logs/vllm
mkdir -p vllm_infer/slurm_logs/single_am_infer

vLLM uses --download-dir "$SLURM_TMPDIR/models" so weights go to fast node-local storage.

Run with SLURM

From the directory where you keep the scripts (or use absolute paths):

export HF_TOKEN="<your_hf_read_token>"
cd /path/to/ece9660/vllm_infer/scripts

sbatch slurm_single.sh   # single-agent / Alpamayo + vLLM pipeline
sbatch slurm_multi.sh    # multi-agent Level-K inference

What each job does

1. Loads modules and activates vllm_infer venv.
2. Starts python serve_vllm.py --port 8123 --model "Qwen/Qwen3-VL-8B-Instruct" ... in the background and waits until http://127.0.0.1:8123/v1/models responds.
3. Switches to multiagent_alpamayo venv and runs the inference script with --vllm-port 8123, --row-start, --row-end, --output-csv, and --media-dir "$SLURM_TMPDIR/media".

Default output locations in the checked-in scripts:

• Single: results/second/single/sampled_batch_${ROW_START}_${ROW_END}.csv
• Multi: results/second/multi/levelk_batch_${ROW_START}_${ROW_END}.csv

Create results/... directories beforehand if your jobs do not auto-create them.

Analysis

All downstream analysis, plotting, and paper figures / tables live under analysis/:

• fig_overall.py, fig_dm.py, fig_agent.py — scripts that generate figures.
• paper_tables.ipynb — notebook for tabular results and table-style outputs.
• data.py — shared helpers for loading or preparing data used by the plots.

Run the Python scripts from the repo root (or adjust paths as needed) after inference CSVs are available.