TTS for Small VLMs on Multilingual Visual MCQ

Code for the paper Test-Time Scaling for Small VLMs on Multilingual Visual MCQ.

🏆 1st place at the ImageCLEF 2026 Multimodal Reasoning shared task (Visual MCQ track).

We study test-time scaling (TTS) strategies for open-weight vision-language models (VLMs) on the EXAMS-V benchmark under a single-GPU, 7B-parameter budget.

Key Results

Three main findings:

1. The dominant scaling axis is per-chain token budget, not chain count: doubling the budget recovers +3.7 pp by eliminating truncation; doubling chains adds only +0.15 pp.
2. PRM-guided beam search underperforms flat self-consistency (-0.39 pp), with search collapsing to unanimous beams on 72% of questions.
3. Neither a generative critic nor a trained PRM beats majority vote, replicated across two policies.

Repository Structure

tts-small-vlm/
├── src/                    # Core pipeline modules
│   ├── backend_vllm.py     # vLLM inference backend
│   ├── backend.py          # HF Transformers backend (PRM model loading)
│   ├── describe.py         # Stage 1: N-sample image description
│   ├── reason.py           # Stage 2: M-sample text-only reasoning
│   ├── verify.py           # Stage 3: majority vote, generative critic, PRM
│   ├── search.py           # PRM-BAS beam-annealing search
│   ├── pipeline.py         # Pipeline orchestrator
│   └── utils/              # Data classes, answer extraction, logging, subsets
├── scripts/
│   ├── experiment.py       # Main experiment runner
│   ├── analyze.py          # Post-hoc analysis (stratified metrics, scaling curves)
│   ├── score_with_critic.py    # Generative critic rescoring (RQ4)
│   ├── score_with_prm.py      # Qwen-VL-PRM rescoring (RQ4)
│   ├── rescore_dtr_with_prm.py # DTR + PRM rescoring (Appendix)
│   ├── repair_parse_failures.py # Guided parse repair
│   ├── format_submission.py    # ImageCLEF competition submission formatting
│   ├── analyze_parse_fail.py   # Parse failure diagnostics
│   └── serve_vllm.sh          # vLLM server launcher
├── configs/                # YAML configs, one per experiment
│   ├── baselines/          # Zero-shot, CoT, SC (Q2.5 and Q3.5)
│   ├── scaling/            # Token budget, chain count, temperature sweep
│   ├── search/             # DTR, PRM-BAS, DTR+PRM
│   └── test/               # Test-set submission config
├── slurm/                  # SLURM job scripts (Snellius HPC)
├── eval/                   # Stratified evaluation script
├── paper/                  # Figure generation scripts
│   └── figures/            # plot_figures.py, generated PDFs
├── tests/                  # Unit tests (49 tests, fully mocked)
├── notebook.ipynb          # Pipeline walkthrough with search and verification
├── requirements.txt
├── .env.example            # Environment variable template
└── LICENSE

Setup

Requirements: Python 3.12+, CUDA 12.x, a GPU with at least 40 GB VRAM (A40/A100).

# Clone and install
git clone <repo-url> && cd tts-small-vlm
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Configure environment (optional, for W&B tracking)
cp .env.example .env
# Edit .env with your API keys

# Run unit tests (no GPU required)
pytest tests/ -v

Models (downloaded automatically on first use):

• Qwen2.5-VL-7B-Instruct (baselines, search experiments)
• Qwen3.5-4B (scaling, best configuration)
• Qwen-VL-PRM-7B (PRM-BAS search, PRM rescoring)

Reproducing Paper Results

Each experiment is driven by a YAML config and a SLURM job script. All scripts assume execution from the repository root.

Running an experiment locally

# Start vLLM server (in a separate terminal)
bash scripts/serve_vllm.sh

# Run a single experiment
python scripts/experiment.py --config configs/scaling/q35_sc_n8_2k.yaml --verbose

Running on a SLURM cluster

Edit the --account field in the SLURM scripts, then:

sbatch slurm/baselines_q25.sbatch      # Table 1, Q2.5 rows
sbatch slurm/baselines_q35.sbatch      # Table 1, Q3.5 rows
sbatch slurm/scaling_q35.sbatch        # Table 2, scaling rows
sbatch slurm/temp_sweep_q35.sbatch     # Table 2, temperature sweep
sbatch slurm/search_q25_dev.sbatch     # Table 3, search comparison
sbatch slurm/prm_bas_q35_val.sbatch    # Section 5.2, PRM-BAS at val scale
sbatch slurm/selectors_q25.sbatch      # Table 4, Q2.5 selectors
sbatch slurm/selectors_q35.sbatch      # Table 4, Q3.5 selectors
sbatch slurm/dtr_prm_rescore_dev.sbatch # Appendix Table 10
sbatch slurm/test_submission.sbatch     # Best config + guided repair on test

Paper result to config mapping

Post-hoc analysis and figures

# Stratified analysis of a completed run
python scripts/analyze.py --run-dir runs/<run_id>

# Generate paper figures (hardcoded data, no GPU needed)
python paper/figures/plot_figures.py

Notebook

notebook.ipynb walks through the full pipeline:

1. Baselines (zero-shot, chain-of-thought, self-consistency)
2. Search strategy (PRM-BAS beam-annealing search within a describe-then-reason scaffold)
3. Verification strategies (training-free generative critic, Qwen-VL-PRM discriminative rescoring)
4. Guided parse repair
5. Stratified evaluation

GPU-dependent cells are guarded with a RUN_INFERENCE flag for readability without hardware.

Data

The EXAMS-V dataset is loaded automatically from HuggingFace. It contains approximately 25,000 real school exam questions across 13 languages and 20 subjects. Our experiments use the validation split (4,651 questions) and a 200-question stratified subset for ablations.

License

This project is licensed under the MIT License. See LICENSE for details.