ViVQA-S1: System-One Typed Decision Architecture & Benchmarking for Vietnamese Visual Question Answering

ViVQA-S1 is an efficient System-One ($O(1)$ single forward pass) visual question answering framework and benchmark tailored for Vietnamese. Unlike standard Vision-Language Models (VLMs) that perform slow autoregressive decoding ($O(L)$ steps), ViVQA-S1 parametrizes VQA as a typed decision task—achieving high accuracy, low latency, and formal confidence calibration with selective fallback.

---

⚡ Key Highlights

• $O(1)$ Single Forward Pass: Decides the answer in a single forward pass without autoregressive token generation.
• Typed Decision Heads: Specialized decision interfaces conditioned on the question category:
  • 🔘 noul (Boolean / Verification): Binary cross-entropy with learned prior.
  • 🔢 score (Counting / Numeric): Ordinal regression and distribution matching ($0 \dots 10+$).
  • 🔠 choice (Candidate Selection): 16-way candidate matching powered by SetAttention and token-level Candidate-to-Visual Cross-Attention.
• Confidence Calibration & Selective Prediction: Post-hoc temperature scaling ($T_{\text{noul}}, T_{\text{score}}, T_{\text{choice}}$) and an escalation head that selectively defers uncertain questions to a heavy System-Two generative VLM.
• Reinforcement Learning from Candidate Distributions (RLCD): Dual-stage training with soft cross-entropy and policy-gradient fine-tuning.
• Multi-Modal Vision Backbones: Built-in support for SigLIP-2, BaMI (Vietnamese BERT), VinVL, and ViT-mBERT.

---

📁 Repository Structure

ViVQA_S1/
├── configs/                  # YAML configurations for all ViSO variants
│   ├── viso_siglip2_bami.yaml            # SigLIP-2 vision + BaMI text encoder
│   ├── viso_siglip2_bami_crossattn.yaml  # With Candidate Cross-Attention
│   ├── viso_flat_siglip2.yaml            # Flat classifier baseline (A0)
│   └── viso_vivqav2.yaml                 # Base ViVQA v2 benchmark config
├── models/                   # Neural network architectures
│   ├── viso/                             # ViSO core modules
│   │   ├── viso_model.py                 # Full ViSO model & FlatS1 wrapper
│   │   ├── encoder.py                    # Multimodal fusion & token unpooling
│   │   ├── heads.py                      # Noul, Score, Choice heads (+CrossAttn)
│   │   └── viso_ft.py                    # End-to-end fine-tuning module
│   ├── mcan.py                           # Modular Co-Attention baseline
│   ├── m4c.py                            # Multimodal Multi-Copy baseline
│   └── modules/                          # Attention, embeddings & fusion layers
├── tasks/                    # Task runners & training loops
│   ├── typed_decision_task.py            # TypedDecisionTask with RLCD & soft-CE
│   └── base_task.py                      # Abstract training task
├── scripts/                  # Training, evaluation & utility scripts
│   ├── train_viso_ft.py                  # End-to-end ViSO training script
│   ├── viso_probe.py                     # Controlled learning probes (400 steps)
│   ├── viso_calibrate.py                 # Post-hoc temperature calibration
│   ├── viso_predict.py                   # Model inference & prediction dumper
│   ├── viso_report.py                    # Multi-view evaluation reporter
│   ├── viso_pack_features.py             # High-speed float16 feature packer
│   └── kaggle_prepare.py                 # Kaggle dataset packaging utility
├── kaggle/                   # Automated Kaggle GPU experiment pipelines
│   ├── experiments.json                  # Experiment matrix (A0 - A6)
│   ├── experiments_siglip2.json          # SigLIP-2 experiment manifests
│   └── viso_matrix_kernel.py             # Multi-GPU Kaggle runner
├── data_utils/               # Tokenizers, vocabs, and feature collators
│   ├── datasets/s1_dataset.py            # PyTorch S1FeatureDataset & collator
│   ├── vocabs/s1_vocab.py                # S1 vocabulary and candidate mapper
│   └── vncorenlp/                        # VnCoreNLP Vietnamese toolkit
├── utils/                    # Calibration, packed features & RLCD helpers
│   ├── viso_calibration.py               # ECE, NLL, calibration optimization
│   ├── viso_packed_features.py           # Memory-mapped binary feature reader
│   └── viso_rlcd.py                      # RLCD loss computation
└── tests/viso/               # Pytest unit & regression test suite

---

📊 ViVQA-S1 Benchmark Dataset

The model is evaluated on ViVQA-S1, a clean, audited Vietnamese Visual Question Answering dataset with full type annotations:

Evaluation Views:

• All: Complete test split (comparable with standard ViVQA lineages).
• Core: Evaluated on verified, clean question-answer pairs.
• Image-Unseen: Strict generalization benchmark on images never seen during training or calibration.

---

🚀 Quickstart

1. Installation

Clone the repository and set up a virtual environment:

git clone https://github.com/Narcolepsyy/vivqa-s1.git
cd vivqa-s1

python3 -m venv .venv
source .venv/bin/activate
pip install -r <(echo "
torch>=2.0.0
transformers>=4.30.0
yacs>=0.1.8
pytest>=7.0.0
numpy
pyyaml
tqdm
tabulate
termcolor
pillow
")

2. Run Unit Tests

Verify that all modules, loss functions, and dataset pipelines pass:

pytest tests/viso -v

3. Training ViSO

To train a ViSO model with SigLIP-2 visual features and BaMI text representations:

python train_viso.py \
    --config-file configs/viso_siglip2_bami.yaml \
    --opts MODEL.DEVICE cuda \
           MODEL.NAME viso_siglip2_run \
           LOSS.NAME rlcd \
           TRAINING.EPOCHS_SUP 25

4. Calibration & Selective Prediction

Optimize post-hoc temperatures on the calibration set (calib) to minimize Expected Calibration Error (ECE):

# 1. Generate predictions and raw logits
python scripts/viso_predict.py \
    --config-file configs/viso_siglip2_bami.yaml \
    --opts MODEL.NAME viso_siglip2_run

# 2. Fit temperature scaling per question type
python scripts/viso_calibrate.py \
    --run-dir output/viso/viso_siglip2_run

5. Multi-View Benchmark Evaluation

Generate paired bootstrap confidence intervals and comprehensive metric reports:

python scripts/viso_report.py \
    --runs ViSO=output/viso/viso_siglip2_run:pertype \
    --split test \
    --out results_test.md

---

🔬 Model Architectures & ViSO Ablations

---

🏆 Benchmark Results on ViVQA-S1 (Test Split)

The table below presents the evaluation results on the ViVQA-S1 Test Split (4,409 items), comparing ViSO Architectures, System-One $\rightarrow$ System-Two Cascades, and External VLMs in System-One Mode (evaluated with standardized letter prompt and single-forward-pass letter-token logits).

Evaluation Views:

• All Acc: Overall accuracy on the full test set.
• Core Acc: Accuracy on clean, verified items.
• Image-Unseen Acc: Strict out-of-distribution evaluation on images never appearing in train or calibration splits.
• Per-type breakdown: noul (Boolean yes/no verification), score (Counting $0 \dots 10+$), choice (16-way candidate selection).

📌 Benchmark Insights

1. ViSO Efficiency & Latency Advantage: ViSO alone operates at an ultra-low inference latency of ~3 ms per question on CPU/Edge devices with only ~103M parameters—more than $10\times$ faster and $15\times$ smaller than sub-billion parameter VLMs, while comfortably outperforming the standard Flat classification baseline (+2.4% overall, +6.2% on counting).
2. Selective Escalation (System-One $\rightarrow$ System-Two): By capitalizing on ViSO's sharp calibration (ECE $\approx 0.020$), the escalation head identifies confident answers locally and defers ambiguous cases to System-Two (Qwen3.5). At 10% ViSO coverage, the cascade reaches 71.2% accuracy (99% of full VLM performance) with significant latency reduction on confident queries. At 50% coverage, accuracy reaches 61.3% while requiring only 0.64x of the VLM's compute budget.
3. Cross-Attention & Architectural Innovations: Introducing token-level Candidate-to-Visual Cross-Attention and Vietnamese BaMI BERT features pushes pure System-One accuracy to 46.2%, effectively mitigating visual feature pooling bottlenecks.
4. LoRA Fine-tuning: For full generative models, parameter-efficient LoRA tuning on Qwen3.5-0.8B yields a +16.1% leap over zero-shot to 72.1%, while Qwen3.5-2B achieves the ceiling at 75.7%.
5. Reference Teacher & "Student Beats Teacher": The Decision API Teacher (used during dataset construction and distillation) achieves 68.1% accuracy with an API latency of ~800 ms and higher calibration error (ECE 0.105). Both the fine-tuned on-device models (Qwen3.5-0.8B LoRA at 72.1%) and the ViSO Cascade (71.2% at 10% coverage) surpass the cloud teacher, demonstrating that task-specific structural alignment and post-hoc calibration yield superior results than relying on uncalibrated remote APIs.