Counterfactual VLM Benchmark

A counterfactual vision-language model benchmark for testing whether VLMs answer from visible evidence or from language and category priors, and whether agentic vision tools help reduce that bias.

Code: github.com/JingyuSunUOE/counterfactual-vlm-benchmark

Data: huggingface.co/datasets/JingyuSun/counterfactual-vlm-benchmark-data

This repository contains evaluation code, question definitions, metadata schemas, visual-evidence tooling, and analysis utilities for five benchmark domains:

• if_exist: expected object parts are removed.
• counting: countable body parts are added or removed.
• fashion: logo and monogram layouts are altered.
• industry: common object colors, patterns, orders, and configurations are altered.
• medical_modality: tumor-region MRI modality is swapped against the surrounding sequence.

The project supports closed-source VLM APIs and open-source VLMs served through an OpenAI-compatible server. It also includes checkpointed evaluation, resume support, structured-output scoring, result aggregation, and agentic vision evidence generation.

Why This Benchmark Exists

Many VLMs answer visual questions using what they expect to see. For example, if a camel image has its hump removed, a model may still answer as if the hump is present because camels are strongly associated with humped bodies.

This benchmark studies that failure mode with paired original and counterfactual images. It also evaluates a newer question: can extra visual views, such as bounding boxes, crops, zoom panels, outlines, or medical contours, help models rely more on visual evidence and less on priors?

The goal is not to propose a new agentic vision method. The goal is to evaluate which visual tool conditions help which models under controlled counterfactual settings.

Benchmark Overview

Counts below refer to the current active metadata in this repository. Originals counts unique original images; CF pairs counts counterfactual metadata records.

Medical data is treated as a separate domain and is not stored under dataset/ or cf_dataset/. Formal medical experiments use the clean BraTS2023 modality-swap pool and typically run on a stratified 10% sample.

Task Format

Each non-medical benchmark uses paired images:

• original: the unmodified image.
• counterfactual: an edited image where a salient expected feature or visual property has been changed.

The evaluation supports three input modes:

The evaluation uses three question types:

Model answers are mapped to three primary labels:

Errors such as API failures, quota stops, empty responses after retry, and runtime failures are tracked separately.

Agentic Vision Tool Conditions

Tool-condition experiments evaluate whether extra visual views help reduce prior-driven answers.

Main tool experiments do not use part prompts such as wing, finger, or toe, because they would directly reveal the target attribute. Medical evidence is derived from BraTS segmentation labels rather than SAM, because natural-image segmentation models are not appropriate for MRI tumor localization.

Repository Layout

.
  dataset/                  Downloaded original images for non-medical benchmarks
  cf_dataset/               Downloaded counterfactual images plus tracked question JSON files
  medical/                  Medical-domain workspace; raw BraTS data is not tracked
  vision_dataset/           Downloaded generated visual evidence; not tracked by Git
  vision_configs/           SAM prompt configs for evidence generation
  gen_code/                 Dataset preparation and evidence generation scripts
  eval_code/                Evaluation runners, analysis, and server wrapper
  eval_results/             Downloaded metadata plus local/generated reports, raw runs, tables, and figures
  scripts/                  Hugging Face data upload/download helpers
  test/                     Non-live schema, CLI, and analysis tests

For GitHub distribution, large image assets, visual evidence, reports, raw API runs, generated figures, and restricted medical source data are excluded from Git. The Hugging Face Dataset release provides the image payload, generated visual evidence, and lightweight eval_results/*/metadata/ runtime artifacts needed by the evaluation scripts.

Installation

Create an environment and install the project in editable mode. This is enough for closed-model API evaluation, metadata utilities, analysis, and most non-live tests:

pip install -e .

For local open-source VLM evaluation through eval_code/run_server_eval.py, use a Linux NVIDIA GPU environment with a CUDA runtime that matches the PyTorch and vLLM wheels. The tested cluster configuration is:

• Linux x86_64 with NVIDIA GPUs; H100 80GB is the primary tested target.
• NVIDIA driver new enough for CUDA 12.8 runtime images.
• Python 3.12 for the CUDA server environment.
• PyTorch CUDA 12.x wheels and vLLM installed with the same CUDA backend.

Do not install vLLM with an unconstrained pip install vllm in this environment. Recent wheels may resolve to a CUDA 13 runtime and fail at server startup with errors such as libcudart.so.13: cannot open shared object file. Qwen3-VL requires vllm>=0.11.0; for clusters with CUDA 12.8-capable drivers, pin vLLM and install with an explicit CUDA 12.8 backend:

python -m pip install --upgrade uv
UV_TORCH_BACKEND=cu128 uv pip install -e .

If you build your own container manually, use a CUDA 12.8 base image or equivalent and install vLLM with the same backend:

uv pip install \
  "vllm==0.11.1" \
  --torch-backend=cu128 \
  --extra-index-url "https://download.pytorch.org/whl/cu128" \
  --index-strategy unsafe-best-match

Verify the serving stack after starting the container with NVIDIA runtime. The vllm._C import requires libcuda.so.1, which is provided by the host driver at docker run --gpus all time and is not available during a plain docker build:

python - <<'PY'
import torch
import vllm
import vllm._C
print("torch", torch.__version__, "cuda", torch.version.cuda)
print("vllm CUDA extension ok")
PY

Closed-model evaluation requires provider API keys, usually configured through environment variables or a local .env file:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...

Do not commit .env files.

Download Data

The GitHub repository does not include image data, generated visual evidence, or local evaluation outputs. The data payload is hosted on Hugging Face.

For small machines, clusters with shared filesystems, or unstable network sessions, download by category instead of pulling the full payload in one command. This avoids long waits on Hugging Face file locks when downloading many small visual-evidence files.

Recommended first step: download core images, question files, metadata, and medical standardized images, but skip generated visual evidence:

python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --exclude "vision_dataset/**"

Then download only the visual-evidence directories needed for your experiment:

# If-exist tool-condition evidence
python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --include "vision_dataset/if_exist/**"

# Counting tool-condition evidence
python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --include "vision_dataset/counting/**"

# Fashion tool-condition evidence
python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --include "vision_dataset/fashion/**"

# Industry tool-condition evidence
python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --include "vision_dataset/industry/**"

# Medical label-derived evidence
python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir . \
  --include "vision_dataset/medical_modality/**"

If you prefer a single full download, run:

python scripts/download_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --local-dir .

If a download is interrupted, rerun the same command. Completed files are reused. If the Hugging Face CLI repeatedly prints Still waiting to acquire lock, stop duplicate download processes and remove stale lock files only after confirming no download is still running.

After download, the expected layout is:

dataset/
cf_dataset/
vision_dataset/
medical/modality_swapping/medical_modality_questions.json
medical/modality_swapping/standardized/      # if included in the HF release
eval_results/if_exist/metadata/
eval_results/counting/metadata/
eval_results/fashion_industry/metadata/
eval_results/medical_modality/metadata/

The HF data package includes lightweight metadata and evidence manifests so the default evaluator paths resolve after download. It intentionally does not include raw model responses, provider caches, reports, tables, or figures.

Publish Data to Hugging Face

To publish the current local data payload to a private Hugging Face Dataset repo:

curl -LsSf https://hf.co/cli/install.sh | bash -s

python scripts/upload_hf_dataset.py \
  --repo-id JingyuSun/counterfactual-vlm-benchmark-data \
  --private \
  --overwrite-staging

The upload script reads HF_TOKEN from .env, builds hf_dataset_release/, writes a dataset card and MANIFEST.json, then calls hf upload-large-folder with progress output. It includes eval_results/*/metadata/ but excludes .env, raw runs, reports, tables, figures, raw BraTS NIfTI data, local environments, and caches. Inspect the private HF repo before making it public.

Quick Start

Closed-model dry run on if_exist:

python eval_code/if_exist/eval_closed_vlm.py \
  --input-mode cf_only \
  --backbone-model gpt-5 \
  --judge-model gpt-4o-mini \
  --tool-condition raw \
  --question-types all \
  --source-variant both \
  --prime off \
  --max-samples 2 \
  --dry-run

Closed-model tool-condition dry run:

python eval_code/if_exist/eval_closed_vlm.py \
  --input-mode cf_only \
  --backbone-model gpt-5 \
  --judge-model gpt-4o-mini \
  --tool-condition crop \
  --evidence-manifest eval_results/if_exist/metadata/if_exist_sam3_object_evidence_manifest.json \
  --evidence-qc-filter pass_review \
  --missing-evidence-policy skip \
  --image-group-labels on \
  --question-types all \
  --source-variant both \
  --prime off \
  --max-samples 2 \
  --dry-run

Open-source VLMs are evaluated through an OpenAI-compatible server. The wrapper can launch local backbone and judge servers and automatically plan GPU placement:

python eval_code/run_server_eval.py \
  --framework vllm \
  --model Qwen/Qwen3-VL-8B-Instruct \
  --served-model-name qwen3vl8b \
  --judge-server-model Qwen/Qwen3-8B \
  --judge-served-model-name qwen3-8b \
  --benchmark if_exist \
  --gpu-placement-policy auto \
  --dry-run \
  -- \
  --input-mode cf_only \
  --backbone-model qwen3vl8b \
  --judge-model qwen3-8b \
  --question-types all \
  --max-samples 1 \
  --dry-run

Reproducibility Workflow

The public repository is organized around a lightweight, reproducible workflow:

1. Download the image and visual-evidence payload from the Hugging Face Dataset release.
2. Install the benchmark dependencies with pip install -e ..
3. Use the downloaded metadata artifacts as the source of truth for default evaluator paths.
4. Run closed-model or OpenAI-compatible server evaluations from eval_code/.
5. Aggregate results with the benchmark-specific analysis scripts.

Internal planning notes, cluster job files, and machine-specific Docker configurations are not part of the public GitHub release. This keeps the repository focused on reusable benchmark code, question definitions, data download utilities, and documentation.

Data and Metadata

Metadata files are the source of truth for benchmark records and evidence manifests. They are not committed to GitHub, but they are included in the Hugging Face Dataset payload so downloaded data can be used directly by the evaluation scripts. Do not infer formal evaluation splits by scanning directories.

See dataset/README.md for original-image data and cf_dataset/README.md for counterfactual-image data. If the metadata paths above are missing after a fresh clone, run python scripts/download_hf_dataset.py --repo-id JingyuSun/counterfactual-vlm-benchmark-data --local-dir ..

Evaluation Metrics

Primary metrics are computed over non-error responses:

• Accuracy: percentage of correct responses.
• Bias Rate: percentage of biased responses.
• Other Rate: percentage of other responses.
• Error Count: API, parsing, quota, judge, or runtime failures.

For tool experiments, compare each tool condition against the matching raw baseline:

• Delta Accuracy = Tool Accuracy - Raw Accuracy
• Delta Bias = Tool Bias Rate - Raw Bias Rate
• Delta Other = Tool Other Rate - Raw Other Rate

Raw-vs-tool comparisons should use the same model, benchmark, input mode, question type, source variant, QC filter, and sample subset.

Validation

Useful non-live checks:

python -m py_compile \
  eval_code/if_exist/eval_closed_vlm.py \
  eval_code/counting/eval_closed_vlm.py \
  eval_code/fashion_industry/eval_pipeline.py \
  eval_code/medical_modality/eval_closed_vlm.py \
  segmentation_sam3.py

python test/if_exist/test_eval_cli.py
python test/counting/test_eval_cli.py
python test/fashion_industry/test_eval_cli.py
python test/medical_modality/test_eval_cli.py
python test/vision_tools/test_segmentation_sam3_cli.py

Release Notes for GitHub

Before publishing:

• Keep .env, .venv/, .idea/, provider caches, and raw API outputs out of Git.
• Do not commit restricted BraTS source data or NIfTI files.
• Do not commit generated vision_dataset/ evidence, image data, metadata, or reports to GitHub under the current release policy.
• Publish large image assets through the Hugging Face Dataset release helper, or use Zenodo/Git LFS only with clear licensing.
• Medical generated images should only be published after confirming source-data license constraints; medical/BraTS2023_GLI/ must remain excluded.

License and Citation

License and citation details should be finalized before public release.