MeasL-Bench

MeasL-Bench (short for Measurement-grounded Language-Vision Benchmark) is the official held-out benchmark released with PRSIMVL for measurement-grounded vision-language evaluation.

It is designed to test a core question from the project: when RGB rendering loses sensor evidence, can a model grounded on measurement-domain input recover and reason more reliably?

Why MeasL-Bench

MeasL-Bench targets observation-interface failures, not only generic VQA difficulty. The benchmark emphasizes:

• low-illumination evidence recovery,
• HDR and exposure-sensitive grounding,
• visibility-sensitive and hallucination-sensitive queries,
• plus standard RGB-sufficient grounding slices for coverage.

This makes it suitable for evaluating both:

1. RGB-native VLMs, and
2. measurement-grounded methods (e.g., Meas.-XYZ pipelines).

Released Data

All image paths in the JSONL records use the release-local form eval_data/image/.... This matches the benchmark layout when this dataset directory is placed under a project root as eval_data/. When reading directly from the Hugging Face dataset repository root, resolve those paths by removing the leading eval_data/ prefix, e.g. eval_data/image/measl_bench_raw_000000.png maps to image/measl_bench_raw_000000.png.

Some local working copies may also contain legacy files named test_raw_full_benchmark.jsonl, test_rgb_full_benchmark.jsonl, and images/. These are local artifacts and are not part of the Hugging Face release.

Hugging Face Usage

from datasets import load_dataset

dataset = load_dataset(
    "kepeng/MeasL-Bench",
    data_files={
        "raw": "test-raw-measl-bench.jsonl",
        "rgb": "test-rgb-measl-bench.jsonl",
    },
)

Capability Taxonomy

MeasL-Bench is organized by capability dimensions rather than implementation buckets.

Total: 2,183 examples.

Evaluation Protocol

• Use identical question-answer records for RAW-side and RGB-side matched evaluation.
• Report lexical metrics (BLEU, ROUGE-L) and optional LLM-as-judge accuracy.
• Keep denominator fixed to 2,183 for overall reporting.

Official release entrypoint:

# dry-run
MODEL_SIZE=2b bash eval/run_infer_and_eval.sh --dry-run

# run RAW benchmark
MODEL_SIZE=2b CUDA_VISIBLE_DEVICES=0 bash eval/run_infer_and_eval.sh

# run RGB benchmark
DATASET=rgb MODEL_SIZE=2b CUDA_VISIBLE_DEVICES=0 bash eval/run_infer_and_eval.sh

Optional judge setup (OpenAI-compatible endpoint):

export JUDGE_API_KEY=YOUR_KEY
JUDGE_URL=https://openrouter.ai/api/v1 \
JUDGE_MODEL=openai/gpt-5 \
MODEL_SIZE=2b CUDA_VISIBLE_DEVICES=0 \
bash eval/run_infer_and_eval.sh

JSONL Format (Minimal)

Each row follows the release pipeline schema used by scripts/test_raw_eval_pipeline_opt/. Typical fields include:

• messages (conversation-style prompt, user question),
• images (one or multiple release-local image paths),
• answer / reference target,
• question_type (capability label in taxonomy above).

Recommended Benchmark Citation Name

When referring to this benchmark in docs/tables, use:

Allegory of the Cave: Measurement-Grounded Vision-Language Learning

@article{xu2026allegory,
  title   = {Allegory of the Cave: Measurement-Grounded Vision-Language Learning},
  author  = {Xu, Kepeng and Xu, Li and He, Gang and Yu, Wenxin},
  journal = {arXiv preprint},
  year    = {2026}
}