ZoomBench: A Fine-Grained Multimodal Perception Benchmark

Overview

ZoomBench is a challenging benchmark designed to evaluate the fine-grained multimodal perception capabilities of Multimodal Large Language Models (MLLMs). It specifically targets scenarios where decisive visual evidence is small, subtle, or easily overwhelmed by global context — situations that demand "zooming-level" perception from a single full image.

It is introduced as part of the Zooming without Zooming project.

Key Statistics

Comparison with Existing Benchmarks

ZoomBench stands out with its hybrid R2I-based construction, automatic evidence annotation, high difficulty, and two unique evaluation protocols (Dual-View and Attention Map Interpretability).

Perceptual Dimensions

ZoomBench covers 6 major categories in fine-grained perception, identified by clustering the generated questions:

Evaluation Protocols

Hybrid Scoring Format

ZoomBench adopts a hybrid evaluation format:

• Multiple-choice questions (MCQ): For cases where answer spaces are naturally discrete or where strict normalization is difficult
• Open questions (OQ): With canonical target answers, for more natural and flexible evaluation

Dual-View Protocol

Each sample includes both a full image and its corresponding key-region crop. This enables quantifying the "zooming gap" — the performance difference between perceiving from the full image versus the cropped region:

Zooming Gap = Accuracy(Cropped View) − Accuracy(Full Image View)

A large zooming gap indicates the model struggles to extract fine-grained details from global context, while a small gap suggests robust fine-grained perception capability. This protocol provides a direct measurement of whether a model has internalized the ability to "zoom" without actually zooming.

Attention Map Interpretability

ZoomBench supports Attention Map Coverage evaluation, which measures whether a model's visual attention is grounded on the task-relevant image region. This provides an interpretability perspective: even when a model answers correctly, low attention coverage on the target region may indicate reliance on shortcuts rather than genuine fine-grained understanding. This analysis interpretably demonstrates whether a model has internalized the zooming ability.

Construction Pipeline

ZoomBench repurposes the Region-to-Image Distillation method for benchmark construction:

1. Region Cropping: Micro-regions are cropped from high-resolution images sourced from diverse image datasets
2. Question & Answer Generation: A powerful MLLM (Gemini-2.5-Pro) proposes questions and candidate answers based on the cropped regions
3. Full-Image Mapping: Questions are mapped back to full images to form challenging perception tasks without any spatial grounding (e.g., bounding boxes). The cropped regions automatically serve as evidence annotations
4. Human-in-the-Loop Verification: Human annotators check the validity, difficulty, and correctness of model-generated QA pairs against both full images and cropped regions. This hybrid approach significantly reduces the burden of from-scratch labeling while maintaining high quality
5. Dimension Clustering: Questions are clustered into 6 fine-grained perception categories

Usage

Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("inclusionAI/ZoomBench")

Evaluation

We provide evaluation scripts in the main project repository:

git clone https://github.com/inclusionAI/Zooming-without-Zooming.git
cd Zooming-without-Zooming

# Convert ZoomBench to evaluation format
cd utils
python convert_benchmark.py

# Run benchmark evaluation
cd ../mm-eval
bash run_baseline.sh

# Attention Map Coverage analysis
cd ../utils
python eval_coverage.py

Related Resources

Citation

@article{wei2026zooming,
  title={Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception},
  author={Wei, Lai and He, Liangbo and Lan, Jun and Dong, Lingzhong and Cai, Yutong and Li, Siyuan and Zhu, Huijia and Wang, Weiqiang and Kong, Linghe and Wang, Yue and Zhang, Zhuosheng and Huang, Weiran},
  journal={arXiv preprint arXiv:2602.11858},
  year={2026}
}

License

Please refer to the project repository for license details. ZoomBench is intended for research and evaluation purposes.

Contact

For questions or issues regarding ZoomBench, please contact:

• Lai Wei: waltonfuture@sjtu.edu.cn
• Liangbo He: langdao.hlb@antgroup.com
• Jun Lan: yelan.lj@antgroup.com
• Zhuosheng Zhang: zhangzs@sjtu.edu.cn
• Weiran Huang: weiran.huang@sjtu.edu.cn