ViVerBench: A Comprehensive Benchmark for Visual Verification

This repository contains ViVerBench, a comprehensive benchmark spanning 16 categories of critical tasks for evaluating visual outcomes in multimodal reasoning. ViVerBench was introduced in the paper Generative Universal Verifier as Multimodal Meta-Reasoner.

• Project Page: https://omniverifier.github.io/
• Code Repository: https://github.com/Cominclip/OmniVerifier
• Paper: https://huggingface.co/papers/2510.13804

Introduction

ViVerBench is designed to assess the fundamental capability of reflection and refinement on visual outcomes during the reasoning and generation process in vision-language models and unified multimodal models. It is part of the work on the Generative Universal Verifier, a novel concept aimed at enhancing next-generation multimodal reasoning.

The benchmark reveals that existing Vision-Language Models (VLMs) consistently underperform across its 16 categories of critical tasks, underscoring a substantial gap from human-level capability in reliable visual verification.

Sample Usage

To evaluate with ViVerBench, you first need to install the OmniVerifier package and store your model outputs in a JSON file.

Installation

git clone https://github.com/Cominclip/OmniVerifier.git
cd OmniVerifier
pip install -e .

ViVerBench Evaluation

We provide two evaluation approaches for ViVerBench: rule-based and model-based. As a first step, store your model outputs in a JSON file (e.g., your_model.json).

For rule-based evaluation:

python viverbench_eval_rule_based.py --model_response your_model.json

For model-based evaluation, which uses GPT-4.1 as the judge model:

python viverbench_eval_model_based.py --model_response your_model.json

Citation

If you use ViVerBench in your research, please cite the following paper:

@article{zhang2025generative,
  author  = {Zhang, Xinchen and Zhang, Xiaoying and Wu, Youbin and Cao, Yanbin and Zhang, Renrui and Chu, Ruihang and Yang, Ling and Yang, Yujiu},
  title   = {Generative Universal Verifier as Multimodal Meta-Reasoner},
  journal = {arXiv preprint arXiv:2510.13804},
  year    = {2025}
}