GenAI-Bench

Paper | 🤗 GenAI Arena | Github

Introduction

GenAI-Bench is a benchmark designed to benchmark MLLMs’s ability in judging the quality of AI generative contents by comparing with human preferences collected through our 🤗 GenAI-Arnea. In other words, we are evaluting the capabilities of existing MLLMs as a multimodal reward model, and in this view, GenAI-Bench is a reward-bench for multimodal generative models.

We filter existing votes collecte visa NSFW filter and other heuristics, and then finally resulting in 1735 votes for image generation, 919 votes for image editing, and 1069 votes for video generation, which is used to evaluate the performance of MLLMs on aligning with human preferences.

We adopts a pairwise comparison template for each tasks, where the model is asked to output 4 labels for each pair of AI generative contents, which are A>B, B>A, A=B=Good, A=B=Bad. We then calculate the average accuracy of the model by comparing the model's prediction with the human preference.

The prompt templates are shown below:

• Image Generation
• Image Editing
• Video Generation

Evaluate a new model

Please refer to our Github READMD: #evaluate-a-model

Contribute a new model

Please refer to our Github READMD: #contributing-a-new-model

Current Leaderboard (on test_v1 split)

(Updated on 2024-08-09)

Citation

@article{jiang2024genai,
  title={GenAI Arena: An Open Evaluation Platform for Generative Models},
  author={Jiang, Dongfu and Ku, Max and Li, Tianle and Ni, Yuansheng and Sun, Shizhuo and Fan, Rongqi and Chen, Wenhu},
  journal={arXiv preprint arXiv:2406.04485},
  year={2024}
}