---

license: cc-by-nc-4.0 task_categories:

• visual-question-answering language:
• en pretty_name: SEED-Bench-H size_categories:
• 1K<n<10K

---

SEED-Bench-H Card

Benchmark details

Benchmark type: SEED-Bench-H is a large-scale benchmark to evaluate Multimodal Large Language Models (MLLMs). It consists of 28K multiple-choice questions with precise human annotations, spanning 34 dimensions, including the evaluation of both text and image generation.

Benchmark date: SEED-Bench-H was collected in April 2024.

Paper or resources for more information: https://github.com/AILab-CVC/SEED-Bench

License: Attribution-NonCommercial 4.0 International. It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use.

Data Sources:

• Dimensions 1-9, 23 (In-Context Captioning): Conceptual Captions Dataset (https://ai.google.com/research/ConceptualCaptions/) under its license (https://github.com/google-research-datasets/conceptual-captions/blob/master/LICENSE). Copyright belongs to the original dataset owner.
• Dimension 9 (Text Recognition): ICDAR2003 (http://www.imglab.org/db/index.html), ICDAR2013(https://rrc.cvc.uab.es/?ch=2), IIIT5k(https://cvit.iiit.ac.in/research/projects/cvit-projects/the-iiit-5k-word-dataset), and SVT(http://vision.ucsd.edu/~kai/svt/). Copyright belongs to the original dataset owner.
• Dimension 10 (Celebrity Recognition): MME (https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation) and MMBench (https://github.com/open-compass/MMBench) under MMBench license (https://github.com/open-compass/MMBench/blob/main/LICENSE). Copyright belongs to the original dataset owners.
• Dimension 11 (Landmark Recognition): Google Landmark Dataset v2 (https://github.com/cvdfoundation/google-landmark) under CC-BY licenses without ND restrictions.
• Dimension 12 (Chart Understanding): PlotQA (https://github.com/NiteshMethani/PlotQA) under its license (https://github.com/NiteshMethani/PlotQA/blob/master/LICENSE).
• Dimension 13 (Visual Referring Expression): VCR (http://visualcommonsense.com) under its license (http://visualcommonsense.com/license/).
• Dimension 14 (Science Knowledge): ScienceQA (https://github.com/lupantech/ScienceQA) under its license (https://github.com/lupantech/ScienceQA/blob/main/LICENSE-DATA).
• Dimension 15 (Emotion Recognition): FER2013 (https://www.kaggle.com/competitions/challenges-in-representation-learning-facial-expression-recognition-challenge/data) under its license (https://www.kaggle.com/competitions/challenges-in-representation-learning-facial-expression-recognition-challenge/rules#7-competition-data).
• Dimension 16 (Visual Mathematics): MME (https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation) and data from the internet under CC-BY licenses.
• Dimension 17 (Difference Spotting): MIMICIT (https://github.com/Luodian/Otter/blob/main/mimic-it/README.md) under its license (https://github.com/Luodian/Otter/tree/main/mimic-it#eggs).
• Dimension 18 (Meme Comprehension): Data from the internet under CC-BY licenses.
• Dimension 19 (Global Video Understanding): Charades (https://prior.allenai.org/projects/charades) under its license (https://prior.allenai.org/projects/data/charades/license.txt). SEED-Bench-2 provides 8 frames per video.
• Dimensions 20-22 (Action Recognition, Action Prediction, Procedure Understanding): Something-Something v2 (https://developer.qualcomm.com/software/ai-datasets/something-something), Epic-Kitchen 100 (https://epic-kitchens.github.io/2023), and Breakfast (https://serre-lab.clps.brown.edu/resource/breakfast-actions-dataset/). SEED-Bench-2 provides 8 frames per video.
• Dimension 24 (Interleaved Image-Text Analysis): Data from the internet under CC-BY licenses.
• Dimension 25 (Text-to-Image Generation): CC-500 (https://github.com/weixi-feng/Structured-Diffusion-Guidance) and ABC-6k (https://github.com/weixi-feng/Structured-Diffusion-Guidance) under their license (https://github.com/weixi-feng/Structured-Diffusion-Guidance/blob/master/LICENSE), with images generated by Stable-Diffusion-XL (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) under its license (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md).
• Dimension 26 (Next Image Prediction): Epic-Kitchen 100 (https://epic-kitchens.github.io/2023) under its license (https://creativecommons.org/licenses/by-nc/4.0/).
• Dimension 27 (Text-Image Creation): Data from the internet under CC-BY licenses.
• Dimension 28 (Few-shot Segmentation): MSCOCO dataset (https://cocodataset.org/) under its licenses (https://creativecommons.org/licenses/by/4.0/legalcode).
• Dimension 29 (Few-shot Kyepoint): MSCOCO dataset (https://cocodataset.org/) under its licenses (https://creativecommons.org/licenses/by/4.0/legalcode).
• Dimension 30 (Few-shot Depth): Middlebury stereo dataset (https://vision.middlebury.edu/stereo/) under CC-BY licenses.
• Dimension 31 (Few-shot Object): MSCOCO dataset (https://cocodataset.org/) under its licenses (https://creativecommons.org/licenses/by/4.0/legalcode).
• Dimension 32 (Image to Latex): Im2Latex dataset (https://lstmvis.vizhub.ai/) under its licenses (https://github.com/HendrikStrobelt/LSTMVis/blob/master/LICENSE.md).
• Dimension 33 (Text-Rich Visual Comprehension): Data from the internet under CC-BY licenses.

Where to send questions or comments about the benchmark: https://github.com/AILab-CVC/SEED-Bench/issues

Intended use

Primary intended uses: The primary use of SEED-Bench-H is evaluate Multimodal Large Language Models in text and image generation tasks.

Primary intended users: The primary intended users of the Benchmark are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.