README
🎨 ColorBench
ColorBench is a multimodal dataset to comprehensively assess capabilities of VLMs in color understanding, including color perception, reasoning, and robustness, introduced in "ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness".
It provides:
- More than 5,800 image-text questions covering diverse application scenarios and practical challenges for VLMs evaluation.
- 3 categories and 11 tasks for various color-centric capabilities evaluation including Perception (Color Recognition, Color Extraction and Object Recognition), Reasoning (Color Proportion, Color Comparison, Color Counting, and more) and Robustness.
📃 Instruction
The data/test*.parquet files contain the dataset annotations and images pre-loaded for processing with HF Datasets.
from datasets import load_dataset
color_bench = load_dataset("umd-zhou-lab/ColorBench")
📂 Dataset Description
The dataset contains the following fields:
| Field Name | Description |
|---|---|
| idx | Global index of the sample in the dataset |
| id | Index of the sample in each task |
| type | Type of category: Perception, Reasoning, or Robustness |
| task | Type of task: Color Recognition, Color Extraction, Color Counting, and more |
| filename | Path to the image |
| image_url | Source of the image |
| prompt | Prompt with question and choices pre-formatted |
| question | Question about the image |
| choices | Answer choices for the question |
| answer | Correct answer to the question |
| image | Image object (PIL.Image) |
Contributors
10 commits
1 commits
Similar repos
README
🎨 ColorBench
ColorBench is a multimodal dataset to comprehensively assess capabilities of VLMs in color understanding, including color perception, reasoning, and robustness, introduced in "ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness".
It provides:
- More than 5,800 image-text questions covering diverse application scenarios and practical challenges for VLMs evaluation.
- 3 categories and 11 tasks for various color-centric capabilities evaluation including Perception (Color Recognition, Color Extraction and Object Recognition), Reasoning (Color Proportion, Color Comparison, Color Counting, and more) and Robustness.
📃 Instruction
The data/test*.parquet files contain the dataset annotations and images pre-loaded for processing with HF Datasets.
from datasets import load_dataset
color_bench = load_dataset("umd-zhou-lab/ColorBench")
📂 Dataset Description
The dataset contains the following fields:
| Field Name | Description |
|---|---|
| idx | Global index of the sample in the dataset |
| id | Index of the sample in each task |
| type | Type of category: Perception, Reasoning, or Robustness |
| task | Type of task: Color Recognition, Color Extraction, Color Counting, and more |
| filename | Path to the image |
| image_url | Source of the image |
| prompt | Prompt with question and choices pre-formatted |
| question | Question about the image |
| choices | Answer choices for the question |
| answer | Correct answer to the question |
| image | Image object (PIL.Image) |
Similar repos
Linked in READMEs
Contributors
10 commits
1 commits