GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation

---

Links:

🚩 News

• ✅ Aug. 18, 2024. 💥 GenAI-Bench-1600 is used by 🧨 Imagen 3 !
• ✅ Jun. 19, 2024. 💥 Our paper won the Best Paper award at the CVPR SynData4CV workshop !

Usage

# load the GenAI-Bench(GenAI-Bench-1600) benchmark
from datasets import load_dataset

dataset = load_dataset("BaiqiL/GenAI-Bench")

Citation Information

@article{li2024genai,
  title={Genai-bench: Evaluating and improving compositional text-to-visual generation},
  author={Li, Baiqi and Lin, Zhiqiu and Pathak, Deepak and Li, Jiayao and Fei, Yixin and Wu, Kewen and Ling, Tiffany and Xia, Xide and Zhang, Pengchuan and Neubig, Graham and others},
  journal={arXiv preprint arXiv:2406.13743},
  year={2024}
}

Description:

Our dataset consists of three parts: GenAI-Bench (Gen-Bench-1600), GenAI-Bench-Video, and GenAI-Bench-Ranking, with Gen-Bench-1600 being the primary dataset. For detailed processing methods of the above datasets of zip format, please refer to dataset.py in code.

GenAI-Bench benchmark (GenAI-Bench-1600) consists of 1,600 challenging real-world text prompts sourced from professional designers. Compared to benchmarks such as PartiPrompt and T2I-CompBench, GenAI-Bench captures a wider range of aspects in the compositional text-to-visual generation, ranging from basic (scene, attribute, relation) to advanced (counting, comparison, differentiation, logic). GenAI-Bench benchmark also collects human alignment ratings (1-to-5 Likert scales) on images and videos generated by ten leading models, such as Stable Diffusion, DALL-E 3, Midjourney v6, Pika v1, and Gen2.

GenAI-Bench:

• Prompt: 1600 prompts sourced from professional designers.
• Compositional Skill Tags: Multiple compositional tags for each prompt. The compositional skill tags are categorized into Basic Skill and Advanced Skill. For detailed definitions and examples, please refer to our paper.
• Images: Generated images are collected from DALLE_3, DeepFloyd_I_XL_v1, Midjourney_6, SDXL_2_1, SDXL_Base and SDXL_Turbo.
• Human Ratings: 1-to-5 Likert scale ratings for each image.

(Other Datasets: GenAI-Bench-Video | GenAI-Bench-Ranking)

Languages

English

Supported Tasks

Text-to-Visual Generation; Evaluation for Automated Evaluation Metrics.

Comparing GenAI-Bench to Existing Text-to-Visual Benchmarks

Dataset Structure

Data Instances

Dataset({
    features: ['Index', 'Prompt', 'Tags', 'HumanRatings', 'DALLE_3', 'DeepFloyd_I_XL_v1', 'Midjourney_6', 'SDXL_2_1', 'SDXL_Base', 'SDXL_Turbo'],
    num_rows: 1600
})

Data Fields

Statistics

(each prompt-image/video pair has three human ratings.)

Data Source

Prompts

All prompts are sourced from professional designers who use tools such as Midjourney and CIVITAI.

Multiple Compositional Tags for Prompts

All tags on each prompt are verified by human annotators.

Generated Images

Generating images using all 1,600 GenAI-Bench prompts from DALLE_3, DeepFloyd_I_XL_v1, Midjourney_6, SDXL_2_1, SDXL_Base and SDXL_Turbo.

Generated Videos

Generated Videos using all 800 GenAI-Bench prompts from Pika, Gen2, ModelScope and Floor33.

Human Ratings

We hired three trained human annotators to individually rate each generated image/video. We pay the local minimum wage of 12 dollars per hour for a total of about 800 annotator hours.

Dataset Construction

Overall Process

• Prompt Collecting: we source prompts from professional designers who use tools such as Midjourney and CIVITAI. This ensures the prompts encompass practical skills relevant to real-world applications and are free of subjective or inappropriate content.
• Compositional Skills Tagging: each GenAI-Bench prompt is carefully tagged with all its evaluated skills. We then generate images and videos using state-of-the-art models like SD-XL and Gen2. We follow the recommended annotation protocol to collect 1-to-5 Likert scale ratings for how well the generated visuals align with the input text prompts.
• Image/Video Collecting and Human Rating: we then generate images and videos using state-of-the-art models like SD-XL and Gen2. We follow the recommended annotation protocol to collect 1-to-5 Likert scale ratings for how well the generated visuals align with the input text prompts.

Leaderboard

Licensing Information

apache-2.0

Maintenance

We will continuously update the GenAI-Bench benchmark. If you have any questions about the dataset or notice any issues, please feel free to contact Baiqi Li or Zhiqiu Lin. Our team is committed to maintaining this dataset in the long run to ensure its quality!