📚 WorldGenBench

A World-Knowledge-Integrated Benchmark/Dataset for Reasoning-Driven Text-to-Image Generation

---

Paper Link: https://huggingface.co/papers/2505.01490

---

Purpose and Scope

WorldGenBench is designed to systematically evaluate text-to-image (T2I) models on two critical but underexplored capabilities:

• World Knowledge Grounding: Understanding background facts assumed but not explicitly stated in prompts.
• Implicit Reasoning: Inferring necessary visual elements beyond direct text description to generate contextually coherent images.

Unlike traditional datasets that emphasize simple alignment or aesthetic quality, WorldGenBench probes whether models can generate factually consistent and semantically rich images when prompts demand historical, scientific, or commonsense knowledge.

WorldGenBench covers:

• Humanities Domain: History, culture, geography prompts from 244 countries/regions.
• Nature Domain: Scientific prompts spanning Astronomy, Physics, Earth Sciences, Chemistry, Biology & Medicine, and Cross-disciplinary fields.

WorldGenBench can also be used to train a reasoning T2I model that can improve the world knowledge and reasoning ability of the model.

---

Dataset Structure/Usage

Prompt: natural-language input describing a grounded world scenario
Knowledge Checklist: binary-labeled factual criteria (✓/✗)

Use in Evaluation / Reward Modeling

A model-generated image is assessed against the Checklist
GPT-4o labels each checklist item as satisfied (1) or not (0)
Reward = (# satisfied) / (total checklist items) ∈ [0,1]

---

Dataset Creation Method

• Prompt Generation: We used a large language model (Claude 3.7) to generate a diverse set of evaluation prompts:
  • 732 prompts for Humanities (3 per country/region)
  • 340 prompts for Nature across 6 scientific disciplines
  • 
• Knowledge Checklists: Each prompt is paired with a knowledge checklist—a structured set of factual and inferential requirements the generated image must satisfy.
• Validation: All prompts and checklists underwent human verification for factual correctness and logical consistency.
• Evaluation Protocol: For each generated image, we compute the Knowledge Checklist Score—the fraction of checklist items satisfied, evaluated by a state-of-the-art vision-language model (GPT-4o) under strict criteria.

---

Example Uses

• Benchmarking T2I Models: Assess new or existing models' ability to handle knowledge-intensive or reasoning-heavy prompts.

  

• Fine-tuning for Robustness: Use WorldGenBench prompts for model fine-tuning to improve factual grounding and reasoning performance. Especially in RL methods.

• Error Analysis: Analyze failure modes where models hallucinate, miss unstated elements, or show reasoning gaps.

• Comparative Studies: Compare diffusion-based and autoregressive T2I architectures in terms of world knowledge integration.

---

Viewer Preview

The dataset is formatted for full compatibility with the Hugging Face Dataset Viewer:

• Each example includes:
  • A text prompt
  • A knowledge checklist (JSON format)
• All examples are self-contained and human-readable.

---

Acknowledgment

We thank the broader community for inspiring rigorous evaluations beyond surface-level generation.