Benchmark.ai Image Benchmark

Built by Rapidata for Benchmark.ai.

This dataset contains 6,123,853 human responses, collected with the Rapidata Python SDK, comparing 43 text-to-image models on 1,514 prompts. Each row is a head-to-head comparison between two models' images for the same prompt, judged by human annotators on up to four questions — Alignment, Preference, Coherence and Text coherence.

Every image is ranked purely by human judgement, no automated metrics. This is the data behind the Benchmark.ai image leaderboard.

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

To evaluate your own models and create a leaderboard, check out our MRI.

At a glance

The leaderboards

Every model is compared on the same kind of image pairs across four independent questions, run as four leaderboards on one Rapidata benchmark:

.vertical-container { display: flex; flex-direction: column; gap: 60px; } .horizontal-container { display: flex; flex-direction: row; justify-content: center; gap: 60px; } .image-container img { max-height: 300px; margin: 0; object-fit: contain; width: auto; box-sizing: content-box; } .image-container img.big { max-height: 400px; } .image-container { display: flex; justify-content: space-around; align-items: center; gap: .5rem } .container { width: 90%; margin: 0 auto; } .text-center { text-align: center; } .score-amount { margin: 0; margin-top: 10px; } .score-percentage { font-size: 12px; font-weight: semi-bold; }

Alignment

The alignment score measures how well the image matches its prompt. Annotators saw the prompt and were asked: "Which image matches the description better?"

Preference

The preference score reflects how visually appealing annotators found each image. Without seeing the prompt, they were asked: "Which image looks higher quality and more visually appealing?"

Coherence

The coherence score measures whether the image is free of artifacts and visual glitches. Without seeing the prompt, annotators were asked: "Which image has more glitches and is more likely to be AI generated?" The image picked less often wins.

Text coherence

The text coherence score measures how cleanly a model renders text. It runs only on the 616 prompts that ask for text. Without seeing the prompt, annotators were asked: "In which image does the text have glitches or is harder to read?" The image picked less often wins.

Rankings

Overall ranking (ELO, aggregated across all four leaderboards)

A model shows "—" where it has no standing on that leaderboard.

By output type and capability

OpenAI's GPT Image 2.5 tops every category the benchmark slices by, so the last column shows the strongest model from any other lab. The full per-category standings, including subject and prompt source, are on Benchmark.ai.

How the dataset was built

1. Prompts (1,514)

The prompts are drawn from established text-to-image evaluation sets, real user prompts and prompts written for this benchmark:

Every prompt is tagged on three more axes, which the live leaderboard can slice by:

• Capability — attribute binding, text, scene composition, knowledge & reasoning, spatial & size, number of objects, surreal, emotion & mood, noisy input (a prompt can carry several)
• Output type — general image, photorealism, UI / UX, illustration & anime, graphic design & marketing, logos & branding, infographics, 3D modeling & rendering
• Subject — objects & abstract, people & portraits, animals, architecture & spaces, vehicles, sci-fi & fantasy, food & drink, nature & landscapes

2. Image generation (43 models)

Each prompt was sent to every model, and the resulting images were uploaded to a Rapidata MRI benchmark as one participant per model. Every model has images for between 1,410 and 1,514 of the 1,514 prompts.

Alibaba Cloud — Qwen-Image, Qwen-Image 2.0 Pro, Qwen-Image-3.0, Wan 2.7 Pro · Baidu — ERNIE Image · Black Forest Labs — FLUX.2 dev, FLUX.2 flex, FLUX.2 klein 9B, FLUX.2 max, FLUX.2 pro · ByteDance — Seedream 4.5, Seedream 5.0 Lite, Seedream 5.0 Pro · Google — Imagen 3, Imagen 4, Imagen 4 Fast, Imagen 4 Ultra, nano-banana-2, nano-banana-pro · HiDream AI — HiDream O1 · Ideogram — Ideogram 4.0 Quality, Ideogram 4.5 Quality · Luma Labs — Luma UNI 1.1 Max · Meta — Muse Image · Microsoft AI — MAI-Image 2.5 · OpenAI — GPT Image 1.5 (high), GPT Image 2.0, GPT Image 2.5 (Flare), GPT Image 2.5 (Sunburst) · Pruna AI — Pruna P-Image, Pruna-P-Ideogram (High), Pruna-P-Ideogram (Low), Pruna-P-Ideogram (Medium), Pruna-P-Ideogram (Very Low) · Recraft — Recraft V4.1 Utility Pro · Reve AI — Reve 2.1 · SpaceXAI — Grok Image Imagine 2.0 (Medium), Grok Image Imagine 2.0 (Thinking), Grok Imagine Image, Grok Imagine Image (quality), Grok Imagine Image 2.0 · Stability AI — Stable Diffusion 3.5 Large · Tencent — HunyuanImage 3.0

3. Human evaluation

The four leaderboards were run over pairwise matchups between the models' images for the same prompt. Alignment, Coherence and Text coherence were answered by dedicated, qualified Rapidata audiences; Preference by the open global pool. Each pair is judged on a subset of the questions, with at least 5 votes per question it was judged on; Text coherence runs only on the 616 text prompts. Per-annotator detail — chosen side, country, language, gender, age bucket, occupation and the annotator's userScore — is preserved in the detailed_results_* columns. Votes came from annotators in 148 countries.

Dataset structure

One row per head-to-head image pair generated from the same prompt.

<q> is one of alignment, preference, coherence and text_coherence. A row's columns for a question it was not judged on are empty.

• The weighted_results_* values are userScore-weighted vote sums, not probabilities — they do not sum to 1. Divide by their sum for a normalised win share.
• Coherence and Text coherence are already flipped. Annotators were asked which image has more glitches, but in this dataset votedFor and weighted_results_* point to the image with fewer glitches, so a higher value is better on every question.

import json
from datasets import load_dataset

ds = load_dataset("Rapidata/benchmark-ai-image-benchmark", split="train")
row = ds[0]
print(row["prompt"], "|", row["model1"], "vs", row["model2"])

for q in ["alignment", "preference", "coherence", "text_coherence"]:
    w1, w2 = row[f"weighted_results_image1_{q}"], row[f"weighted_results_image2_{q}"]
    if w1 is None or w1 + w2 == 0:
        continue  # this pair was not judged on q
    print(f"{q}: {row['model1']} wins {w1 / (w1 + w2):.0%} of the weighted vote")

Licensing & attribution

This dataset combines material under different terms:

• Prompts. Prompts taken from public evaluation sets (PartiPrompts, DrawBench, DiffusionDB, MONET, T2I-CompBench, HRS-Bench, ABC-6K, DALLE3-EVAL) remain under their original licenses; Rapidata-original and community prompts are released under CC-BY-4.0.
• Generated images. Produced by third-party models. Model outputs are governed by the terms of use of each respective model provider.
• Human annotations. Collected via Rapidata, released under CC-BY-4.0.

About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit rapidata.ai to learn more about how we're revolutionizing human feedback collection for AI development.

Explore all of our live benchmarks on Benchmark.ai.

Related datasets

• Rapidata/svg-benchmark — the same three core questions for SVGs generated by frontier LLMs.
• Rapidata/camera-movement and Rapidata/relative-camera-movement — camera control in image-to-video models.