Rapidata Camera Movement Benchmark

Built by Rapidata.

This dataset contains 281,738 human responses, collected with the Rapidata Python SDK, comparing how well 14 image-to-video models and world models execute a described camera movement from a single still image. Each row is a head-to-head comparison between two models' clips generated from the same still and the same instruction, judged by human annotators who watched a reference animation of the requested movement.

The task is deliberately narrow: the scene must stay exactly as it is, and only the camera may move — a dolly, a truck, a pedestal, a pan, a tilt, a roll, a zoom, or a combination of them. Every clip is ranked purely by human judgement, no automated metrics.

If you get value from this dataset and would like to see more in the future, please consider liking it ❤️

To evaluate your own models and create a leaderboard, check out our MRI.

At a glance

The leaderboard

Every model is compared on the same kind of clip pairs on one question, run as one leaderboard on a Rapidata benchmark:

Annotators are shown the movement's reference animation on top and the two generated clips below it. They never read the text instruction: the prompts are written for a generator (degrees, constraints, frames of reference), while the reference clip shows a human unambiguously what the camera is supposed to do.

.vertical-container { display: flex; flex-direction: column; gap: 60px; } .horizontal-container { display: flex; flex-direction: row; justify-content: center; gap: 60px; } .media-container video, .media-container img { max-height: 260px; margin: 0; object-fit: contain; width: auto; box-sizing: content-box; } .media-container { display: flex; justify-content: space-around; align-items: flex-start; gap: .5rem } .container { width: 90%; margin: 0 auto; } .text-center { text-align: center; } .score-amount { margin: 0; margin-top: 10px; } .score-percentage { font-size: 12px; font-weight: semi-bold; }

Examples

Each example shows the input still, the reference clip annotators were shown, and the two generated clips with the share of the (userScore-weighted) votes each one received. The green border marks the winner.

One motion

Two motions at once

Three motions in sequence

Overall ranking (ELO on the Movement leaderboard)

The live leaderboard additionally lets you slice the standings by number of motions (1 / 2 / 3), by the individual motion a prompt contains (move forward, tilt down, roll clockwise, …) and by scene, so you can see which models hold up as the instruction gets harder and which axes they drop.

How the dataset was built

This section is intentionally detailed so the generation process is fully reproducible and transparent.

1. Movement set (30 camera movements)

The movements cover all six rigid-body degrees of freedom of a camera, each in both directions, plus zoom — and then combine them. They are graded into three difficulty tiers, which is the axis the benchmark reports on:

The wording follows strict rules so that every sentence isolates one skill:

• while means simultaneous and then means sequential, and neither word is used for anything else. The two-move tier is split between the two, because they fail independently.
• No prompt contains a negation. Diffusion text encoders do not represent "without turning" — it embeds toward turning — so every constraint is stated affirmatively: "holding its direction of view fixed", "keeping its view level", "staying in place".
• Angles are spelled out ("forty-five degrees") and turns (yaw) and rolls (lens axis) deliberately keep different verbs.
• No prompt names scene content the still might lack (an interior has a floor but no horizon). This is also why the five orbit movements of the original 35-movement set were dropped: "orbits the subject" needs a single subject that many stills do not have, which makes those prompts ungradable.

Every two- and three-move prompt is composed of one-move prompts from the same set, so a failure can be pinned to a dropped axis: does a model that passes every part still fail the chain?

2. Input stills (22 images)

The stills are copyright-free photographs (one CG interior render) from Pexels and Pixabay, chosen for built environments people inhabit — a mirrored room, a Gothic nave, a library hall, a supermarket aisle, a crowded public square, a car park, a bowling alley, a workshop, a conservatory, a hotel corridor — plus a handful of landscapes and landmarks (Lake Louise, the Canadian Rockies, the Sphinx, the Colosseum, Manhattan from above). Each was picked for a named hard property: parallax that cannot be faked by cropping, populated vertical structure so a rise is measurable, repeated bays or shelves that act as a ruler, mirrors that contradict a wrong reflection.

Not every movement suits every still, so the prompt-to-image mapping is hand-picked rather than a cross product: a descend needs headroom below the lens, a wide pan needs something worth looking at off-axis, a tilt down on a near-nadir aerial reaches straight-down where the scene ends. The result is 600 prompts — between 17 and 30 movements per still.

3. Reference clips (one per movement)

A sentence like "the camera rolls forty-five degrees clockwise, staying in place" is precise but not self-evident, so each movement gets a short looping GIF that shows an annotator the movement before they judge whether a model performed it. The clips are rendered from scratch with a small numpy raytracer, and all 30 share one synthetic scene, so the only thing that differs between two clips is the camera. Rotations, zooms and compound moves visibly decelerate and stop at their stated angle; sequential (then) moves get a beat of stillness at the seam, which is the only thing separating "one, then the other" from "both at once" in a picture. The reference_gif column links this clip for every row.

4. Video generation (14 models)

Each still and its selected instructions were sent to every model as an image-to-video request. Two very different kinds of model compete on the same footing:

Eight commercial image-to-video models, generated at 1080p / 5 s through their public API endpoints with default settings — prompt expansion included, because the point is to compare each model as it comes, not as we tuned it. Audio was switched off wherever there was a switch, since nothing is graded on sound.

Google — Gemini Omni Flash 1.1, Veo 3.1 · Alibaba — Wan 3.0, Happy Horse 1.1 · Kuaishou — Kling V3 Pro · MiniMax — H3 · xAI — Grok Imagine Video 1.5 · Shengshu AI — Vidu Q3

Six research world models, run on their own GPUs with camera-controlled builds where the model exposes camera conditioning, at the models' native output (81 frames at 16 fps for most, 128 frames at 24 fps for Alaya World, 121 frames at 24 fps for Cosmos 3 Nano — all about 5 s). Every model was handed the same seed for the same prompt, derived deterministically from the prompt identifier.

NVIDIA — Cosmos 3 Nano, Cosmos Predict 2.5, Sana WM · Alaya Lab — Alaya World · AMAP — Dream X World · Shanghai AI Laboratory — Yume 1.5

Every model produced a clip for all 600 prompts, so each model meets every other one on identical inputs.

5. Human evaluation

The clips were uploaded to a Rapidata MRI benchmark as one participant per model, and the Movement leaderboard was run over the resulting pairwise matchups. Annotators came from a qualified audience: before seeing any benchmark pair, each one had to pass a gate of known-answer clip pairs asking the same question, at 75% accuracy or better. Every pair received at least 5 votes (6.6 on average), and per-annotator detail — chosen side, country, language, gender, age bucket, occupation and the annotator's userScore — is preserved in the detailed_results_movement column. Votes came from annotators in 128 countries.

Dataset structure

One row per head-to-head clip pair generated from the same still and the same instruction.

The two weighted_results_* values are userScore-weighted vote sums, not probabilities — they do not sum to 1. Divide by their sum for a normalised win share, which is what the example percentages above show.

import json
from datasets import load_dataset

ds = load_dataset("Rapidata/camera-movement", split="train")
row = ds[0]
print(row["prompt"], "|", row["model1"], "vs", row["model2"])

w1, w2 = row["weighted_results_video1_movement"], row["weighted_results_video2_movement"]
print(f"win share {row['model1']}: {w1 / (w1 + w2):.0%}")

votes = json.loads(row["detailed_results_movement"])
print(len(votes), "votes, first one:", votes[0])

Licensing & attribution

This dataset combines material under different terms:

• Input stills. Copyright-free photographs sourced from Pexels and Pixabay, used under their respective licenses.
• Movement instructions and reference clips. Authored and rendered by the dataset creators, released under CC-BY-4.0.
• Generated clips. Produced by third-party models. Model outputs are governed by the terms of use of each respective model provider.
• Human annotations. Collected via Rapidata, released under CC-BY-4.0.

About Rapidata

Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit rapidata.ai to learn more about how we're revolutionizing human feedback collection for AI development.

Explore our latest model rankings on our website.

Known limitations

• Gemini Omni Flash 1.1 does not keep the still's framing. Its endpoint accepts only 16:9 or 9:16, so a 3:2 still is cropped or padded before generation. It is the one model whose clips are not framed like the others.
• Veo 3.1 clips are 6 s, not 5 s. The endpoint only offers 4, 6 or 8 seconds; the tie was rounded up so that a clip never ends before the movement does.
• MiniMax H3 was rendered at its "2K" tier, which the provider documents as an upscale of a 768p base result.
• Orbits are absent by design. The five orbit movements of the original set name a subject to circle, which many stills do not have, so they were removed before evaluation rather than graded inconsistently.