Dataset Card for the Movie Gen Benchmark

Movie Gen is a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. Here, we introduce our evaluation benchmark "Movie Gen Bench Video Bench", as detailed in the Movie Gen technical report (Section 3.5.2).

To enable fair and easy comparison to Movie Gen for future works on these evaluation benchmarks, we additionally release the non cherry-picked generated videos from Movie Gen on Movie Gen Video Bench.

Dataset Summary

Movie Gen Video Bench consists of 1003 prompts that cover all the different testing aspects/concepts:

1. human activity (limb and mouth motion, emotions, etc.)
2. animals
3. nature and scenery
4. physics (fluid dynamics, gravity, acceleration, collisions, explosions, etc.)
5. unusual subjects and unusual activities. Besides a comprehensive coverage of different key testing aspects, the prompts also have a good coverage of high/medium/low motion levels at the same time.

Dataset Splits

We are releasing two versions of the benchmark:

1. Test (test): This version includes only the prompts, making it easier to download and use the benchmark.
2. Test with Generations (test_with_generations): This version includes both the prompts and the Movie Gen model’s outputs, allowing for comparative evaluation against the Movie Gen model.

Usage

from datasets import load_dataset

# to download only the prompts
dataset = load_dataset("meta-ai-for-media-research/movie_gen_video_bench_no_generations")["test"]
for example in dataset:
  print(example)
  break

# to download the prompts and movie gen generations
dataset = load_dataset("meta-ai-for-media-research/movie_gen_video_bench", split="test_with_generations", streaming=True)
for example in dataset:
  break

# to display Movie Gen generated video and the prompt on jupyter notebook
import mediapy

with open("tmp.mp4", "wb") as f:
    f.write(example["video"])

video = mediapy.read_video("tmp.mp4")
print(example["prompt"])
mediapy.show_video(video)

Licensing Information

Licensed with CC-BY-NC License.