🎬 About the Dataset

SF20K is a large-scale dataset featuring 20,143 short films, totaling over 3,584 hours of video content. Sourced from YouTube and Vimeo, the dataset is composed of amateur films, which minimizes data leakage from the pre-training corpora of large models. This unique characteristic makes SF20K an ideal benchmark for evaluating a model's true video understanding capabilities.

The dataset is designed to challenge models with story-level reasoning through two primary tasks:

1. Multiple-Choice Question Answering (MCQA): Models must select the correct answer from four options based on their understanding of the film's narrative.
2. Open-Ended Question Answering (OEQA): A more demanding task where models must generate free-text answers to questions about the film.

✨ Key Features

• Massive Scale: The largest publicly available movie dataset with 20,143 films.
• Long-Form Content: An average film duration of 11 minutes pushes the boundaries of long-context reasoning in video.
• Rich Narratives: A diverse range of genres and stories provides a robust testbed for story-level understanding.
• Limited Data Contamination: The focus on amateur films ensures a fair evaluation, as the content is unlikely to have been seen by models during pre-training.
• Multi-Modal Annotations: Includes video frames, subtitles, and QA pairs.

Dataset Splits

• SF20K: 20,143 movies, 3,584 hours, 191,007 QA pairs.
• SF20K-Train: 19,071 movies, 3,393 hours, 180,841 QA pairs.
• SF20K-Test: 1,072 movies, 244 hours, 4,885 QA pairs.
• SF20K-Test-Silent: 90 movies, 20 hours, 419 QA pairs.
• SF20K-Test-Expert (public): 50 movies, 11 hours, 538 QA pairs.
• SF20K-Test-Expert (private): 45 movies, 10 hours, 441 QA pairs.

🚀 Getting Started

Accessing the Dataset

The SF20K dataset is hosted on the Hugging Face Hub and can be easily loaded using the datasets library.

# Make sure you have the 'datasets' library installed
# pip install datasets

from datasets import load_dataset

# Load the SF20K dataset
dataset = load_dataset("rghermi/sf20k")

# You can then access different splits (e.g., train, test)
print(dataset["train"][0])

Dataset Structure

Each sample in the dataset contains the following fields:

📊 Evaluation

The dataset is designed for evaluating models on their ability to perform long-form video reasoning. The primary metrics are accuracy for the MCQA task and LLM-QA-Eval (i.e., LLM-based text similarity assessment) for the OEQA task.

📜 Citation

If you use the SF20K dataset in your research, please cite our paper:

@article{ghermi2025longstoryshortstorylevel,
      title={Long Story Short: Story-level Video Understanding from 20K Short Films}, 
      author={Ridouane Ghermi and Xi Wang and Vicky Kalogeiton and Ivan Laptev},
      year={2025},
}