Video Generation Model Evaluation Dataset
2
9 commits
1 linked in READMEs
updated Jan 19, 2025
README
Video Generation Model Evaluation Dataset
This dataset contains human annotations for videos generated by different video generation models. The annotations evaluate the quality of generated videos across multiple dimensions.
Dataset Structure
Each JSON file represents one evaluation dimension and follows this structure:
Key Components
- prompt_en: The English text prompt used to generate the videos
- videos: Paths to video files generated by different models
- human_anno: Human annotation scores from 4 different annotators
Models Evaluated
The dataset includes videos generated by 7 different models:
- cogvideox5b
- kling
- gen3
- videocrafter2
- pika
- show1
- lavie
Evaluation Dimensions and Scales
| Dimension | Description | Scale |
|---|---|---|
| Static Quality | ||
| Image Quality | Evaluates technical aspects including clarity and sharpness | 1-5 |
| Aesthetic Quality | Assesses visual appeal and artistic composition | 1-5 |
| Dynamic Quality | ||
| Temporal Consistency | Measures frame-to-frame coherence and smoothness | 1-5 |
| Motion Effects | Evaluates quality of movement and dynamics | 1-5 |
| Video-Text Alignment | ||
| Video-Text Consistency | Overall alignment with text prompt | 1-5 |
| Object-Class Consistency | Accuracy of object representation | 1-3 |
| Color Consistency | Matching of colors with text prompt | 1-3 |
| Action Consistency | Accuracy of depicted actions | 1-3 |
| Scene Consistency | Correctness of scene environment | 1-3 |
Usage
This dataset can be used for:
- Evaluating and comparing different video generation models
- Analyzing human perception of generated videos
- Training automated video quality assessment models
- Studying inter-annotator agreement in video quality assessment
Contributors
7 commits
2 commits
Similar repos
Linked in READMEs
Video Generation Model Evaluation Dataset
2
9 commits
1 linked in READMEs
updated Jan 19, 2025
README
Video Generation Model Evaluation Dataset
This dataset contains human annotations for videos generated by different video generation models. The annotations evaluate the quality of generated videos across multiple dimensions.
Dataset Structure
Each JSON file represents one evaluation dimension and follows this structure:
Key Components
- prompt_en: The English text prompt used to generate the videos
- videos: Paths to video files generated by different models
- human_anno: Human annotation scores from 4 different annotators
Models Evaluated
The dataset includes videos generated by 7 different models:
- cogvideox5b
- kling
- gen3
- videocrafter2
- pika
- show1
- lavie
Evaluation Dimensions and Scales
| Dimension | Description | Scale |
|---|---|---|
| Static Quality | ||
| Image Quality | Evaluates technical aspects including clarity and sharpness | 1-5 |
| Aesthetic Quality | Assesses visual appeal and artistic composition | 1-5 |
| Dynamic Quality | ||
| Temporal Consistency | Measures frame-to-frame coherence and smoothness | 1-5 |
| Motion Effects | Evaluates quality of movement and dynamics | 1-5 |
| Video-Text Alignment | ||
| Video-Text Consistency | Overall alignment with text prompt | 1-5 |
| Object-Class Consistency | Accuracy of object representation | 1-3 |
| Color Consistency | Matching of colors with text prompt | 1-3 |
| Action Consistency | Accuracy of depicted actions | 1-3 |
| Scene Consistency | Correctness of scene environment | 1-3 |
Usage
This dataset can be used for:
- Evaluating and comparing different video generation models
- Analyzing human perception of generated videos
- Training automated video quality assessment models
- Studying inter-annotator agreement in video quality assessment
Similar repos
Linked in READMEs
Contributors
7 commits
2 commits