Dataset Description

MMLottieBench is a comprehensive evaluation protocol for multi-modal vector animation generation. The lack of mature and standardized benchmarks and metrics for vector animation generation poses significant challenges in evaluating (1) the quality of generated vector animations and (2) the extent to which generators faithfully follow multi-modal instructions.

Our benchmark addresses these challenges by providing:

• Real Subset: 450 samples curated from artist-designed Lottie animations
• Synthetic Subset: 450 samples generated using state-of-the-art AI models
• Three Tasks: Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie

Motivation

MMLottieBench aims to construct a benchmark that:

1. Reliably reflects a model's practical utility in real-world scenarios
2. Avoids train-test overlap that commonly arises in conventional dataset splits
3. Ensures fairness and long-term robustness by including synthetic data to mitigate potential contamination from future models

Dataset Structure

Splits

• real: Real Subset with 450 samples from professional designers
• synthetic: Synthetic Subset with 450 samples generated by AI models

Features

Task Distribution

Real Subset (450 samples)

• Text-to-Lottie: 150 textual prompts derived from real Lottie animations
• Text-Image-to-Lottie: 150 samples with one rendered frame and textual description from real Lottie animations
• Video-to-Lottie: 150 rendered animation videos from real Lottie files

Synthetic Subset (450 samples)

• Text-to-Lottie: 150 textual prompts synthesized using GPT-4o
• Text-Image-to-Lottie: 150 vector-style images generated by Gemini-3-Pro Image with motion descriptions
• Video-to-Lottie: 150 reference videos generated by Seedance 1.0 from shape and motion descriptions

Benchmark Construction

Real Subset

The Real Subset consists of samples curated from artist-designed Lottie animations collected from professional designers. All evaluation samples are strictly disjoint from the training data, ensuring assessment on genuinely unseen, real-world content.

Key Features:

• Professionally designed by artists
• Diverse styles and complexity levels
• Real-world animation patterns
• Zero overlap with training data

Synthetic Subset

To ensure the fairness and long-term robustness of our benchmark—particularly to mitigate potential contamination from future models trained on overly similar data—we construct a complementary Synthetic Subset via instruction-based synthesis using state-of-the-art generative models.

Text-to-Lottie Synthesis

We synthesize 150 textual prompts using GPT-4o with carefully designed meta-prompts. The generation instruction ensures high-quality, diverse, and challenging animation prompts suitable for evaluating Lottie generation models.

Key Requirements:

• Icon-friendly design: Flat 2D vector style with transform-only animations (position/rotation/scale/opacity)
• Color specification: Every prompt explicitly specifies solid colors (e.g., "a yellow star", "a blue folder")
• Multi-phase motion: At least 3 distinct motion steps with clear temporal order
• Simultaneous motion: 70+ prompts include concurrent transformations (e.g., "rotating while sliding")
• Return-to-start loops: Each animation must loop seamlessly back to the initial state

Motion Complexity Distribution:

• Easy (30 prompts): Single object with 3 motion steps
• Medium (70 prompts): Single object with 4+ steps or 2 coordinated objects
• Hard (50 prompts): Single object with 5-6 steps or 2-3 coordinated objects with staggered timing

Object Type Coverage:

• UI Icons (50): Loading, progress, buttons, notifications, navigation, media controls
• Common Icons (40): Communication, files, documents, tools, commerce symbols
• Simple Illustrations (30): Nature, animals, food items (minimal icon-style)
• Shapes & Abstract (20): Basic geometric shapes and abstract symbols
• People & Characters (10): Simple stick figures or silhouettes

Allowed Motion Primitives:

• Position: sliding, gliding, drifting, moving, floating, dropping, rising
• Rotation: rotating, spinning, turning, tilting, swinging, wobbling
• Scale: growing, shrinking, pulsing, expanding, compressing, bouncing
• Opacity: fading in/out, appearing, disappearing, flashing, brightening, dimming

Example Prompts:

• "a pink cupcake fading in, sliding upward while shrinking, then sliding downward to its start while bouncing, and finally fading out, looping smoothly"
• "a cyan arrow loop symbol with appearing, rising while spinning, stays still briefly, then dropping back to its start while turning, repeating in a steady loop"

Text-Image-to-Lottie Synthesis

1. Prompt Generation: Using GPT-4o to generate diverse image descriptions suitable for vector animation
2. Image Generation: Using Gemini-3-Pro Image to generate corresponding vector-style images
3. Motion Description: Adding animation descriptions to guide the generation

Video-to-Lottie Synthesis

1. Description Generation: Creating shape and motion descriptions using GPT-4o
2. Video Generation: Using Seedance 1.0 to generate reference videos with parameters:
   • Resolution: 480p
   • Duration: 2 seconds
   • FPS: 8
   • Aspect Ratio: 1:1
   • Style: Pure 2D vector Lottie animation

Why Synthetic Subset?

The synthetic nature of MMLottieBench's Synthetic Subset provides several key advantages:

Evaluation Metrics

MMLottieBench evaluates models across multiple dimensions to comprehensively assess both visual quality and semantic alignment:

Text-to-Lottie and Text-Image-to-Lottie Tasks

• FVD (Fréchet Video Distance) ↓: Measures the visual quality of generated animations
• CLIP Similarity ↑: Evaluates alignment between text prompts and rendered animation frames
• Object Alignment (Obj. Align) ↑ (0-10): Measures object presence, type, count, visual traits, and spatial relations
• Motion Alignment (Motion Align) ↑ (0-10): Assesses correctness of motion type, direction, magnitude, target objects, and smoothness (evaluated independently of object accuracy)

Both alignment metrics use Claude-3.5-Sonnet as an LLM judge. Invalid generations are omitted from evaluation, and blank outputs receive a score of 0.

Video-to-Lottie Task

• FVD (Fréchet Video Distance) ↓: Measures visual quality
• PSNR (Peak Signal-to-Noise Ratio) ↑: Evaluates pixel-level reconstruction quality
• SSIM (Structural Similarity Index) ↑: Measures structural similarity with reference video
• DINO ↑: Evaluates semantic similarity using self-supervised features

Model Efficiency Metrics

• Token Efficiency (# Tokens): Average token length of generated Lottie JSON using the Qwen2.5-VL tokenizer
• Computational Cost (Time): Average generation time per sample in seconds. For closed-source APIs, timing includes full API latency for realistic comparison
• Success Rate: Percentage of valid Lottie animations successfully generated

Quantitative Evaluations

We provide comprehensive quantitative comparisons between state-of-the-art baseline methods across both Real Subset and Synthetic Subset. Bold numbers and underlined numbers represent the best and second-best performance respectively.

Real Subset Results

Text-to-Lottie Task

Text-Image-to-Lottie Task

Video-to-Lottie Task

Synthetic Subset Results

Text-to-Lottie Task

Text-Image-to-Lottie Task

Video-to-Lottie Task

Key Observations

1. Superior Visual Quality: Our method consistently achieves the best FVD scores across all tasks and subsets, demonstrating superior visual quality in generated Lottie animations.

2. Strong Motion Alignment: Our model excels in motion alignment, significantly outperforming baselines in capturing and reproducing complex animation patterns (5.94 vs 4.68 on Real Text-to-Lottie).

3. High Success Rate: With success rates of 80-93%, our method reliably generates valid Lottie animations, far exceeding general-purpose VLMs like GPT-4o (7-28%) and Qwen2.5-VL (0%).

4. Balanced Performance: While some baselines excel in specific metrics (e.g., Recraft in CLIP score), our method achieves the best overall balance across visual quality, semantic alignment, and generation reliability.

5. Token Efficiency Trade-off: Our method uses more tokens (13-42k) compared to VLM baselines (0.3-2.3k) but significantly fewer than optimization-based methods like Recraft (50-54k), striking a balance between expressiveness and efficiency.

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("OmniLottie/MMLottieBench")

# Access specific subsets
real_subset = dataset["real"]
synthetic_subset = dataset["synthetic"]

# Filter by task type
text2lottie = real_subset.filter(lambda x: x["task_type"] == "Text-to-Lottie")
image2lottie = real_subset.filter(lambda x: x["task_type"] == "Text-Image-to-Lottie")
video2lottie = real_subset.filter(lambda x: x["task_type"] == "Video-to-Lottie")

# Example: iterate over text-to-lottie samples
for sample in text2lottie:
    print(f"ID: {sample['id']}")
    print(f"Text: {sample['text']}")
    print(f"Subset: {sample['subset']}")
    # Generate Lottie animation based on the prompt

Benchmark Statistics

Overall Statistics

Data Characteristics

• ID Format: MD5 hash for unique identification
• Text Length: Varies from brief descriptions to detailed motion specifications
• Image Format: PNG with transparency
• Video Format: MP4, 480p resolution, 8 FPS
• Animation Patterns: Fading, sliding, rotating, scaling, bouncing, wobbling, and combinations

Citation

If you use MMLottieBench in your research, please cite:

@article{yang2026omnilottie,
  title={OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens},
  author={Yiying Yang and Wei Cheng and Sijin Chen and Honghao Fu and Xianfang Zeng and Yujun Cai and Gang Yu and Xinjun Ma},
  journal={arXiv preprint arxiv:2603.02138},
  year={2026}
}

License

This dataset is released under the Apache 2.0 License.

Acknowledgments

• Real Subset samples are curated from professional Lottie designers
• Synthetic Subset is generated using GPT-4o, Gemini-3-Pro Image, and Seedance 1.0
• Special thanks to the open-source community for tools and frameworks

Contact

For questions, issues, or contributions, please open an issue on our GitHub repository or contact us at [25113050158@m.fudan.edu.cn].

Updates

• 2026-02: Initial release with 900 samples across 6 task-subset combinations
• 2026-02: Added comprehensive documentation and evaluation metrics

---

Note: This benchmark is designed for research purposes to advance the field of vector animation generation. All synthetic data generation processes are fully documented to ensure transparency and reproducibility.