Rethinking Video Generation Model for the Embodied World

If you like our project, please give us a star ⭐ on GitHub for the latest update.

Key features

• 4M robotic video clips(10K+ hours) for large-scale video generation training.
• 1300+ fine-grained robotic skills, covering diverse actions and task primitives.
• Multi-modal physical annotations, including RGB, depth, and optical flow.
• Multi-robot and multi-task diversity, spanning various robot types, scenarios, and action skills.
• Rich object interactions, enabling complex and realistic robot behavior modeling.

Dataset Structure

RoVid-X provides structured annotations for each video clip in JSON format, where each entry is indexed by the video filename.

📄 JSON Example
{
  "5bebd280785add39d0e40cbea9e0c545.mp4": {
    "verb": "grab@@place",
    "task_caption": "grab the tape and place it into the basket",
    "short_caption": "The robotic arm moves toward the tape, grasps it, and places it into the basket.",
    "detailed_caption": "The video opens with an overhead view of a table covered with a red-and-white checkered tablecloth in a room equipped with technical devices. On the table..."
  }
}

Download

You can download RoVid-X directly from Hugging Face using the official CLI.

# (Optional) If you are in mainland China, use the Hugging Face mirror
export HF_ENDPOINT=https://hf-mirror.com

# Download the dataset
huggingface-cli download DAGroup-PKU/RoVid-X --local-dir datasets/RoVid-X

📚 Citation

If you find this dataset useful, please cite our paper:

@article{deng2026rethinking,
  title={Rethinking Video Generation Model for the Embodied World},
  author={Deng, Yufan and Pan, Zilin and Zhang, Hongyu and Li, Xiaojie and Hu, Ruoqing and Ding, Yufei and Zou, Yiming and Zeng, Yan and Zhou, Daquan},
  journal={arXiv preprint arXiv:2601.15282},
  year={2026}
}