PusaV0.5 Training Dataset

Code Repository | Model Hub | Training Toolkit | Dataset |Pusa Paper | FVDM Paper | Follow on X | Xiaohongshu

Dataset Overview

This repository contains the pre-encoded training dataset used for fine-tuning the Pusa-V0.5 video generation model. The dataset consists of 52,695 pre-encoded latent samples derived from VIDGEN-1M, total size is 785GB, though Pusa-V0.5 was trained using only 16,000 of this dataset.

Dataset Structure

The dataset is organized into two main directories:

PusaV0.5_Training/
  videos/
    xxxx.latent.pt  # Pre-encoded video latents
    xxxx.latent.pt
    ...
  captions/
    xxxx.embed.pt   # Pre-encoded text embeddings
    xxxx.embed.pt
    ...

• videos/: Contains pre-encoded video latents in PyTorch tensor format. Atually, the corresponding videos (.mp4 files) are also provided in videos/, you may check them out for more details.
• captions/: Contains corresponding text embeddings for each video

Dataset Details

• Total Samples: 52,695 video-text embedding pairs
• Source: Randomly sampled from VIDGEN-1M
• Format: Pre-encoded latents (.pt files) ready for training
• Used in Pusa-V0.5: 16,000 samples from this dataset were used to train the released Pusa-V0.5 model

Usage

Download the Dataset

huggingface-cli download RaphaelLiu/PusaV0.5_Training --repo-type dataset --local-dir <path_to_dataset_directory>

Unzip the Dataset

cd <path_to_dataset_directory>/PusaV0.5_Training
bash unzip.sh

Using with Mochi-Full-Finetuner

This dataset is designed to work seamlessly with the Mochi-Full-Finetuner repository for training Pusa or Mochi models:

python -u /path/to/src/genmo/mochi_preview/train_pusa.py \
  --world_size=8 \
  --model_dir="/path/to/model/directory" \
  --data_path="/path/to/PusaV0.5_Training/videos"

Note: When specifying --data_path, provide only the path to the videos directory. The training script will automatically locate the captions directory by replacing "videos" with "captions" in the base path.

Creating Your Own Dataset

If you wish to create your own dataset in the same format, follow the instructions in the Mochi LoRA Training repository. Your dataset should match the structure shown above, with corresponding latent and embedding files for each sample.

Citation

If you use this dataset in your research, please cite:

@misc{Liu2025pusa,
  title={Pusa: Thousands Timesteps Video Diffusion Model},
  author={Yaofang Liu and Rui Liu},
  year={2025},
  url={https://github.com/Yaofang-Liu/Pusa-VidGen},
}

@article{liu2024redefining,
  title={Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach},
  author={Liu, Yaofang and Ren, Yumeng and Cun, Xiaodong and Artola, Aitor and Liu, Yang and Zeng, Tieyong and Chan, Raymond H and Morel, Jean-michel},
  journal={arXiv preprint arXiv:2410.03160},
  year={2024}
}