[ICLR'25] MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation

Trung X. Pham^1*, Tri Ton^1*, Chang D. Yoo^1†
1KAIST, South Korea
*Co-first authors (equal contribution), †Corresponding authors

📣 News

• [02/2025]: Code released.
• [01/2025]: MDSGen accepted to ICLR 2025 🎉.
• [10/2024]: Paper uploaded to arXiv. Check out the manuscript here.(https://arxiv.org/abs/2410.02130).

To-Dos

• Release model weights on Google Drive.
• Release inference code
• Release training code & dataset preparation

⚙️ Environmental Setups

1. Clone MDSGen.

git clone https://github.com/triton99/MDSGen
cd MDSGen

2. Create the environment.

conda create -n mdsgen python=3.8
conda activate mdsgen 
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
pip3 install -r requirements.txt
pip install -e .

📁 Data Preparations

T.B.D

🚀 Getting Started

Trainning

T.B.D

Download Checkpoints

The pretrained checkpoints can be downloaded on Google Drive.

Inference

Before running inference, extract CAVP features from the video described in the Diff-Foley.

To run the inference code, you can use the following command:

python inference.py --ckpt ./ckpts/mdsgen_audioldm.pt --video_feat_path ./sources/__2MwJ2uHu0_000004.npz

📖 Citing MDSGen

If you find our repository useful, please consider giving it a star ⭐ and citing our paper in your work:

@inproceedings{pham2024mdsgenfastefficientmasked,
  title     = {MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation},
  author    = {Trung X. Pham and Tri Ton and Chang D. Yoo},
  year      = {2025},
  booktitle = {International Conference on Learning Representations},
}

🤗 Acknowledgements

Our code is based on MDT and Diff-Foley. We thank the authors for their excellent work!