🤗 WildSongBench

A benchmark for full-song music generation

192 prompts · 94 Chinese · 98 English

🎵 YuE2 project · 🚀 Quick start · 📊 Benchmarks · 🔁 Reproduce · 📚 Citation

           

WildSongBench (WSB) contains 192 song-generation prompts: 94 Chinese and 98 English, used in the YuE2 benchmarks. This repository provides prompts, exact inputs and seeds, reference scores, and a minimal evaluator for SongBench, Q3O, MuQ-MuLan, AllMusicCaps and Qwen3-ASR PER. Audio is not included.

🚀 Quick start

from datasets import load_dataset

wsb = load_dataset("m-a-p/WildSongBench", split="test")

Each record contains a prompt. Generation inputs contain the corresponding style, lyrics and exact seeds for every prompt.

📊 Benchmarks

Recorded paper results on all 192 prompts. SongBench averages seven quality dimensions; Q3O measures prompt control on a 0–5 scale. MuQ-MuLan and AllMusicCaps measure audio–text similarity. PER is phoneme error rate.

Systems use their recorded candidate-selection protocols; this is not a matched-compute comparison. Full results and protocols.

🔁 Reproduce standard YuE2

Use YuE2-3B with YuE2-Vae-legacy for benchmark scoring. The scripts pin the tested checkpoints and inference version. The benchmark recipe retains inference version 0.1.3 with updated public repository defaults; model weights and numerical inference code are unchanged.

Run from the downloaded repository in a fresh Python 3.12 environment on Linux, with a C/C++ compiler for Q3O's key scorer. Q3O requires an 80 GB GPU; the other metrics support 24 GB. Public evaluator weights download once at fixed revisions (~83 GB for all metrics). The installer creates the required environments. Select your GPU by its numeric index below.

python -m pip install huggingface-hub==0.36.2
export CUDA_VISIBLE_DEVICES=0
python code/install_inference.py
tar --no-same-owner -xzf evaluation/wsb_metrics.tar.gz
python wsb_metrics/wsb_eval.py prepare \
  --metrics all --download huggingface --install

python code/generate.py --candidates 2 --backend torch --output runs/wsb
python wsb_metrics/wsb_eval.py score \
  --input runs/wsb/evaluation_inputs.jsonl \
  --output runs/scores --mode standard --metrics per,songbench
python wsb_metrics/wsb_eval.py score \
  --input runs/wsb/evaluation_inputs.jsonl \
  --per-results runs/scores/per_song.jsonl \
  --output runs/additional --metrics mulan,allmusiccaps,q3o

For prompt row i (zero-based, 0–191), generate two candidates with seeds 831001 + i and 831019 + i. The script applies each seed to symbolic planning, semantic generation and acoustic synthesis. Four ASR passes estimate each candidate's PER; the lower-PER candidate is selected for every metric. Score all 192 prompts and retain capped outputs. Q3O uses the paper's fixed candidate weights and traditional key scoring. The scripts handle preprocessing and tie breaks.

On a 24 GB GPU, prepare per,songbench,mulan,allmusiccaps and omit q3o from the last command. The second scoring command reuses the first run's PER selection and evaluates exactly those 192 songs.

Selected scores are in runs/scores/summary.json (SongBench/PER) and runs/additional/summary.json (other metrics), under standard_selected.groups.all.

Fresh SongBench scores are 6.73 (H800) and 6.74 (RTX 4090), against 6.73 in the paper. On the original 192 songs, MuQ-MuLan and AllMusicCaps mean scores match within 0.000001. Q3O gives 4.69 vs 4.68 on the original audio and 4.71 on fresh H800 generations; equality to two decimals remains unresolved. Best-of-8 has not been rerun in full.

📚 Citation

Technical report coming soon. For now, please cite YuE when using WildSongBench in your research.

@article{yuan2025yue,
  title = {{YuE}: Scaling Open Foundation Models for Long-Form Music Generation},
  author = {Yuan, Ruibin and Lin, Hanfeng and Guo, Shuyue and Zhang, Ge and Pan, Jiahao and Zang, Yongyi and Liu, Haohe and Liang, Yiming and Ma, Wenye and Du, Xingjian and Du, Xinrun and Ye, Zhen and Zheng, Tianyu and Jiang, Zhengxuan and Ma, Yinghao and Liu, Minghao and Tian, Zeyue and Zhou, Ziya and Xue, Liumeng and Qu, Xingwei and Li, Yizhi and Wu, Shangda and Shen, Tianhao and Ma, Ziyang and Zhan, Jun and Wang, Chunhui and Wang, Yatian and Chi, Xiaowei and Zhang, Xinyue and Yang, Zhenzhu and Wang, Xiangzhou and Liu, Shansong and Mei, Lingrui and Li, Peng and Wang, Junjie and Yu, Jianwei and Pang, Guojian and Li, Xu and Wang, Zihao and Zhou, Xiaohuan and Yu, Lijun and Benetos, Emmanouil and Chen, Yong and Lin, Chenghua and Chen, Xie and Xia, Gus and Zhang, Zhaoxiang and Zhang, Chao and Chen, Wenhu and Zhou, Xinyu and Qiu, Xipeng and Dannenberg, Roger and Liu, Jiaheng and Yang, Jian and Huang, Wenhao and Xue, Wei and Tan, Xu and Guo, Yike},
  journal = {arXiv preprint arXiv:2503.08638},
  year = {2025},
  eprint = {2503.08638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08638}
}