🎶 SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment

---

📊 Overview

🎯 A comprehensive benchmark designed to evaluate the quality of AI-generated songs across multiple musical dimensions with fine-grained assessment capabilities. SongBench provides a specialized framework for song assessment across seven key dimensions grounded in the essential elements considered during actual song composition process.

• 🎤 Vocal - Assesses vocal quality, including clarity, pitch stability, and vocal techniques (e.g., vibrato, portamento).
• 🎸 Instrument - Evaluates the realism and synthesis quality of instruments and their acoustic fidelity.
• 🎼 Melody - Assesses melodic richness and memorability, focusing on the creative quality of the musical lines.
• 📋 Structure - Evaluates the organization of song sections (e.g., verse, chorus, bridge), ensuring natural transitions and adherence to compositional logic.
• 🎹 Arrangement - Assesses the artistry of the harmonic framework and instrumental orchestration.
• 🎚️ Mixing - Evaluates post-production quality, focusing on the balance between tracks and the clarity of spatial imaging.
• 🌟 Musicality - A holistic appraisal of the song's overall artistic impact and auditory pleasure.

🚀 Quick Start

Installation

git clone https://github.com/Tencent/SongBench.git
cd SongBench
pip install -r requirements.txt

Evaluation

# Evaluate a single audio file:
python eval.py -i /path/to/audio.mp3 -o /path/to/output

# Evaluate a list of audio files:
python eval.py -i /path/to/audio_list.txt -o /path/to/output

# Evaluate all audio files in a directory:
python eval.py -i /path/to/audio_directory -o /path/to/output

---

🎯 Experimental Results

1、Correlation Analysis

2、Comparative Evaluation

3、AB Test

---

📋 Test Prompts

The out-of-distribution (OOD) test set contains prompts used to generate songs for evaluation. Experimental results reported in the paper are primarily based on songs generated from these prompts. The complete test set is available at prompts/test100.jsonl.

Each sample is represented as a JSON object containing:

• idx: sample identifier
• lyric: structured lyrics with section tags
• text_prompt: textual description of musical style, emotion, and instrumentation

Example:

{
  "idx": "1",
  "lyric": "[intro]\n\n[verse]\n漂泊在无尽的沧海\n寻觅那颗失落的心\n...",
  "text_prompt": "Pop Ballad (Mandopop), Resilient, Piano, Electric Guitar, String Section, Drum Kit"
}

---

📜 License

SongBench is licensed under the SongBench License Terms. Please see the LICENSE.txt file for complete details.

Key License Terms:

• Free for academic and research use
• Commercial use is prohibited
• Requires attribution
• Includes warranty disclaimer

Third-Party Components: This project builds upon open-source components that remain licensed under their original licenses. Please refer to the LICENSE.txt file for complete attribution details.

---

🙏 Acknowledgement

This project references and builds upon the excellent work from:

• SongEval.
• MuQ

---

📖 Citation

If you use SongBench in your research, please cite our paper:

@misc{wu2026songbenchfinegrainedmultiaspectbenchmark,
      title={SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment}, 
      author={Dapeng Wu and Shun Lei and Wei Tan and Guangzheng Li and Yunzhe Wang and Huaicheng Zhang and Lishi Zuo and Zhiyong Wu},
      year={2026},
      eprint={2604.25937},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2604.25937}, 
}