🎼 WikiMT-X: Multimodal Music Benchmark Dataset

WikiMT-X (WikiMusicText-eXtended) is the first multimodal MIR benchmark dataset that combines text, sheet music, and audio recordings with 1,000 carefully curated triplets, primarily from 20th-century Western music. It is designed to evaluate text-to-music generation, music captioning, music information retrieval (MIR), and music classification across multiple modalities, providing a rich and diverse resource for advancing music AI research. 🚀

---

❓ Why WikiMT-X

The original WikiMT dataset had several limitations:

❌ Limited text diversity – Text came from Wikipedia, mostly background-only, lacking variety.
❌ No audio data – The absence of audio recordings restricted multimodal evaluation.
❌ Low-quality genre labels – Labels were automatically assigned using keyword matching, leading to inaccuracies.

✅ How We Improved It

WikiMT-X introduces major enhancements, making it a robust multimodal benchmark:

🔹 Rich text annotations → Using llama-3.1-sonar-large-128k-online (Perplexity AI), we generated diverse textual descriptions from sheet music metadata.

🔹 Four distinct textual perspectives:

• 📜 Background – Historical and cultural context.
• 🎼 Analysis – Musical structure and composition insights.
• 📝 Description – Generalized overview, excluding identifiable details (e.g., title, artist, lyrics).
• 🖼️ Scene – Imagery and mood depiction.

🔹 Aligned sheet music with audio → We manually matched sheet music with corresponding audio recordings from YouTube and removed duplicates.

🔹 Refined genre annotations → We restructured genre categories and manually re-annotated the dataset for improved accuracy.

---

🎯 Benchmarking Tasks

WikiMT-X serves as a benchmark for four major multimodal music tasks:

🎶 1. Text-to-Music Generation

• Generating sheet music or audio from text prompts.
• Conditioning on background, analysis, description, and scene annotations.

📝 2. Music Captioning

• Generating textual descriptions from sheet music or audio.
• Captions are extracted from multiple semantic perspectives.

🔍 3. Music Information Retrieval (MIR)

• Text-to-music retrieval → Find music based on textual descriptions.
• Music-to-text retrieval → Retrieve contextual metadata from audio or sheet music.
• Cross-modal retrieval using sheet music and audio.

🎼 4. Music Classification

• Genre classification across sheet music, audio, and text.

---

📊 Dataset Summary

WikiMT-X consists of 1,000 multimodal triplets, making it an ideal benchmark for multimodal MIR research:

✅ Text-to-music generation with structured text conditioning.
✅ Music captioning using diverse textual perspectives.
✅ Cross-modal MIR for retrieval tasks.
✅ Music classification across multiple modalities.

This dataset enables comprehensive evaluation of multimodal music models. 🌎🎶

📜 Citation

If you find WikiMT-X useful in your work, please consider citing our paper:

@misc{wu2025clamp3universalmusic,
  title={CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages}, 
  author={Shangda Wu and Zhancheng Guo and Ruibin Yuan and Junyan Jiang and Seungheon Doh and Gus Xia and Juhan Nam and Xiaobing Li and Feng Yu and Maosong Sun},
  year={2025},
  eprint={2502.10362},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2502.10362}
}