MusiCRS: Benchmarking Audio-Centric Conversational Recommendation

Dataset Description

• Homepage: https://github.com/rohan2810/musiCRS
• Repository: https://github.com/rohan2810/musiCRS
• Paper: https://arxiv.org/abs/2509.19469
• Point of Contact: Rohan Surana, Junda Wu

Dataset Summary

MusiCRS is the first benchmark for audio-centric conversational music recommendation that explicitly links authentic user conversations with corresponding audio tracks. Built from large-scale Reddit discussions where users naturally exchange music suggestions, MusiCRS enables rigorous evaluation of cross-modal understanding in conversational recommendation systems.

The dataset contains:

• 477 high-quality Reddit conversations spanning 7 music genres
• 3,589 unique musical entities (songs, albums, artists)
• Audio grounding via validated YouTube URLs
• ~100 candidates per conversation with ground truth annotations
• Authentic music discovery scenarios from real-world discussions

Supported Tasks

• Conversational Recommendation: Rank candidate songs based on user queries and audio content
• Multimodal Retrieval: Evaluate models across audio-only, query-only, and combined modalities
• Music Information Retrieval: Test cross-modal understanding of musical concepts
• Audio-Text Alignment: Assess how well models ground textual descriptions in actual audio

Languages

The conversations are in English, sourced from Reddit music communities.

Dataset Structure

Data Instances

Each instance contains:

{
    'query': str,                    # LLM-generated query from Reddit post
    'combined_text': str,            # Full Reddit post + top comments
    'name': str,                     # Submission ID (e.g., "t3_vkhefz")
    'source_subreddit': str,         # Genre-specific subreddit
    'submission_entities': List[str],# Extracted entities from post
    'combined_comment_entities': List[str],  # Entities from comments
    'final_candidates': List[str],   # ~100 candidate songs for ranking
    'limited_gt': List[str],         # Ground truth (top 10 upvoted items)
    'ground_truth': List[str],       # Full ground truth (all upvoted items)
    'valid_urls': List[str],         # YouTube URLs for audio (timestamped)
    'url': str,                      # Original Reddit post URL
    'score': float,                  # Reddit post score
    'bucket': str,                   # Post category (e.g., "Music post")
    'comments': List[dict]           # Raw comment data with scores/entities
}

Data Fields

Data Splits

The dataset contains a single train split with 477 conversations.

Genre Distribution

Dataset Creation

Curation Rationale

Existing music recommendation datasets predominantly rely on:

• Implicit feedback (listening logs, clicks) without explicit preferences
• Metadata and tags rather than actual audio content
• Synthetic conversations (LLM-generated) lacking authenticity
• Single modality evaluation

MusiCRS addresses these gaps by:

1. Grounding recommendations in real user conversations from Reddit
2. Linking each conversation to actual audio tracks via YouTube
3. Enabling multimodal evaluation (audio, text, or both)
4. Capturing authentic music discovery scenarios with diverse queries

Source Data

Initial Data Collection

1. Reddit Mining: Collected 2.7M submissions and 28.5M comments from 7 music subreddits
2. Filtering: Retained threads with:
   • Valid YouTube links
   • ≥3 first-level replies
   • Comments ≥5 characters
3. Entity Extraction: Used Qwen2.5-7B to extract songs, artists, albums
4. Query Generation: Generated concise queries with Gemma-3-12b-1t
5. Audio Validation: Verified YouTube URLs link to actual music

Who are the source data producers?

Reddit users from music-focused communities:

• r/classicalmusic
• r/hiphopheads
• r/electronicmusic
• r/metal
• r/popheads
• r/indieheads
• r/jazz

Annotations

Annotation Process

• Ground Truth: Top 10 entities ranked by comment upvotes
• Candidates: 10 ground truth + ~90 random same-subreddit entities (shuffled)
• Quality Control: Human verification of entity extraction and audio links

Who are the annotators?

• Automated Extraction: Qwen2.5-7B (entity extraction), Gemma-3-12b-1t (query generation)
• Human Verification: Research team validated the extracted entity, generated query, and the youtube links validity.

Additional Information

Licensing Information

This dataset is released under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).

Citation Information

If you use MusiCRS in your research, please cite:

@article{surana2025musicrs,
  title={MusiCRS: Benchmarking Audio-Centric Conversational Recommendation},
  author={Surana, Rohan and Namburi, Amit and Mundada, Gagan and Lal, Abhay and Novack, Zachary and McAuley, Julian and Wu, Junda},
  journal={arXiv preprint arXiv:2509.19469},
  year={2025}
}

Leaderboard

We encourage researchers to submit results to our leaderboard tracking performance across:

• Modalities: Audio-only, Query-only, Audio+Query
• Genres: Per-genre performance breakdown
• Metrics: Recall@K, nDCG@K, MRR

See our GitHub repository for baseline results and submission instructions.