MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

📖 arXiv | 🎬 MMAR Demo Video | 🛠️ GitHub Code | 🔊 MMAR Audio Download (HuggingFace)

News

• [Feb. 2026] Final results of the Interspeech 2026 Audio Reasoning Challenge are now available on the leaderboard page! Check out our challenge report for a detailed analysis and insights.
• [Feb. 2026] We released the MMAR-Rubrics, an updated version of the MMAR benchmark with a novel instance-level rubric-based protocol for reasoning process quality evaluation. Check out the evaluation script for more details.
• [Dec. 2025] We organized the Interspeech 2026 Audio Reasoning Challenge, with both Single Model Track and Agent Track, to advance the research on audio reasoning.

Overview of MMAR

We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. Examples include:

The metadata for MMAR is available in this file. Unlike previous benchmarks, MMAR not only covers traditional modalities such as speech, audio, and music, but also extends to their mix, collected from in-the-wild videos. The distribution of data across these modalities is illustrated in the left figure. Furthermore, each question is annotated with a designated category and sub-category, as shown in the right figure.

For each question, we also provide the URL and corresponding timestamp of the original video, as well as the spoken language (if present) in the clip. To prevent potential data leakage into training for reasoning models, we have withheld reasoning cues and chain-of-thought annotations, which will be released at an appropriate time.

Leaderboard

We benchmark the models (open-source or API) on MMAR across five model categories:

1. Large Audio Language Models (LALMs)
2. Large Audio Reasoning Models (LARMs)
3. Omni Language Models (OLMs)
4. Large Language Models (LLMs) with audio captions as input
5. Large Reasoning Models (LRMs) with audio captions as input

Dataset Creation

The MMAR benchmark was constructed with a comprehensive pipeline. The process includes:

1. Brainstorming challenging questions
2. Building a taxonomy through human-LLM collaboration
3. Heuristic-based data collection and annotation
4. Crawling audio data and enriching content across multiple slots
5. Performing iterative correction and quality inspection to ensure high data fidelity

Test Your Model !

MMAR Accuracy Evaluation Script

To ensure a smooth integration into existing evaluation pipelines, we adopt an evaluation methodology modified from MMAU, implemented in evaluation.py. The input to the evaluation script should be the same as MMAR-meta.jsonl, with an additional key named answer_prediction, which stores the model prediction for each question.

To run the script:

python evaluation.py  --input INPUT_JSONL_PATH

MMAR-Rubrics for CoT Quality Evaluation Script

We also provide an evaluation script for the MMAR-Rubrics, which assesses the quality of the reasoning process generated by models or agents. The implementation is available in evaluation_rubrics.py. The input to this script should be similar to the previous one, but with two additional key named thinking_prediction and answer_prediction, which contains the model-generated CoT and final answer, respectively.

Evaluation Protocol:

1. Answer Correctness: If the answer_prediction is incorrect, the score is 0.
2. Reasoning Quality: If the answer_prediction is correct, an LLM-as-a-judge evaluates the thinking_prediction on a scale of 0.2 to 1.0 (in 0.2 increments).
3. Stability Mechanism:
   • The rubrics are designed to be instance-level, rather than system-level, to provide fine-grained feedback on the reasoning process for each question. The rubrics are stored in the MMAR-meta.jsonl file with rubric as keys.
   • To account for variance, each submission is calculated based on 5 independent evaluation runs. The final score for each metric will be the mean of the 3 middle runs, effectively discarding the highest and lowest results.

To run the script:

export OPENAI_API_KEY="your_openai_api_key_here"
python evaluation_rubrics.py  --input INPUT_JSONL_PATH

Acknowledge

We gratefully acknowledge that our evaluation code is modified from the official implementation of MMAU.

Citation

@article{ma2025mmar,
  title={MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix},
  author={Ma, Ziyang and Ma, Yinghao and Zhu, Yanqiao and Yang, Chen and Chao, Yi-Wen and Xu, Ruiyang and others},
  journal={Proc. NeurIPS},
  year={2025}
}

@article{ma2025arc,
  title={The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agents},
  author={Ma, Ziyang and Xu, Ruiyang and Ma, Yinghao and Yang, Chao-Han Huck and Li, Bohan and Kim, Jaeyeon and Xu, Jin and Li, Jinyu and Busso, Carlos and Yu, Kai and Chng, Eng Siong and Chen, Xie},
  journal={arXiv preprint arXiv:2602.14224},
  year={2026}
}