MOSS-Music

English | 简体中文

MOSS-Music is an open-source music understanding model from MOSI.AI, the OpenMOSS team, and Shanghai Innovation Institute. Built on the same audio backbone as MOSS-Audio, MOSS-Music is further specialised on music via dedicated continual pre-training and supervised fine-tuning — targeting musical captioning, lyrics ASR, structural analysis, chord / key / tempo reasoning, and long-form musical question answering. In this release, we provide two 8B models: MOSS-Music-8B-Instruct and MOSS-Music-8B-Thinking. The Instruct variant is optimised for direct instruction following on musical prompts, while the Thinking variant provides stronger chain-of-thought reasoning for musical analysis.

News

• 2026.05.01: 🎉🎉🎉 We have released MOSS-Music.
• 2026.05.01: 🎉🎉🎉 We have released MOSS-Music-Data-Pipeline for large-scale music data annotation and processing.

Contents

• Introduction
• Model Architecture
  • DeepStack Cross-Layer Feature Injection
  • Time-Aware Representation
• Released Models
• Music Data Pipeline
• Evaluation
• Quickstart
  • Environment Setup
  • SGLang Serving
  • Local Inference (Transformers)
  • Gradio App
• More Information
• LICENSE
• Citation

Introduction

Music is not just audio plus lyrics — understanding it requires perceiving harmonic structure, rhythm, timbre, instrumentation, performance nuance, and the textual content of the lyrics, and reasoning about them jointly across time. MOSS-Music is built to unify these capabilities within a single model.

• Lyrics ASR & time-aligned transcription: Accurate singing ASR with sentence- and word-level timestamps, robust to backing tracks.
• Musical captioning & tagging: Natural-language descriptions of mood, genre, instrumentation, production style, and emotional trajectory.
• Key / tempo / chord reasoning: Identifies musical key, beats, downbeats, and chord progressions, including timestamped chord transcription.
• Structural analysis: Segments a song into intro / verse / chorus / bridge / outro and reasons about repetition and contrast.
• Instrument & voice recognition: Identifies prominent instruments and singing voices (solo / chorus, gender, register).
• Musical QA and long-form analysis: Open-ended question answering grounded in a full track, including chain-of-thought reasoning in the Thinking variant.

Model Architecture

MOSS-Music inherits the MOSS-Audio modular design, comprising three components: an audio encoder, a modality adapter, and a large language model. Raw audio is first encoded by MOSS-Audio-Encoder into continuous temporal representations at 12.5 Hz, which are then projected into the language model's embedding space through the adapter and finally consumed by the LLM for auto-regressive text generation.

Rather than relying on off-the-shelf audio frontends, we train a dedicated encoder from scratch to obtain more robust acoustic representations, tighter temporal alignment, and better extensibility across musical styles, singing, and non-speech acoustic content.

DeepStack Cross-Layer Feature Injection

Using only the encoder's top-layer features tends to lose low-level prosody, transient events, and local time-frequency structure. To address this, we adopt a DeepStack-inspired cross-layer injection module between the encoder and the language model: in addition to the encoder's final-layer output, features from earlier and intermediate layers are selected, independently projected, and injected into the language model's early layers, preserving multi-granularity information from low-level acoustic details to high-level semantic abstractions.

This design is especially well-suited for music understanding, as it helps retain rhythm, timbre, transients, and instrumental texture — information that a single high-level representation cannot fully capture, yet is critical for chord recognition, structural analysis, and nuanced musical description.

Time-Aware Representation

Time is a critical dimension in music understanding. To enhance explicit temporal awareness, we adopt a time-marker insertion strategy during pre-training: explicit time tokens are inserted between audio frame representations at fixed time intervals to indicate temporal positions. This design enables the model to learn "what happened when" within a unified text generation framework, naturally supporting timestamped lyrics ASR, beat / downbeat localisation, section boundary detection, and long-song retrospective QA.

Building on the MOSS-Audio backbone, MOSS-Music is further enhanced through:

• continual pre-training on a large, diverse music corpus produced by the data annotation and processing pipeline MOSS-Music-Data-Pipeline, with an emphasis on singing, lyrics, and full-song coverage;
• supervised fine-tuning (SFT) on music-centric instruction data covering captioning, lyrics ASR, chord / key / structural analysis, and long-form musical QA;
• additional reasoning tuning for the Thinking variant.

Released Models

Smaller (4B) variants and additional sizes may follow. Stay tuned!

Music Data Pipeline

The training data used by MOSS-Music is produced by an end-to-end pipeline that goes from raw audio to chat-formatted training samples. That pipeline is available at MOSS-Music-Data-Pipeline, which hosts duration detection, MIR feature extraction, song-structure segmentation, lyrics ASR, metadata cleanup, and ALM-driven caption / query generation with models such as Qwen3-Omni, MusicFlamingo, and other audio-language models.

Evaluation

We evaluate MOSS-Music on a diverse suite of public music understanding benchmarks. Key results:

• Music QA and understanding: MOSS-Music-8B-Instruct achieves 80.38 average accuracy across 8 public music QA benchmarks (excluding the three NSynth note-recognition tracks), ranking first among all compared models in our current evaluation set.
• Music captioning: In our preliminary GPT-5.4-as-a-Judge evaluation, the MOSS-Music series leads both caption benchmarks, with MOSS-Music-8B-Thinking reaching 4.53 on MusicCaps and MOSS-Music-8B-Instruct reaching 4.58 on SDD.
• Lyrics ASR for singing voice: MOSS-Music-8B-Thinking achieves the best average lyrics recognition error across MUSDB18, MIR-1K and Opencpop (15.88% avg WER/CER), clearly ahead of all compared audio-language baselines including Gemini-3.1-Pro-Preview, MusicFlamingo and Qwen3-Omni. Detailed timestamped-ASR results will be released in a later update.
• Chord transcription: MOSS-Music supports chord transcription, including timestamped chord transcription for harmonic analysis, accompaniment reference, and related downstream use cases. Detailed benchmark results will be released in a later update.

Music QA & Understanding (Accuracy↑)

Avg is computed over 8 public music QA benchmarks: MMAU-music, MMAU-mini-music, MMAU-Pro-music, MMAR-music, MuChoMusic, Music-AVQA, GTZAN, and Medley-Solos-DB.

We exclude the three NSynth tracks from the main average because they focus on fine-grained isolated-note recognition, including instrument-family, acoustic/electronic source, and exact pitch discrimination from short single-note clips. Some compared audio-language models are not explicitly designed for this note-level classification setting, so we report NSynth separately for reference rather than mixing it into the headline average.

Music Captioning

We further report a preliminary GPT-5.4-as-a-Judge music captioning comparison on MusicCaps and Song Describer Dataset (SDD). Scores are on a 1-5 scale across 9 dimensions: genre/style, mood/affect, tempo/rhythm, instrumentation/timbre, vocals, melody/harmony, structure/form, production/audio quality, and scene/use case.

• Overall caption quality: the MOSS-Music series remains strongest across both caption benchmarks, with MOSS-Music-8B-Thinking reaching 4.53 on MusicCaps and MOSS-Music-8B-Instruct reaching 4.58 on SDD.
• Stronger structural descriptions: MOSS-Music shows the clearest gains on structure / form / progression, especially on SDD.
• Competitive baselines on instrumentation and scene semantics: MusicFlamingo and Gemini-3.1-Pro remain competitive on instrumentation/timbre, while Gemini-3.1-Pro is strongest on scene / use case.

MusicCaps

Song Describer Dataset (SDD)

Lyrics ASR (WER / CER↓)

We further evaluate MOSS-Music on singing-voice lyrics ASR across three representative benchmarks:

• MUSDB18 — English pop songs with backing tracks, scored with WER;
• MIR-1K — Chinese karaoke clips with background music, scored with CER;
• Opencpop — clean Mandarin studio singing, scored with CER.

Avg is the unweighted mean of the three dataset-level error rates.

MOSS-Music-8B-Thinking achieves the lowest average lyrics-ASR error (15.88%) across these three datasets, with particular gains on MIR-1K (Chinese karaoke with accompaniment) and Opencpop (clean Mandarin singing). MOSS-Music also inherits the strong timestamp-aware ASR ability from MOSS-Audio; detailed singing-timestamp ASR results will be added soon.

Chord Transcription

MOSS-Music supports chord transcription, including timestamped chord transcription that tracks chord progression over time. This can be useful for harmonic analysis, accompaniment reference, music education, and related use cases. Detailed benchmark results will be added soon.

Quickstart

Environment Setup

We recommend Python 3.12 with a clean Conda environment. The commands below are enough for local inference.

Recommended setup

git clone https://github.com/OpenMOSS/MOSS-Music.git
cd MOSS-Music

conda create -n moss-music python=3.12 -y
conda activate moss-music

conda install -c conda-forge "ffmpeg=7" -y
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"

Optional: FlashAttention 2

If your GPU supports FlashAttention 2, you can replace the last install command with:

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"

SGLang Serving

[!IMPORTANT] To achieve the best generation quality and fully leverage the model's capabilities, we strongly recommend using SGLang Serving for inference.

See the full SGLang guide in moss_music_usage_guide.md.

Download the model first:

hf download OpenMOSS-Team/MOSS-Music-8B-Instruct --local-dir ./weights/MOSS-Music-8B-Instruct
hf download OpenMOSS-Team/MOSS-Music-8B-Thinking --local-dir ./weights/MOSS-Music-8B-Thinking

The shortest setup is:

cd sglang
pip install -e "python[all]"
pip install nvidia-cudnn-cu12==9.16.0.29
cd ..

sglang serve \
  --model-path ./weights/MOSS-Music-8B-Instruct \
  --trust-remote-code

You can replace ./weights/MOSS-Music-8B-Instruct with ./weights/MOSS-Music-8B-Thinking if needed.

If you use the default torch==2.9.1+cu128 runtime, installing nvidia-cudnn-cu12==9.16.0.29 is recommended before starting sglang serve.

Local Inference (Transformers)

For a quick local sanity check without SGLang, simply run:

python infer.py

Edit MODEL_PATH, AUDIO_PATH, the prompt and the sampling hyper-parameters at the top of infer.py to point to your own model weights and audio.

[!NOTE] This Transformers path is mainly for quick verification and debugging. For best generation quality and throughput, please prefer SGLang Serving.

Gradio App

Start the Gradio demo with:

python app.py

The server address and port can be overridden via the MOSS_MUSIC_SERVER_NAME and MOSS_MUSIC_SERVER_PORT environment variables, and the default model ID via MOSS_MUSIC_MODEL_ID.

More Information

• MOSI.AI: https://mosi.cn
• OpenMOSS: https://www.open-moss.com
• MOSS-Audio (backbone): https://github.com/OpenMOSS/MOSS-Audio
• MOSS-Music Data Pipeline: https://github.com/wx9songs/MOSS-Music-Data-Pipeline

LICENSE

Models in MOSS-Music are licensed under the Apache License 2.0.

Citation

@misc{mossmusic2026,
      title={MOSS-Music Technical Report},
      author={OpenMOSS Team},
      year={2026},
      howpublished={\url{https://github.com/OpenMOSS/MOSS-Music}},
      note={GitHub repository}
}