📢 News

• 2025-04-21: Released MiDashengLM-0.6B, a lightweight variant based on Qwen3-0.6B, available in FP32 for HuggingFace inference and GGUF (3-bit to 16-bit) for CPU deployment via llama.cpp. MDL-0.6B-FP32 | MDL-0.6B-GGUF | Demo | Video
• 2025-10-21: Model updated from 0804 (tech report version) to 1021, supports audio-embedded instructions (e.g., "Follow the instruction at the start of an audio.") previously text-only.
• 2025-10-09: Uploaded several newly quantized model variants for resource-constrained devices.
• 2025-09-24: Released the mdl-toolkit, a user-friendly fine-tuning toolkit for MiDashengLM. ESC-50 example Notebook: en | 中文
• 2025-09-04: vLLM now officially supports MiDashengLM. Deploy dasheng-lm with vLLM. And we're developing the 4-bit quantized version, please stay tuned.
• ​2025-09-01: vLLM integration PR submitted to the official vLLM repository. Preview available in our fork during review. See Issue #17 for details.

🔥 Key Highlights

Flexible Model Sizes

• Available in 7B and 0.6B variants, covering high-performance servers to CPU-only edge devices.
• 0.6B supports CPU deployment via llama.cpp and runs directly in the browser via WebAssembly.

State-of-the-Art Performance

• Outperforms Qwen2.5-Omni-7B, Kimi-Audio-Instruct-7B on multiple key audio understanding tasks.

High Efficiency (MDL-7B)

• 3.2× throughput speedup at comparable batch sizes compared to Qwen2.5-Omni-7B.
• 20x throughput speedup by increasing furhter batchsizes. We tested up to a batch size=512 for 30s audio input on 80GB GPUs. Baselines only support batch size = 8.
• Time-to-first-token (TTFT) speedup of up to 4x compared to Qwen2.5-Omni-7B.

Caption-based Alignment

• Trained with general audio captions (instead of ASR transcripts) to achieve holistic audio understanding.

Full Transparency

• Public-source training data and reproducible pipeline.
• Apache License 2.0 for both research and commercial use.

Acknowledgment and Model Foundation

Although MiDashengLM demonstrates superior audio understanding performance and efficiency compared to Qwen2.5-Omni models, we acknowledge Qwen2.5-Omni as a remarkable and respected foundational work in the field. Our 7B model specifically uses Qwen2.5-Omni-7B Thinker as the initialization for decoder training, building upon its robust architecture and weight initialization. The 0.6B variant uses Qwen3-0.6B as the decoder initialization, offering a lightweight alternative for resource-constrained deployment.

The audio encoder is built upon Dasheng, an open-source audio encoder for general audio understanding with state-of-the-art performance. Dasheng serves as the core foundation enabling MiDashengLM's exceptional performance.

Framework

MiDashengLM integrates the powerful Dasheng audio encoder with the Qwen2.5-Omni-7B Thinker decoder through a unique caption-based alignment strategy. Unlike conventional ASR-driven approaches, our model leverages general audio captions to capture comprehensive audio representations encompassing speech, environmental sounds, and musical elements in a unified textual format. This design enables holistic audio understanding while maintaining exceptional computational efficiency.

Why Captions Instead of ASR?

ASR Limitations:

• Discards huge amount of non-speech audio (music/environmental sounds).
• Misses paralinguistic info (speaker emotion, acoustic properties).
• Monotonic alignment provides trivial learning signal.

Caption Advantages:

• Utilizes all audio content.
• Captures global audio context.
• Non-monotonic alignment provides a hard learning signal.

Novel Open Source Dataset for Training: ACAVCaps

ACAVCaps is a meticulously curated 38,662-hour collection of general audio captions derived from the open-source ACAV100M audio repository. While leveraging ACAV100M's extensive raw audio materials, we completely re-engineered the annotation process to create a dataset for holistic audio understanding. We devide the dataset into six categories:

The figure below illustrates our data curation pipeline for ACAVCaps:

Each caption is generated through a three-step process:

1. Multi-expert analysis (speech, vocal, music, acoustics)
2. LLM reasoning synthesizing metadata with DeepSeek-R1
3. Filtering for audio-text consistency with Dasheng-GLAP

We will release the ACAVCaps dataset after the ICASSP 2026 review process.

Available Model Variants

We provide multiple precision / quantization formats to cover different deployment and fine-tuning scenarios:

MiDashengLM-0.6B

The 0.6B GGUF models can be deployed on CPU using llama.cpp, or run directly in the browser. See the GGUF model page for instructions, or try the WASM demo (video).

MiDashengLM-7B

Usage Guidance:

• FP32: Use only when numerical precision is critical (e.g., for rigorous reproduction or benchmarking). For general use, it consumes more resources without a corresponding quality gain.
• BF16: Recommended for most general-purpose scenarios, including inference and fine-tuning. It delivers quality comparable to FP32 while being significantly faster on modern GPUs (e.g., A100, H100, RTX 4090).
• FP8: Optimized for Hopper-class (H100 and newer) GPUs, leveraging hardware support for enhanced performance and memory savings. While older GPUs may see limited performance gains, FP8 can still be used to conserve VRAM, and storage.
• GPTQ W4A16: An ideal choice for resource-constrained environments. It offers broad GPU compatibility and a smaller memory footprint, making it suitable for deployment where VRAM, memory, or storage is limited, provided that a slight trade-off in quality is acceptable.

The full list model variants of MiDashengLM: Hugging Face / Model Scope

Usage

Load Model

from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

# For 7B:
model_id = "mispeech/midashenglm-7b-1021-bf16"
# For 0.6B:
# model_id = "mispeech/midashenglm-0.6b-fp32"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

If you are in a region with limited access to Hugging Face resources, you may want to use hf-mirror as a mirror of Hugging Face:

export HF_ENDPOINT=https://hf-mirror.com

Construct Prompt

user_prompt = "Caption the audio."  # You may try any other prompt

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_prompt},
            {
                "type": "audio",
                "path": "/path/to/example.wav",
                # or "url": "https://example.com/example.wav"
                # or "audio": np.random.randn(16000)
            },
        ],
    },
]

Generate Output

import torch

with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    )
    generation = model.generate(**model_inputs)
    output = tokenizer.batch_decode(generation, skip_special_tokens=True)  # ["An engine is idling."]

Fine-tuning

We appreciate the ms-swift implementation contributed by @JimmyMa99 in ms-swift#5325.

We also provide MDL-Toolkit, a user-friendly fine-tuning toolkit for MiDashengLM.

Deploy with vLLM

vLLM provides a high-performance, user-friendly library for LLM inference and serving.

Install vLLM with pip or from source:

# Set up using Python-only build (without compilation)
git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 pip install --editable .

# Full build (with compilation)
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

You can find sample code for offline execution in the VLLM repository audio_language.

# Offline inference
python3 examples/offline_inference/audio_language.py -m midashenglm

# Online serving using OpenAI-compatible server
python3 -m vllm.entrypoints.openai.api_server --model mispeech/midashenglm-7b --tensor-parallel-size 1 --served-model-name default --port 8000 --dtype float16 --max_model_len 4096 --trust_remote_code

Results

The technical report primarily evaluates the midashenglm-7b-0804-fp32 model, which represents our initial release from August 4, 2025. Note that our current best-performing model is midashenglm-7b-1021, though the technical report has not been updated to include its results. For detailed experimental tables and performance metrics, please refer to the Hugging Face model pages or the technical report.

Reproduction Instructions

To reproduce our results, we provide:

• Prompts (prompt.csv)
• Evaluation scripts
• Example JSONL files

1. Install Dependencies for Evaluation (No need this for inference)

pip install -r requirements.txt

2. Generate Model Outputs

Generate responses using the model's official framework with prompts from prompt.csv.

3. Convert Outputs to JSONL Format

Format model outputs using the example JSONL files:

4. Evaluate Results

Execute the corresponding evaluation scripts:

# Automatic Speech Recognition (WER)
# Uses: lang, text, model_output
python evaluate/wer/compute_wer.py -i evaluate/jsonl/MiDashengLM_LibriSpeech_test-clean.jsonl

# Single-target Audio Tagging (ACC)
# Uses: label, model_output
python evaluate/compute_at_acc.py -i evaluate/jsonl/MiDashengLM_NSynth.jsonl

# Gender Recognition (ACC)
# Uses: label, model_output
python evaluate/compute_gender_acc.py -i evaluate/jsonl/MiDashengLM_VoxCeleb-Gender.jsonl

# Multi-target Audio Tagging (mAP)
# Uses: dataset_name, label, model_output, model_name
python evaluate/compute_map.py -i evaluate/jsonl/MiDashengLM_FSD50K.jsonl

# Audio Captioning (FENSE)
# Uses: audio, text, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_AutoACD.jsonl

# Open Audio QA (FENSE)
# Uses: audio, answer, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_MusicQA.jsonl

# Audio QA with Options (ACC)
# Uses: answer, model_output
python evaluate/compute_qa_acc.py -i evaluate/jsonl/MiDashengLM_MuChoMusic.jsonl

5. Evaluate on MECAT and MMAU benchmarks

Please refer to the official repositories for evaluation on the MECAT and MMAU benchmarks.

Efficiency

MiDashengLM-7B demonstrates superior inference efficiency compared to Qwen2.5-Omni-7B, achieving 3.2× speedup at comparable batch sizes and an overall potential speedup of 20.2× with larger batches.

Tested on 80GB GPU with 30s audio, 100-token output.

Citation

MiDashengLM is under the Apache License 2.0, and we encourage its use in both research and business applications.

If you find MiDashengLM useful in your research, please consider citing our work:

@techreport{midashenglm7b,
  title      = {MiDashengLM: Efficient Audio Understanding with General Audio Captions},
  author     = {{Horizon Team, MiLM Plus}},
  institution= {Xiaomi Inc.},
  year       = {2025},
  note       = {Contributors: Heinrich Dinkel et al. (listed alphabetically in Appendix B)},
  url        = {https://arxiv.org/abs/2508.03983},
  eprint     = {2508.03983},
}