Ming-UniAudio

📝Technical Report ｜🌐Project Page ｜🤗 Hugging Face｜ 🤖 ModelScope

Table of Contents

• Introduction
• Updates
• Key Features
• Evaluation
  • Speech Tokenizer
  • Speech Understanding
  • Speech Generation
  • Speech Editing
• Model & Benchmark Downloads
• Environment Preparation
• Example Usage
• SFT
• Citation
• Join Us

Introduction

Ming-UniAudio is a novel framework that unifies speech understanding, generation, and editing. Its core is a unified continuous speech tokenizer that effectively unifies semantic and acoustic features within an end-to-end model. We developed a speech language model that strikes a balance between generation and understanding capabilities based on the unified continuous audio tokenizer. Leveraging this foundational model, which exhibits robust performance in both domains, we further trained a dedicated speech editing model built upon Ming-Lite-Omni. Crucially, Ming-UniAudio is the first to enable universal, free-form speech editing guided solely by natural language instructions, handling complex semantic and acoustic modifications without manual region specification.

• 🔥 First unified continuous speech tokenizer for both understanding and generation tasks: MingTok-Audio
• 🔥 First Speech LLM with unifed continuous tokenizer for both understanding and generation: Ming-UniAudio
• 🔥 First universal free-form speech editing model for various semantic and acoustic editing task without any timestamp condition: Ming-UniAudio-Edit
• 🔥 First benchmark for free-form speech editing: Ming-Freeform-Audio-Edit-Benchmark

Updates

• Support VLLM Inference
• Technical Report
• ASR & TTS SFT recipes
• Streaming TTS
• Ming-UniAudio Blog

Key Features

Ming-UniAudio features key optimizations as follows, compared to other audio-assisted LLMs:

• Unified Continuous Speech Tokenizer: Ming-UniAudio proposes a unified continuous speech tokenizer MingTok-Audio based on a VAE framework with a causal Transformer architecture, the first continuous speech tokenizer to effectively integrate semantic and acoustic features, and enables a closed-loop system with LLMs through hierarchical feature representations, makes it suitable for both understanding and generation tasks

• Unified Speech Language Model for Generation and Understanding: We pretrain an end-to-end unified speech language model with a single LLM backbone for both understanding and generation tasks, enhanced with a Diffusion Head to ensure high-quality speech synthesis.
• Instruction-Guided Free-Form Speech Editing: We introduce the first instruction-guided, free-form speech editing framework that supports comprehensive semantic and acoustic edits without requiring explicit edit regions, along with Ming-Freeform-Audio-Edit, the first open-source evaluation set for such tasks.

Evaluation

In various benchmark tests, Ming-UniAudio demonstrates highly competitive results compared to industry-leading models of similar scale.

Speech Tokenizer

Speech Understanding

Speech Generation

Speech Editing

Denoise

Model & Benchmark Downloads

You can download our latest model and Benchmark from both Huggingface and ModelScope.

If you're in mainland China, we strongly recommend you to download our model from 🤖 ModelScope.

pip install modelscope
modelscope download --model inclusionAI/Ming-UniAudio-16B-A3B --local_dir inclusionAI/Ming-UniAudio-16B-A3B  --revision master

Note: This download process will take several minutes to several hours, depending on your network conditions.

Environment Preparation

Installation with pip

pip install -r requirements.txt

Installation with docker

You can set up the environment using Docker in two ways.

• Option 1: Pull from Docker Hub (Recommended)

# 1. Pull the pre-built image
docker pull yongjielv/ming_uniaudio:v1.1

# 2. Run the container
docker run -it --gpus all yongjielv/ming_uniaudio:v1.1 /bin/bash

• Option 2: Build from Source

# 1. Build the image
docker build -t ming-uniaudio:v1.1 -f ./docker/ming_uniaudio.dockerfile .

# 2. Run the container
docker run -it --gpus all ming-uniaudio:v1.1 /bin/bash

Example Usage

We provide a step-by-step running example:

Step 1 - Download the source code

git clone	https://github.com/inclusionAI/Ming-UniAudio
cd Ming-UniAudio

Step 2 - Download the Ming-UniAudio model weights and create a soft link to the source code directory

Download our model following Model & Benchmark Downloads

mkdir inclusionAI 
ln -s /path/to/inclusionAI/Ming-UniAudio-16B-A3B inclusionAI/Ming-UniAudio-16B-A3B

Step 3 - Enter the code directory, you can refer to the following codes to run the Ming-UniAudio model.

python3 cookbooks/test.py

For detailed usage, please refer to demo.ipynb.

Note: We test the examples on hardware of NVIDIA H800-80GB/H20-96G with CUDA 12.4.

SFT

We have open-sourced the Supervised Fine-Tuning (SFT) part for speech generation, which supports both full-parameter and LoRA training. Please follow the recipes to start training.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{yan2025minguniaudiospeechllmjoint,
      title={Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation}, 
      author={Canxiang Yan and Chunxiang Jin and Dawei Huang and Haibing Yu and Han Peng and Hui Zhan and Jie Gao and Jing Peng and Jingdong Chen and Jun Zhou and Kaimeng Ren and Ming Yang and Mingxue Yang and Qiang Xu and Qin Zhao and Ruijie Xiong and Shaoxiong Lin and Xuezhi Wang and Yi Yuan and Yifei Wu and Yongjie Lyu and Zhengyu He and Zhihao Qiu and Zhiqiang Fang and Ziyuan Huang},
      year={2025},
      eprint={2511.05516},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.05516}, 
}

Join Us

The Ant Group's Bailing (百灵) large language model team is seeking talented speech algorithm engineers for R&D in areas including Speech Understanding, Generation, Dialogue, and Tokenizers. If you're passionate about building state-of-the-art speech technology, send your resume to lyuyongjie.lyj@antgroup.com.