中文说明

MiDashengLM-Gen is an end-to-end framework that uses a pre-trained Large Language Model and audio tokenizer as the backbone, combined with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. It generates coherent 16 kHz audio scenes that simultaneously blend speech, music, sound effects, and environmental acoustics from structured text descriptions.

Highlights

• Autoregressive generation with per-token flow matching — combines LLM sequence modeling with continuous flow-based generation, enabling variable-length output via a learned stop head
• LLM-conditioned high-dimensional audio latents — generates over 768-dimensional semantic-acoustic latents (25 Hz) conditioned on LLM hidden states, achieving rich acoustic modeling without quantization artifacts
• Audio-text alignment pre-training — bridges the modality gap by mapping audio latents into the LLM's token space before generation training, critical for cross-modal synthesis
• Strong speech intelligibility with competitive mixed-audio quality — approaches dedicated TTS performance on Seed-TTS while maintaining mixed-audio scene generation, and supports 9 languages with emotion control

Architecture

Left: training pipeline with flow matching loss. Right: autoregressive inference pipeline.

Key components:

Input Format

Input uses structured multi-view captions with special tokens to describe different aspects of an audio scene. Use <|unknown|> for absent elements.

Quick Start (HuggingFace)

pip install torch torchaudio "transformers>=4.51" einops safetensors soundfile tqdm numpy x-transformers

from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = model.cuda()

result = model.generate(
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit "
    "<|asr|> And that is why I never buy cheap luggage anymore! "
    "<|speech|> expressive comedic male voice "
    "<|music|> sudden upbeat jazz band sting "
    "<|sfx|> uproarious crowd laughter "
    "<|env|> intimate comedy club"
)

sf.write("output.wav", result["audio"], result["sample_rate"])

Batch generation:

texts = [
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club",
    "<|caption|> Upbeat instrumental music with occasional whistling and low audio quality. <|asr|> <|unknown|> <|speech|> <|unknown|> <|sfx|> Intermittent whistling emerges in later segment alongside persistent instrumental playback. <|music|> Bright melodic composition with guitar, piano, and rhythmic percussion elements, maintaining a consistently upbeat character. <|env|> Low-fidelity recording with compressed dynamics and limited frequency response.",
]
result = model.generate(texts)
for i, audio in enumerate(result["audio"]):
    sf.write(f"output_{i}.wav", audio, result["sample_rate"])

Generation parameters:

result = model.generate(
    "...",
    eval_cfg=2.0,           # Classifier-free guidance scale
    stop_threshold=0.5,     # Stop prediction threshold
    min_stop_step=5,        # Minimum steps before stopping
    seed=42,                # Random seed for reproducibility
)

Advanced Usage (from source)

For checkpoint-based inference or local evaluation, clone this repository and use infer.py directly.

Installation

uv sync
# Run inference with:
#   uv run python infer.py ...   (recommended, auto-uses .venv)
# OR
#   source .venv/bin/activate && python infer.py ...

Single Text Generation

# Checkpoint auto-downloaded from Zenodo on first run
uv run python infer.py \
    --text "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club" \
    --output_dir ./output

# Or specify checkpoint path explicitly
uv run python infer.py \
    --resume /path/to/DashengLM-Gen-checkpoint.pth \
    --text "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club" \
    --output_dir ./output

Batch Generation (JSONL)

Each line is a JSON object with audio_id and caption (or content):

{"audio_id": "sample_001", "caption": "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club"}

uv run python infer.py \
    --resume /path/to/DashengLM-Gen-checkpoint.pth \
    --input_list /path/to/your_input.jsonl \
    --output_dir ./output

Parameters

Output

• 16 kHz mono WAV files
• Auto-truncated by stop prediction
• Batch mode: named by audio_id field; single mode: output.wav

Citation

If you use this work, please cite:

@article{sun2026midashenglmgen,
  title={MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching},
  author={Sun, Xingwei and Dinkel, Heinrich and Li, Gang and Mei, Jiahao and Niu, Yadong and Han, Zerui and Jiang, Yuepeng and Zhou, Jiahao and Fan, Lichun and Luan, Jian},
  journal={arXiv preprint},
  eprint={2608.11804},
  archivePrefix={arXiv},
  year={2026}
}

License

This project is licensed under the Apache License 2.0.

Use Restrictions

You are solely responsible for your use of MiDashengLM-Gen and any outputs, actions, or consequences arising therefrom, and you agree not to use MiDashengLM-Gen or any derivatives thereof:

• For any unlawful, fraudulent, or malicious purpose, or in any manner that violates any applicable laws or regulations;
• To infringe upon the intellectual property rights, privacy rights, publicity rights, or other lawful rights or interests of any third party;
• To exploit, harm, harass, defame, unlawfully discriminate against, or otherwise adversely affect any individual or group, including minors or vulnerable persons;
• For any military purpose or application.