AudenAI/azeros

Model

AZeroS

2

3 commits

2 linked in READMEs

updated Jan 24, 2026

See the code

README

AZeroS

AZeroS (Auden Zero-instruction-tuned Speech-LLM) extends a frozen LLM to speech via Self-Generated Instruction-Free Tuning (SIFT). It keeps the LLM and audio encoders frozen and trains lightweight projection modules on speech–text pairs, achieving strong semantic and paralinguistic performance with modest training cost, generalizing well to unseen instructions.

🔗 Paper: https://arxiv.org/pdf/2601.06086
🔗 Code: https://github.com/AudenAI/Auden/tree/main/examples/azeros
🔗 Model: https://huggingface.co/AudenAI/azeros
🔗 Auden Repo: https://github.com/AudenAI/Auden

🔍 What Can This Model Do?

  • 🎙️ Speech understanding (semantic content understanding and dialog)
  • 😊 Paralinguistic analysis (emotion, age, gender, etc.)

Quick Start

import torch
from model import AZerosModel

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AZerosModel.from_pretrained("AudenAI/azeros").to(device)

wav_files = ["speech1.wav", "speech2.wav"]
messages = [
    [
        {
            "role": "user",
            "content": f"{model.audio_token_wrapped} Please analyze speech content and paralinguistic information.",
        }
    ]
    for _ in wav_files
]

generate_config = {
    "max_new_tokens": 200,
    "num_beams": 1,
    "do_sample": False,
    "min_length": 1,
    "repetition_penalty": 1.0,
    "length_penalty": 1.0,
    "top_p": None,
    "top_k": None,
    "temperature": None,
}

outputs = model.generate(wav_files, messages, **generate_config)
print(outputs)

Auden Setup (Required)

This model relies on the Auden codebase for loading and inference:

git clone https://github.com/AudenAI/Auden.git
cd Auden
pip install -e .
cd examples/azeros

📌 Model Characteristics

  • Input: Raw audio waveform (16 kHz) or text
  • Output: Text responses regarding to the input
  • Backend LLM: Qwen2.5-7B-Instruct
  • Encoders: TTA and Auden-Voice
  • Architecture: Frozen LLM + frozen audio encoders + lightweight projection modules
  • Training paradigm: Self-Generated Instruction-Free Tuning (SIFT)

📊 Evaluation

VoiceBench

ModelAlpaca EvalComm EvalWild VoiceSD-QABBHAdv BenchIF EvalOBQAMMSUOverall
Text Only Model
Qwen2.54.664.554.6262.0380.0099.0470.1484.8471.5782.69
Qwen2.5 (TN)4.614.534.5663.8456.3098.8566.1174.0764.5177.52
Cascaded System
Whisper+GPT-4o4.804.474.6275.7787.2098.2776.5192.9781.6987.80
Whisper+Qwen2.54.644.334.2158.5052.8598.2763.9978.2469.0076.05
End-to-end Speech-LLM
GPT-4o4.784.494.5875.5084.1098.6576.0289.2380.2586.75
Moshi2.011.601.3015.6447.4044.2310.1225.9324.0429.51
Phi-4-multimodal3.813.823.5639.7861.80100.0045.3565.9342.1964.32
GLM-4-Voice3.973.423.1836.9852.8088.0825.9253.4139.7556.48
Qwen2-Audio3.423.292.7631.6553.0099.0426.3548.3536.1453.77
DeSTA2.53.732.523.3046.4762.4097.6965.4772.7558.5666.04
Qwen2.5-Omni3.883.773.5246.7563.7097.3140.1981.5461.4568.26
Qwen3-Omni-30B4.744.544.5876.9080.4099.3077.8089.7068.1085.49
AZeroS (ours)4.444.183.9160.2256.3098.6561.2972.0959.0173.13

AIRBench

ModelGenderEmotionAgeLIDEntityIntentAvgChat
Cascaded System
Whisper+GPT-4o21.9059.5041.1096.8069.8087.7062.807.54
Whisper+Qwen2.528.3650.8036.4088.0073.6082.7059.987.34
End-to-end Speech-LLM
GPT-4o*49.10*76.0061.6085.80*7.53
Gemini2.5-pro90.7060.7034.1099.1068.5092.2074.228.52
SALMONN35.5029.9048.7028.1051.7036.7038.436.16
GLM-4-Voice23.9122.9518.7025.4027.9021.1023.335.53
Qwen2-Audio64.7148.1523.1077.8087.0084.7064.247.20
DeSTA2.584.2464.3065.6097.3065.2083.7076.727.57
Qwen2.5-Omni89.7654.8544.8089.7079.7088.6074.576.97
Qwen3-Omni-30B91.1162.2036.9097.7080.4090.7076.507.85
AZeroS (ours)86.7571.4561.3084.8073.6085.6077.258.28

An additional prompt is added to ensure steady output of choices: “Please make your choice among A/B/C/D and do not output other texts.”

⚠️ Limitations

  • Trained on public datasets; performance may degrade on out-of-domain audio.
  • Not designed for safety-critical applications.

Citation

If you use AZeroS in your research, please cite:

@article{shao2026azeros,
  title={AZEROS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning},
  author={Shao, Yiwen and Liu, Wei and Li, Jiahong and Wang, Tianzi and Wei, Kun and Yu, Meng and Yu, Dong},
  journal={arXiv preprint arXiv:2601.06086},
  year={2026}
}
audio
audio-text-to-text
azeros
instruction-free
paralinguistic
safetensors
speech
speech-llm

Contributors

yshao18

3 commits

AudenAI/azeros

Model

AZeroS

2

3 commits

2 linked in READMEs

updated Jan 24, 2026

See the code

README

AZeroS

AZeroS (Auden Zero-instruction-tuned Speech-LLM) extends a frozen LLM to speech via Self-Generated Instruction-Free Tuning (SIFT). It keeps the LLM and audio encoders frozen and trains lightweight projection modules on speech–text pairs, achieving strong semantic and paralinguistic performance with modest training cost, generalizing well to unseen instructions.

🔗 Paper: https://arxiv.org/pdf/2601.06086
🔗 Code: https://github.com/AudenAI/Auden/tree/main/examples/azeros
🔗 Model: https://huggingface.co/AudenAI/azeros
🔗 Auden Repo: https://github.com/AudenAI/Auden

🔍 What Can This Model Do?

  • 🎙️ Speech understanding (semantic content understanding and dialog)
  • 😊 Paralinguistic analysis (emotion, age, gender, etc.)

Quick Start

import torch
from model import AZerosModel

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AZerosModel.from_pretrained("AudenAI/azeros").to(device)

wav_files = ["speech1.wav", "speech2.wav"]
messages = [
    [
        {
            "role": "user",
            "content": f"{model.audio_token_wrapped} Please analyze speech content and paralinguistic information.",
        }
    ]
    for _ in wav_files
]

generate_config = {
    "max_new_tokens": 200,
    "num_beams": 1,
    "do_sample": False,
    "min_length": 1,
    "repetition_penalty": 1.0,
    "length_penalty": 1.0,
    "top_p": None,
    "top_k": None,
    "temperature": None,
}

outputs = model.generate(wav_files, messages, **generate_config)
print(outputs)

Auden Setup (Required)

This model relies on the Auden codebase for loading and inference:

git clone https://github.com/AudenAI/Auden.git
cd Auden
pip install -e .
cd examples/azeros

📌 Model Characteristics

  • Input: Raw audio waveform (16 kHz) or text
  • Output: Text responses regarding to the input
  • Backend LLM: Qwen2.5-7B-Instruct
  • Encoders: TTA and Auden-Voice
  • Architecture: Frozen LLM + frozen audio encoders + lightweight projection modules
  • Training paradigm: Self-Generated Instruction-Free Tuning (SIFT)

📊 Evaluation

VoiceBench

ModelAlpaca EvalComm EvalWild VoiceSD-QABBHAdv BenchIF EvalOBQAMMSUOverall
Text Only Model
Qwen2.54.664.554.6262.0380.0099.0470.1484.8471.5782.69
Qwen2.5 (TN)4.614.534.5663.8456.3098.8566.1174.0764.5177.52
Cascaded System
Whisper+GPT-4o4.804.474.6275.7787.2098.2776.5192.9781.6987.80
Whisper+Qwen2.54.644.334.2158.5052.8598.2763.9978.2469.0076.05
End-to-end Speech-LLM
GPT-4o4.784.494.5875.5084.1098.6576.0289.2380.2586.75
Moshi2.011.601.3015.6447.4044.2310.1225.9324.0429.51
Phi-4-multimodal3.813.823.5639.7861.80100.0045.3565.9342.1964.32
GLM-4-Voice3.973.423.1836.9852.8088.0825.9253.4139.7556.48
Qwen2-Audio3.423.292.7631.6553.0099.0426.3548.3536.1453.77
DeSTA2.53.732.523.3046.4762.4097.6965.4772.7558.5666.04
Qwen2.5-Omni3.883.773.5246.7563.7097.3140.1981.5461.4568.26
Qwen3-Omni-30B4.744.544.5876.9080.4099.3077.8089.7068.1085.49
AZeroS (ours)4.444.183.9160.2256.3098.6561.2972.0959.0173.13

AIRBench

ModelGenderEmotionAgeLIDEntityIntentAvgChat
Cascaded System
Whisper+GPT-4o21.9059.5041.1096.8069.8087.7062.807.54
Whisper+Qwen2.528.3650.8036.4088.0073.6082.7059.987.34
End-to-end Speech-LLM
GPT-4o*49.10*76.0061.6085.80*7.53
Gemini2.5-pro90.7060.7034.1099.1068.5092.2074.228.52
SALMONN35.5029.9048.7028.1051.7036.7038.436.16
GLM-4-Voice23.9122.9518.7025.4027.9021.1023.335.53
Qwen2-Audio64.7148.1523.1077.8087.0084.7064.247.20
DeSTA2.584.2464.3065.6097.3065.2083.7076.727.57
Qwen2.5-Omni89.7654.8544.8089.7079.7088.6074.576.97
Qwen3-Omni-30B91.1162.2036.9097.7080.4090.7076.507.85
AZeroS (ours)86.7571.4561.3084.8073.6085.6077.258.28

An additional prompt is added to ensure steady output of choices: “Please make your choice among A/B/C/D and do not output other texts.”

⚠️ Limitations

  • Trained on public datasets; performance may degrade on out-of-domain audio.
  • Not designed for safety-critical applications.

Citation

If you use AZeroS in your research, please cite:

@article{shao2026azeros,
  title={AZEROS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning},
  author={Shao, Yiwen and Liu, Wei and Li, Jiahong and Wang, Tianzi and Wei, Kun and Yu, Meng and Yu, Dong},
  journal={arXiv preprint arXiv:2601.06086},
  year={2026}
}
audio
audio-text-to-text
azeros
instruction-free
paralinguistic
safetensors
speech
speech-llm

Contributors

yshao18

3 commits