Awesome-SpeechLM-Survey

🎉🎉🎉Our survey paper "Recent Advances in Speech Language Models: A Survey" has been accepted to ACL 2025 main conference!

This is the Github repository for paper: Recent Advances in Speech Language Models: A Survey. In this paper, we survey the field of Speech Language Models (SpeechLMs), which are capable of performing end-to-end speech interactions with humans and serve as autoregressive foundation models.

News

• The short version of our audio reasoning survey paper has been accepted to EMNLP 2026 Findings! 🎉
• Our paper SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis has been accepted to ACM Multimedia 2026 (Dataset Track)! 🎉
• We have released TextPro-SLM: Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM on arXiv! 🎉
• We have released A Survey of Audio Reasoning in Multimodal Foundation Models on arXiv! 🎉
• Our paper TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving has been accepted to Interspeech 2026 Long Paper Track as an Oral presentation! 🎉 Check it out on GitHub.
• Our paper MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models has been accepted to Findings of ACL 2026! 🎉 Check it out on GitHub.
• We have released VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models on arXiv!🎉 Also check it out on Github.

Introduction

Why SpeechLMs? SpeechLMs are used for end-to-end speech-based interactions. Traditional ASR + LLM + TTS setups suffer from information loss and cumulative errors during conversion. SpeechLMs directly model speech data, capturing both semantic and paralinguistic information for richer interactions!

In this repository, we distinguish between two closely related model families:

• Speech Language Models (SpeechLMs) directly accept speech/audio and generate speech/audio. They may operate in turn-based, streaming, or full-duplex settings, but do not rely on an external ASR-LLM-TTS cascade as their core interaction mechanism.
• Large Audio Language Models (LALMs) use a generative language model to understand speech, environmental sound, or music and produce text. Speech-centric LLM-ASR models are included as a dedicated LALM category, while conventional encoder-only ASR models are outside the scope of this list.

Models supporting both text and speech output are listed under SpeechLMs to avoid duplication.

Taxonomy

We introduce a novel taxonomy for SpeechLMs, categorizing them based on their architecture and training recipes.

Speech Language Models (SpeechLMs)

The models below support speech/audio input and speech/audio output within the model. The list covers turn-based, streaming, and full-duplex interaction.

Commercial and Closed Voice Systems

These entries are tracked separately because public information may describe a product or API rather than disclose a standalone end-to-end model architecture.

SpeechLM Training and Alignment

These papers improve the interaction behavior of full-duplex SpeechLMs rather than introducing a separate foundation-model family.

Large Audio Language Models (LALMs)

LALMs accept speech, environmental sound, and/or music and generate text. Models that additionally generate speech are listed only in the SpeechLM section above.

General Audio Understanding

Audio Reasoning, Music, and Prosody

Speech-Centric LALMs and LLM-ASR

This category contains speech-conditioned generative language models whose primary output is text. Conventional encoder-only ASR systems are excluded.

Related Audio and Speech Generation Systems

The following works are closely related but are not included in the strict SpeechLM or LALM lists because they do not provide a unified speech-input/speech-output dialogue model or an audio-conditioned generative text LLM.

SpeechLM Tokenizers

Semantic Tokenizers

Acoustic Tokenizers

Mixed Tokenizers

Popular Training Datasets

Evaluation Benchmarks

Citation

@article{cui2024recent,
  title={Recent advances in speech language models: A survey},
  author={Cui, Wenqian and Yu, Dianzhi and Jiao, Xiaoqi and Meng, Ziqiao and Zhang, Guangyan and Wang, Qichao and Guo, Yiwen and King, Irwin},
  journal={arXiv preprint arXiv:2410.03751},
  year={2024}
}