Awesome-Large-Speech-Model
A repository used to organize content related to Large Speech(Audio) Model, including paper, data, applications, tools and so on.
outline
Papers
Survey
- WavChat: A Survey of Spoken Dialogue Models , 2024 , arXiv [paper] [github]
- Recent Advances in Speech Language Models: A Survey , 2024 , arXiv [paper]
- A Comprehensive Survey on Deep Multimodal Learning with Missing Modality , 2024 , arXiv [paper]
- A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks , 2024 , arXiv [paper]
- MM-LLMs: Recent Advances in MultiModal Large Language Models , 2024 , ACL [paper] [demo]
- A Survey on Multimodal Large Language Models , 2023 , arXiv [paper] [github]
- Sparks of Large Audio Models: A Survey and Outlook , 2023 , arXiv [paper]
Speech/Audio-Extended Large Language Model
Multi-modal
- Baichuan-Omni Technical Report , 2024 , arXiv [paper] [github]
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , 2024 , arXiv [paper]
- The Llama 3 Herd of Models , 2024 , arXiv [paper]
- Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers , 2024 , arXiv [paper] [github]
- VITA: Towards Open-Source Interactive Omni Multimodal LLM , 2024 , arXiv [paper] [demo]
- NExT-GPT: Any-to-Any Multimodal LLM , 2024 , ICML [paper] [github]
- AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling , 2024 , arXiv [paper] [github]
- Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration , 2023 , arXiv [paper] [github]
Real-time interaction
- Language Model Can Listen While Speaking , 2024 , arXiv [paper] [demo]
- SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training , 2024 , arXiv [paper] [demo]
- SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation , 2024 , arXiv [paper]
- Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data , 2024 , arXiv [paper]
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM , 2024 , arXiv [paper] [github] [demo]
- OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation , 2024 , arXiv [paper] [demo]
- Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models , 2024 , arXiv [paper] [github]
- Moshi: a speech-text foundation model for real-time dialogue , 2024 [paper] [github]
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models , 2024 , arXiv [paper] [github]
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming , 2024 , arXiv [paper] [github]
Speech/Audio understanding & generation
- GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities , 2024 , arXiv [paper] [github] [demo]
- WavLLM: Towards Robust and Adaptive Speech Large Language Model , 2024 , EMNLP [paper] [github]
- EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions , 2024 , arXiv [paper] [demo]
- MooER: LLM-based Speech Recognition and Translation Models from Moore Threads , 2024 , arXiv [paper] [github]
- Qwen2-Audio Technical Report , 2024 , arXiv [paper] [github]
- An Embarrassingly Simple Approach for LLM with Strong ASR Capacity , 2024 , arXiv [paper] [github]
- SALMONN: Towards Generic Hearing Abilities for Large Language Models , 2024 , ICLR [paper] [github]
- LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT , 2023 , arXiv [paper] [demo]
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models , 2023 , arXiv [paper] [github]
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities , 2023 , EMNLP [paper] [github]
- Pengi: An Audio Language Model for Audio Tasks , 2023 , NeurIPS [paper] [github]
- AudioPaLM: A Large Language Model That Can Speak and Listen , 2023 , arXiv [paper] [demo]
Speech Recognition & Translation
- Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition , 2024 , arXiv [paper] [demo]
- OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer , 2024 , Interspeech [paper] [github]
- Seamless: Multilingual Expressive and Streaming Speech Translation , 2023 , arXiv [paper] [github]
- SeamlessM4T: Massively Multilingual & Multimodal Machine Translation , 2023 , arXiv [paper] [github]
- Whisper: Robust Speech Recognition via Large-Scale Weak Supervision , 2022 , arXiv [paper] [github]
Speech Synthesis
- Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis , 2024 , arXiv [paper] [github]
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching , 2024 , arXiv [paper] [github]
- FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications , 2024 , arXiv [paper] [github] [demo]
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer , 2024 , arXiv [paper] [github] [demo]
- Natural language guidance of high-fidelity text-to-speech with synthetic annotations , 2024 , arXiv [paper] [demo]
- CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens , 2024 , arXiv [paper] [github] [demo]
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models , 2024 , arXiv [paper] [demo]
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models , 2024 , arXiv [paper] [github]
- Boosting Large Language Model for Speech Synthesis: An Empirical Study , 2023 , arXiv [paper]
- SoundStorm: Efficient Parallel Audio Generation , 2023 , arXiv [paper] [demo]
- VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers , 2023 , arXiv [paper] [github]
- PromptTTS: Controllable Text-to-Speech with Text Descriptions , 2022 , arXiv [paper] [demo]
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech , 2021 , ICLR [paper] [demo]
Self-supervised Pre-trained Speech Model
- BEATs: Audio Pre-Training with Acoustic Tokenizers , 2022 , arXiv [paper] [github]
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language , 2022 , arXiv [paper] [github]
- W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training , 2021 , arXiv [paper]
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing , 2021 , arXiv [paper]
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , 2021 , arXiv [paper]
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations , 2020 , NeurIPS [paper] [github]
- wav2vec: Unsupervised Pre-Training for Speech Recognition , 2019 , arXiv [paper] [github]
Speech Tokenization/Discretization
- TS3-Codec: Transformer-Based Simple Streaming Single Codec , 2024 , arXiv [paper]
- Universal Speech Token Learning Via Low-Bitrate Neural Codec and Pretrained Representations , 2024 , IEEE Journal of Selected Topics in Signal Processing [paper]
- Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis , 2024 , ICLR [paper] [github] [demo]
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling , 2024 , arXiv [paper] [github] [demo]
- RepCodec: A Speech Representation Codec for Speech Tokenization , 2024 , ACL [paper] [github]
- FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec , 2024 , arXiv [paper] [github] [demo]
- Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation , 2024 , Interspeech [paper] [demo]
- ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis , 2024 , arXiv [paper] [demo]
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models , 2024 , ICLR [paper] [github]
- Finite Scalar Quantization: VQ-VAE Made Simple , 2023 , arXiv [paper] [github]
- High-Fidelity Audio Compression with Improved RVQGAN , 2023 , NeurIPS [paper] [github]
- Encodec: High Fidelity Neural Audio Compression , 2022 , arXiv [paper] [github]
- SoundStream: An End-to-End Neural Audio Codec , 2021 , TASLP [paper] [github] [demo]
- vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations , 2019 , arXiv [paper]
- VQ-VAE: Neural Discrete Representation Learning , 2017 , NeurIPS, [paper] [code]
Music Model
- Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation , 2024 , arXiv [paper]
- Seed-Music: A Unified Framework for High Quality and Controlled Music Generation , 2024 , arXiv [paper]
- QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation , 2024 , arXiv [paper] [github] [github]
- M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models , 2024 , arXiv [paper] [github]
- SingSong: Generating musical accompaniments from singing , 2023 , arXiv [paper] [example]
- Simple and Controllable Music Generation , 2023 , NeurIPS , [paper] [github]
- MusicLM: Generating Music From Text , 2023 , arXiv [paper] [demo]
- MuLan: A Joint Embedding of Music Audio and Natural Language , 2022 , ISMIR [paper] [github]
Speech/Audio/Music Data
- Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation , 2024 , SLT [paper] [huggingface]
- WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark , 2024 , Interspeech [paper]
- MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Research , 2024 , Interspeech [paper] [huggingface]
- YODAS: Youtube-Oriented Dataset for Audio and Speech , 2023 , ASRU [paper] [demo]
- GigaST: A 10,000-hour Pseudo Speech Translation Corpus , 2023 , Interspeech [paper] [demo]
- AudioCaps: Generating Captions for Audios in The Wild , 2019 , ACL [paper] [demo]
Benchmark
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark , 2024 , arXiv [paper] [github] [demo]
- AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension , 2024 , ACL [paper] [github]
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words , 2024 , arXiv [paper] [github]
Safety & Risk
- GPT-4o System Card , 2024 [paper]
- Safe Guard: an LLM-agent for Real-time Voice-based Hate Speech Detection in Social Virtual Reality , 2024 , arXiv [paper]
- SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models , 2024 , ACL [paper]
- Warning: humans cannot reliably detect speech deepfakes , 2024 , arXiv [paper]
Others
- Low-latency Speech Enhancement via Speech Token Generation , 2024 , ICASSP [paper]
- Genhancer: High-Fidelity Speech Enhancement via Generative Modeling on Discrete Codec Tokens , 2024 , Interspeech [paper]
- Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model , 2023 , arXiv [paper] [github] [demo]
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models , 2023 , NeurIPS [paper] [demo]
- AudioLM: a Language Modeling Approach to Audio Generation , 2022 , arXiv [paper] [demo]
Open-source Large Speech/Audio Models
- Whisper
- [github]
- Task: ASR
- Institution: OpenAI
- Introduction: Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. It supports speech recognition for 100 languages. Five model sizes are trained, including tiny(39 M), base(74 M), small(244 M), medium(769 M), large(1550 M) and the newest large v3 model is trained on 5M hours of labeled data.
- Seamless
- [github]
- Task: ASR/S2TT/S2ST/T2TT/T2ST
- Institution: Meta
- Introduction: Seamless is a family of AI models that enable more natural and authentic communication across languages. It contains three main models, SeamlessM4T, SeamlessExpressive and SeamlessStreaming. SeamlessM4T is a massive multilingual multimodal machine translation model supporting around 100 languages. SeamlessM4T serves as foundation for SeamlessExpressive, a model that preserves elements of prosody and voice style across languages and SeamlessStreaming, a model supporting simultaneous translation and streaming ASR for around 100 languages. SeamlessExpressive and SeamlessStreaming are combined into Seamless, a unified model featuring multilinguality, real-time and expressive translations.
- SenseVoice
- [github]
- Task: ASR
- Institution: Alibaba
- Introduction: SenseVoice is a speech foundation model with multiple speech understanding capabilities, including automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and audio event detection (AED).
- CosyVoice
- [github]
- Task: TTS
- Institution: Alibaba
- Qwen2-Audio
- [github]
- Task: S2T(Voice Chat & Audio Analysis)
- Institution: Alibaba
- Introduction: The latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. Two distinct audio interaction modes are introductioned: (1) voice chat: users can freely engage in voice interactions with Qwen2-Audio without text input; (2) audio analysis: users could provide audio and text instructions for analysis during the interaction; Two models are released of the Qwen2-Audio series: Qwen2-Audio-7B and Qwen2-Audio-7B-Instruct.
- SALMONN
- [github]
- Task: S2T(General Audio Proceessing)
- Institution: Tsinghua University and ByteDance
- Introduction: SALMONN is a large language model (LLM) enabling speech, audio events, and music inputs. Instead of speech-only input or audio-event-only input, SALMONN can perceive and understand all kinds of audio inputs and therefore obtain emerging capabilities such as multilingual speech recognition and translation and audio-speech co-reasoning.
- Reverb
- [github]
- Task: ASR & Speech Diarization
- Institution: Rev
- Introduction: Open source inference and evaluation code for Rev's state-of-the-art speech recognition and diarization models. The speech recognition (ASR) code uses the WeNet framework and the speech diarization code uses the Pyannote framework. Reverb ASR was trained on 200,000 hours of English speech, all expertly transcribed by humans. The speech recognition models released outperform all existing open source speech recognition models across a variety of long-form speech recognition domains.
- NVIDIA NeMo ASR
- [github]
- Task: ASR
- Institution: NVIDIA
- Introduction: NVIDIA NeMo team released a number of inference optimizations for CTC, RNN-T, and TDT models that resulted in up to 10x inference speed-up. These models now exceed an inverse real-time factor (RTFx) of 2,000, with some reaching RTFx of even 6,000.
- Westlake-Omni
- [github]
- Task: S2S(Speech Interaction)
- Institution: xinchen-ai
- Introduction: Westlake-Omni is an open-source Chinese emotional speech interaction large language model that utilizes discrete representations to achieve unified processing of speech and text modalities. The model supports low-latency generation and high-quality Chinese emotional speech interaction.
- GLM-4-Voice
- [github]
- Task: S2S(Speech Interaction)
- Institution: Zhipu AI
- Introduction: GLM-4-Voice is an end-to-end voice model launched by Zhipu AI. GLM-4-Voice can directly understand and generate Chinese and English speech, engage in real-time voice conversations, and change attributes such as emotion, intonation, speech rate, and dialect based on user instructions.
- MiniCPM-o
- [github]
- Task: S2S(Multi-modal LLM)
- Institution: THUNLP & Modelbest
- Introduction: MiniCPM-o is the latest series of end-side multimodal LLMs (MLLMs) ungraded from MiniCPM-V. The models can now take image, video, text, and audio as inputs and provide high-quality text and speech outputs in an end-to-end fashion.
Datasets
Only common benchmarks and large-scale training datasets are listed here.
- Speech Recognition & Translation
- Speech Synthesis
- Audio Caption
- Audio Question Answering
| datasets | language | task | duration/hours |
|---|
| Librispeech [download] | English | ASR | 960 |
| Aishell-1 [dowmload] | Mandarin | ASR | 170 |
| GigaSpeech [download] | English | ASR | 10k |
| WenetSpeech [download] | Mandarin | ASR | 10k |
| Libri-Light [download] | English | ASR | 60K |
| Common Voice [download] | Multilingual | ASR | >20k |
| MuST-C/MuST-C v2 | Multilingual | ST | 500 |
| CoVoST/CoVoST-2 [download] | Multilingual | ST | 2880 |
| GigaST [download] | English-German/Chinese | ST | 10k |
| YODAS [download] | Multilingual | ASR/Self-supervised | >500k |
| MSR-86K [download] | Multilingual | ASR | 86k |
| LibriTTS [download] | English | TTS | 585 |
| WenetSpeech4TTS [download] | Mandarin | TTS | 10k |
| Emilia [download] | Multilingual | TTS | 101k |
| fleurs [download] | Multilingual | ASR/ST/MT | 10 |
| AudioCaps [download] | - | AC | - |
| VoiceAssistant-400K [download] | English | AQA | - |
- Silero VAD: A pre-trained enterprise-grade Voice Activity Detector. [github]
- ESPnet: End-to-end speech processing toolkit [github]
- Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit [github]
- Huggingface speech-to-speech: An effort for an open-sourced and modular GPT-4o. [github]
- faster-whisper: Faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is a fast inference engine for Transformer models. [github]
- whisper_streaming: Whisper realtime streaming for long speech-to-text transcription and translation. [github]
- SLAM-LLM: A deep learning toolkit that allows researchers and developers to train custom multimodal large language model (MLLM), focusing on Speech, Language, Audio, Music processing. [github]
- swift: Supports training(PreTraining/Fine-tuning/RLHF), inference, evaluation and deployment of 300+ LLMs and 50+ MLLMs (multimodal large models). [github] [DOCUMENTATION]
- Huggingface Parler-TTS: A lightweight text-to-speech (TTS) model that can generate high-quality, natural sounding speech in the style of a given speaker (gender, pitch, speaking style, etc). [github] [demo]
- RTC: Real-time communication (RTC) is a real-time communication technology that allows the exchange of audio, video, or data between computers or other devices. You can find some demos in the github open-sourced by google. And many companies are also provide RTC PaaS survices.
- edge-tts: edge-tts is a Python module that allows you to use Microsoft Edge's online text-to-speech service from within your Python code or using the provided edge-tts or edge-playback command. [github]
- fish-speech: Input a 10 to 30-second vocal sample to generate high-quality TTS output. [github]
Applications
