If you like our project, please give us a star ⭐ on GitHub for the latest update.

• Awesome Codec, TTS & Speech LM
  • Neural Codec Models
  • Zero-Shot Text-to-Speech Synthesizers
  • Speech Language Models
    • End-to-End (Paradigms) Models
    • Cascaded (Paradigms) Models
  • Benchmark
  • Survey
• Music Generation
• Some Interesting Models
• Speech DataSet
• Some Interesting knowledge
  • Blog & Courses
  • Minor Points of Concern
• Reference

Awesome Codec, TTS & Speech LM

1. Acoustic Tokens: Acoustic tokens focuses on speech compression and reconstruction, which rely on encoder-decoder architectures with residual vector quantization (RVQ). Specifically, these models quantify speech features (which are downsampled from raw wavforms by one encoder) into a series of discrete tokens, then use one decoder to upsample these discrete tokens into the speech, calculating the reconstruction loss against the original signal. By this approach, we can get discrete acoustic tokens with impressive compression rates and high-fidelity acoustic information, making it more suitable for tasks such as speech synthesis and emotion analysis. (requires maintaining reconstruction ability and a low bitrate)
2. Semantic Tokens: Semantic tokens involve applying clustering algorithms such as K-means to extract features from self-supervised learning models, using the cluster indices as discrete representations. And it is prediction-based modeling, these models are trained for representation learning by predicting future frames in an autoregressive manner or by using surrounding frames to predict masked frames. This approach tends to prioritize capturing linguistic information within speech, making it particularly useful for recognition and understanding tasks.
3. Speech Large Language Models: These models are trained on top of speech and acoustic tokens in a language modeling approach. They demonstrate proficiency in tasks on speech understanding and speech generation. (From speech-trident)

Neural Codec Models

• [2025/12] PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning [paper][code]
• [2025/11] BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction [paper][code][demo]
• [2025/10] UniTok-Audio: A Unified Audio Generation Framework Via Universal Discrete Token [paper][code][demo] :heavy_check_mark:
• [2025/10] Low Resource Audio Codec Challenge Track1: Transparency Codec [paper][demo]
• [2025/10] PhoenixCodec: Taming Neural Speech Coding for Extreme Low-Resource Scenarios [paper][demo]
• [2025/10] SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain [paper]
• [2025/10] Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding [paper][code][demo] :heavy_check_mark:
• [2025/10] SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization [paper][code][demo] :heavy_check_mark:
• [2025/10] MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding [paper][code][demo] :heavy_check_mark:
• [2025/10] U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation [paper][code][demo] :heavy_check_mark:
• [2025/10] LDCodec: A high quality neural audio codec with low-complexity decoder [paper]
• [2025/10] FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates [paper][code][demo] :heavy_check_mark:
• [2025/10] LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models [paper][code] :heavy_check_mark:
• [2025/10] BridgeCode: A Dual Speech Representation Paradigm for Autoregressive Zero-Shot Text-to-Speech Synthesis [paper][demo]
• [2025/10] Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates [paper][code]
• [2025/09] Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis [paper][code]
• [2025/09] MBCodec:Thorough disentangle for high-fidelity audio compression [paper]
• [2025/09] FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation [paper][code] :heavy_check_mark:
• [2025/09] MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement [paper]
• [2025/09] FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs [paper][code] :heavy_check_mark:
• [2025/09] CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio [paper][code] :heavy_check_mark:
• [2025/09] DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners [paper][demo]
• [2025/09] Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding [paper][demo]
• [2025/08] Exploring Disentangled Neural Speech Codecs from Self-Supervised Representations [paper]
• [2025/08] DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models [paper][code][demo] :heavy_check_mark:
• [2025/08] NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference [paper][code][demo] :heavy_check_mark:
• [2025/08] SpectroStream: A Versatile Neural Codec for General Audio [paper]
• [2025/08] SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [paper][demo]
• [2025/07] HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling [paper][code]
• [2025/06] XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in LowBitrate Speech Codecs [paper][code]
• [2025/06] DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding [paper]
• [2025/06] CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate [paper][demo]
• [2025/06] USAD: Universal Speech and Audio Representation via Distillation [paper][HF]
• [2025/06] Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQ [paper][code][demo] :heavy_check_mark:
• [2025/06] LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization [paper]
• [2025/06] TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling [paper][code][demo] :heavy_check_mark:
• [2025/05] MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation [paper][code] :heavy_check_mark:
• [2025/05] SwitchCodec: A High-Fidelity Nerual Audio Codec With Sparse Quantization [paper]
• [2025/05] DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec [paper][demo]
• [2025/05] Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate [paper]
• [2025/05] PAST: Phonetic-Acoustic Speech Tokenizer [paper][code][demo] Code Comming Soon
• [2025/05] Universal Semantic Disentangled Privacy-preserving Speech Representation Learning [paper][demo]
• [2025/05] Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding [paper][code][demo] :heavy_check_mark:
• [2025/05] Toward a Sparse and Interpretable Audio Codec [paper][code][demo] :heavy_check_mark:
• [2025/04] DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation [paper][code][demo] :heavy_check_mark:
• [2025/04] ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling [paper][demo]
• [2025/04] A Streamable Neural Audio Codec with Residual Scalar-Vector Quantization for Real-Time Communication [paper][demo]
• [2025/04] One Quantizer is Enough: Toward a Lightweight Audio Codec [paper][code] SQCodec :heavy_check_mark:
• [2025/03] QINCODEC: Neural Audio Compression with Implicit Neural Codebooks [paper][demo]
• [2025/03] STFTCodec: High-Fidelity Audio Compression through Time-Frequency Domain Representation [paper]
• [2025/03] Designing Neural Synthesizers for Low Latency Interaction [paper][code][demo] :heavy_check_mark:
• [2025/03] Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens [paper][code][demo] BiCodec :heavy_check_mark:
• [2025/03] FlowDec: A flow-based full-band general audio codec with high perceptual quality [paper][code][demo] :heavy_check_mark:
• [2025/03] UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation [paper][demo]
• [2025/02] UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook [paper][code] Code Comming Soon
• [2023/08] RepCodec: A Speech Representation Codec for Speech Tokenization [paper][code] semantic speech tokenization :heavy_check_mark:
• [2024/04] SRCodec: Split-Residual Vector Quantization for Neural Speech Codec [paper][demo]
• [2024/09] Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference [paper][code][demo] :heavy_check_mark:
• [2025/02] FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks [paper][code][demo] :heavy_check_mark:
• [2025/02] Vector Quantization by Distribution Matching [paper][code] Wasserstein-VQ | codebook collapse :heavy_check_mark:
• [2024/12] Preventing Local Pitfalls in Vector Quantization via Optimal Transport [paper][code] OptVQ | codebook collapse :heavy_check_mark:
• [2025/02] ComplexDec: A Domain-robust High-fidelity Neural Audio Codec with Complex Spectrum Modeling [paper][demo]
• [2025/02] Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis [paper][code][infer-unofficial] X-Codec-2.0 :heavy_check_mark:
• [2025/01] MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization [paper][code] :heavy_check_mark:
• [2025/01] SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models [paper][code] Code Comming Soon
• [2024/12] FreeCodec: A disentangled neural speech codec with fewer tokens [paper][code][demo] Code Comming Soon | speaker encoder, content encoder and prosody encoder
• [2024/11] TS3-Codec: Transformer-Based Simple Streaming Single Codec [paper] free-convolution
• [2024/11] Scaling Transformers for Low-bitrate High-Quality Speech Coding [paper][code][demo] transformer-based and scale it into 1B parameter range :heavy_check_mark:
• [2024/11] PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain [paper][demo] Code Comming Soon | Music Tokenizer, Similar to MsCodec
• [2024/11] Wavehax: Aliasing-Free Neural Waveform Synthesis Based on 2D Convolution and Harmonic Prior for Reliable Complex Spectrogram Estimation [paper][code][demo] aliasing-free :heavy_check_mark:
• [2024/11] VChangeCodec: A High-efficiency Neural Speech Codec with Built-in Voice Changer for Real-time Communication [paper][demo] integrates the Voice Changer model directly into the speech Codec
• [2024/11] Towards Codec-LM Co-design for Neural Codec Language Models [paper] Code Comming Soon | proposing several codec-LM co-design strategies
• [2024/11] Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations [paper][IEEE] UniCodec ｜ several information-disentangled discrete tokens, similar to ns3_codec
• [2024/11] hertz-dev [code] WaveCodec :heavy_check_mark:
• [2024/11] SimVQ: Addressing Representation Collapse in Vector Quantized Models with One Linear Layer [paper][code] codebook collapse :heavy_check_mark:
• [2024/11] MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios [paper][demo] discrete cosine transform (MDCT) as input
• [2024/10] LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec [paper][demo]
• [2024/10] Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding [paper][demo] Code Comming Soon
• [2024/10] Pushing the frontiers of audio generation [blog] google deepmind
• [2024/11] DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models [paper] Double-Codebook Speaker-invariant Clustering
• [2024/10] A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation [paper][demo] Is predicting the remaining RVQ codes necessary?
• [2024/10] APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm [paper][demo] two-stage joint-individual training paradigm
• [2024/10] Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding [paper][demo] MsCodec, Multi-Scale Encoding
• [2024/10] LSCodec: Low-Bandwidth and Speaker-Decoupled Discrete Speech Codec [paper][demo] speaker timbre decouple
• [2024/10] DM-Codec: Distilling Multimodal Representations for Speech Tokenization [paper][code] acoustic properties, semantic meaning, and contextual clues :heavy_check_mark:
• [2024/10] ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs [paper][demo] address codebook collapse based on intra- and inter-codebook optimization
• [2024/10] Code Drift: Towards Idempotent Neural Audio Codecs [paper][demo] Idempotence – the stability of a codec’s decoded output under multiple rounds of encoding and decoding
• [2024/09] Learning Source Disentanglement in Neural Audio Codec [paper][code][demo] :heavy_check_mark:
• [2021/10] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing [paper][code] semantic information & content generation :heavy_check_mark:
• [2021/08] W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training [paper]
• [2021/06] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units [paper][code] semantic information & content generation :heavy_check_mark:
• [2020/06] wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations [paper][code] :heavy_check_mark:
• [2024/10] Low Bitrate High-Quality RVQGAN-based Discrete Speech Tokenizer [paper][code][demo] finetuned-version of DAC :heavy_check_mark:
• [2024/09] BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec [paper][code][demo] low-bitrate neural speech codec :heavy_check_mark:
• [2024/10] Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models [paper][demo] Inconsistency
• [2024/09] Reverse Engineering of Supervised Semantic Speech Tokenizer (S3Tokenizer) proposed in CosyVoice [code] S3Tokenizer :heavy_check_mark:
• [2024/09] FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates [paper] Flow Matching
• [2024/09] ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech [paper][code] Comprehensive Platform :heavy_check_mark:
• [2024/09] MuCodec: Ultra Low-Bitrate Music Codec [paper][code][demo] Music Codec :heavy_check_mark:
• [2024/09] Audio Codec Augmentation for Robust Collaborative Watermarking of Speech Synthesis [paper][code][demo] Watermarking :heavy_check_mark:
• [2024/09] NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization [paper][code] Code Comming Soon
• [2024/09] Speaking from Coarse to Fine: Improving Neural Codec Language Model via Multi-Scale Speech Coding and Generation [paper][demo] CoFi-Speech
• [2024/09] SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2024/08] Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model [paper][code][demo] X-Codec :heavy_check_mark:
• [2024/08] WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling [paper][code][demo] :heavy_check_mark:
• [2024/08] Music2Latent: Consistency Autoencoders for Latent Audio Compression [paper][code][demo] continuous latent space :heavy_check_mark:
• [2024/08] SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [paper][demo]
• [2024/06] SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models [paper][code][demo] SQ-Codec | Code Comming Soon
• [2024/02] Language-Codec: Reducing the Gaps Between Discrete Codec Representation and Speech Language Models [paper][code][demo] :heavy_check_mark:
• [2024/04] ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers [paper][code] :heavy_check_mark:
• [2024/07] SuperCodec: A Neural Speech Codec with Selective Back-Projection Network [paper][code][demo] :heavy_check_mark:
• [2024/07] dMel: Speech Tokenization made Simple [paper][code][demo] :heavy_check_mark:
• [2024/02] APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding [paper][code][demo] :heavy_check_mark:
• [2024/06] Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation [paper][demo]
• [2024/07] CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens [paper][code][demo] :heavy_check_mark:
• [2023/06] Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis [paper][code][demo] :heavy_check_mark:
• [2024/04] SNAC: Multi-Scale Neural Audio Codec [paper][code][demo] :heavy_check_mark:
• [2024/06] UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner [paper][code] LLM-Codec :heavy_check_mark:
• [2024/01] Finite Scalar Quantization: VQ-VAE Made Simple [paper][code] FSQ, no codebook collapse :heavy_check_mark:
• [2024/06] Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2023/09] Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer [paper]
• [2024/06] BiVocoder: A Bidirectional Neural Vocoder Integrating Feature Extraction and Waveform Generation [paper][demo]
• [2024/04] The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge [paper]
• [2023/06] UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding [paper][code][demo] acoustic model CTX-txt2vec and vocoder CTX-vec2wav | speech continuation and editing | similar to Encoder-Decoder :heavy_check_mark:
• [2024/06] Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder [paper]
• [2024/06] Coding Speech through Vocal Tract Kinematics [paper][code] :heavy_check_mark:
• [2024/05] HILCodec: High Fidelity and Lightweight Neural Audio Codec [paper][code][demo] :heavy_check_mark:
• [2024/04] SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound [paper][code][demo] :heavy_check_mark:
• [2024/01] Residual Quantization with Implicit Neural Codebooks [paper][code] Qinco :heavy_check_mark:
• [2024/01] SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models [paper][code][demo] :heavy_check_mark:
• [2024/01] Residual Quantization with Implicit Neural Codebooks [paper][code] :heavy_check_mark:
• [2023/10] Acoustic BPE for Speech Generation with Discrete Tokens [paper][code] :heavy_check_mark:
• [2023/09] BANC: Towards Efficient Binaural Audio Neural Codec for Overlapping Speech [paper][code][demo] :heavy_check_mark:
• [2023/09] Fewer-token Neural Speech Codec with Time-invariant Codes [paper][code][demo] Ti-Codec :heavy_check_mark:
• [2023/09] FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec [paper][code][demo] :heavy_check_mark:
• [2023/09] High Fidelity Neural Audio Compression [paper][code][code-Unofficial] [demo] Encodec :heavy_check_mark:
• [2023/09] Soundstorm: Efficient parallel audio generation [paper][demo]
• [2023/09] High-Fidelity Audio Compression with Improved RVQGAN [paper][code][demo] DAC :heavy_check_mark:
• [2023/09] SpatialCodec: Neural Spatial Speech Coding [paper][code][demo] :heavy_check_mark:
• [2023/05] HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec [paper][code] AcademiCodec & Group-RVQ :heavy_check_mark:
• [2023/05] AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec [paper][code][demo] :heavy_check_mark:
• [2023/01] InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt [paper][code][demo] :heavy_check_mark:
• [2022/09] AudioLM: a Language Modeling Approach to Audio Generation [paper][demo]
• [2021/07] SoundStream: An End-to-End Neural Audio Codec [paper][code][demo] :heavy_check_mark:

Zero-Shot Text-to-Speech Synthesizers

• [2025/12] GLM-TTS Technical Report [paper][code] :heavy_check_mark:
• [2025/12] DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec [paper][code][demo]
• [2025/12] M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2025/10] SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity [paper][code][demo] :heavy_check_mark:
• [2025/10] ParaStyleTTS: Toward Efficient and Robust Paralinguistic Style Control for Expressive Text-to-Speech Generation [paper][code][demo] :heavy_check_mark:
• [2025/10] neutts-air [code]
• [2025/10] Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech [paper][code][demo] :heavy_check_mark:
• [2025/09] DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptation [paper][code] :heavy_check_mark:
• [2025/09] Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook [paper][demo]
• [2025/09] VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning [code][demo]
• [2025/09] FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot [paper][code][demo] :heavy_check_mark:
• [2025/08] AudioStory: Generating Long-Form Narrative Audio with Large Language Models [paper][code] :heavy_check_mark:
• [2025/08] VibeVoice: A Frontier Open-Source Text-to-Speech Model [paper][code][demo] :heavy_check_mark:
• [2025/08] NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations [paper][code][demo] :heavy_check_mark:
• [2025/08] Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech [paper][demo]
• [2025/08] A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding [paper][code][demo] :heavy_check_mark:
• [2025/07] TTS-1 Technical Report [paper][code][demo] :heavy_check_mark:
• [2025/07] MOSS: Text to Spoken Dialogue Generation [code][demo]
• [2025/06] Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis [paper][code]
• [2025/06] IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech [paper][code][demo] :heavy_check_mark:
• [2025/06] ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching [paper][code][demo] :heavy_check_mark:
• [2025/06] VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation [paper][code][demo] :heavy_check_mark:
• [2025/06] Voice Impression Control in Zero-Shot TTS [paper][demo]
• [2025/05] Chatterbox TTS [code][demo]
• [2025/05] CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training [paper][demo]
• [2025/05] UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information [paper][code] Code Comming Soon
• [2025/05] Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis [paper][code] Code Comming Soon
• [2025/05] MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder [paper][demo]
• [2025/05] FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech [paper][demo]
• [2025/04] Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget [paper][code] :heavy_check_mark:
• [2025/04] SLED-TTS: Efficient Speech Language Modeling via Energy Distance in Continuous Space [code][demo]
• [2025/04] dia [code][demo]
• [2025/04] EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting [paper][demo] Code Comming Soon
• [2025/04] Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis [paper][demo]
• [2025/04] AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesis [paper][demo]
• [2025/04] RWKVTTS: Yet another TTS based on RWKV-7 [paper][code] :heavy_check_mark:
• [2025/04] F5R-TTS: Improving Flow Matching based Text-to-Speech with Group Relative Policy Optimization [paper][demo]
• [2025/04] MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2025/03] FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System [paper][demo]
• [2025/03] MoonCast: High-Quality Zero-Shot Podcast Generation [paper][code][demo] :heavy_check_mark:
• [2025/03] Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens [paper][code][demo] :heavy_check_mark:
• [2025/02] Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement [paper][code][demo] :heavy_check_mark:
• [2025/02] Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [paper][demo]
• [2025/02] SyncSpeech: Low-Latency and Efficient Dual-Stream Text-to-Speech based on Temporal Masked Transformer [paper][demo] Code Comming Soon
• [2025/02] FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching [paper][demo]
• [2025/02] IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System [paper][code][demo] :heavy_check_mark:
• [2025/02] Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance [paper][demo] Code Comming Soon
• [2025/02] Beta Release of Zonos-v0.1 [blog][code] :heavy_check_mark:
• [2025/01] Kokoro-TTS [checkpoints][huggingface]
• [2025/01] MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Model [paper][demo] Code Comming Soon
• [2025/01] DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions [paper][code][demo] :heavy_check_mark:
• [2025/01] TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch [paper] data scaling and deployment efficiency
• [2024/12] CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation [paper][demo]
• [2024/12] VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis [paper][demo] environment-aware speech synthesis
• [2024/12] Autoregressive Speech Synthesis with Next-Distribution Prediction [paper][demo] KALL-E
• [2024/12] ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis [paper][code][demo] Code Comming Soon
• [2024/12] Interleaved Speech-Text Language Models are Simple Streaming Text to Speech Synthesizers [paper] IST-LM
• [2024/12] CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models [paper][code][demo] :heavy_check_mark:
• [2024/12] TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch [paper]
• [2024/11] Visatronic: A Multimodal Decoder-Only Model for Speech Synthesis [paper] Code Comming Soon | Text & Video to Speech
• [2024/11] Debatts: Zero-Shot Debating Text-to-Speech Synthesis [paper][demo] Debating TTS & Dataset
• [2024/11] OuteTTS-0.1-350M [blog][code] :heavy_check_mark:
• [2024/12] The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024 [paper] ISCSLP 2024
• [2024/10] The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings [paper] zero-shot spontaneous style voice cloning | ISCSLP 2024
• [2024/07] ICAGC 2024: Inspirational and Convincing Audio Generation Challenge 2024 [paper] emotional & background audio generation | ISCSLP 2024
• [2024/11] Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis [paper][code] :heavy_check_mark:
• [2024/10] STTATTS: Unified Speech-To-Text And Text-To-Speech Model [paper][code]
• [2024/10] SPIRIT LM: Interleaved Spoken and Written Language Model [paper][code][demo] :heavy_check_mark:
• [2023/05] Better speech synthesis through scaling [paper][code][blog] Tortoise TTS :heavy_check_mark:
• [2024/10] F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching [paper][code][demo] :heavy_check_mark:
• [2024/09] Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models [paper][demo]
• [2024/09] FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications [paper][code][demo] voice cloning for dubbing and human-like speech generation for chatbots :heavy_check_mark:
• [2024/09] MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer [paper][code][demo] Masked Generative Model | Similar to Seed-TTS :heavy_check_mark:
• [2024/08] VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling [paper][code][demo] :heavy_check_mark:
• [2024/08] Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation [paper][demo]
• [2024/04] FlashSpeech: Efficient Zero-Shot Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2024/07] CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens [paper] [code][demo] :heavy_check_mark:
• [2024/07] Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization [paper][demo] Human FeedBack
• [2024/06] E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS [paper][demo] similar to Seed-TTS
• [2023/11] HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2024/06] TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers [paper][code][demo] :heavy_check_mark:
• [2024/01] CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech [paper][demo]
• [2024/06] DiTTo-TTS: Efficient and Scalable Zero-Shot Text-to-Speech with Diffusion Transformer [paper][demo]
• [2024/06] VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment [paper][demo]
• [2024/06] Autoregressive Diffusion Transformer for Text-to-Speech Synthesis [paper][demo]
• [2024/06] VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers [paper][demo]
• [2024/06] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model [paper][code][demo] :heavy_check_mark:
• [2024/06] ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec [paper][code][demo] :heavy_check_mark:
• [2024/08] SSL-TTS: Leveraging Self-Supervised Embeddings and kNN Retrieval for Zero-Shot Multi-speaker TTS [paper][demo] SSL
• [2024/08] VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech [paper][demo] LORA
• [2024/08] StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speech [paper][demo] LORA
• [2024/08] EELE: Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech [paper] LORA
• [2024/07] Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models [paper][demo] Spontaneous
• [2024/01] EmotiVoice 😊: a Multi-Voice and Prompt-Controlled TTS Engine [code] :heavy_check_mark:
• [2024/06] Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment [paper][demo] Monotonic Alignment
• [2024/01] Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction [paper][demo] Transducer/End-to-End
• [2024/01] VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech [paper][code][demo] Code Comming Soon | Transducer
• [2024/06] High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model [paper][demo] Transducer/End-to-End
• [2023/02] Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision [paper][code][demo] SpearTTS | WhisperSpeech :heavy_check_mark:
• [2024/02] Natural language guidance of high-fidelity text-to-speech with synthetic annotations [paper][code][demo] Prompt Control | Parler-TTS :heavy_check_mark:
• [2024/06] WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark [paper][demo]
• [2024/06] Seed-TTS: A Family of High-Quality Versatile Speech Generation Models [paper][demo]
• [2024/06] Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback [paper] Human Feedback
• [2024/04] SpeechAlign: Aligning Speech Generation to Human Preferences [paper][code][demo] Human Feedback :heavy_check_mark:
• [2024/04] StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations [paper][code][demo] Lian Liru(连丽如) dataset :heavy_check_mark:
• [2024/04] TextrolSpeech: A Text Style Control Speech Corpus with Codec Language Text-to-Speech Models [paper][code][demo] Code Comming Soon
• [2024/03] HAM-TTS: Hierarchical Acoustic Modeling for Token-Based Zero-Shot Text-to-Speech with Model and Data Scaling [paper][demo]
• [2024/01] Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis [paper][demo]
• [2024/03] NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models [paper][demo]
• [2024/01] NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers [paper][demo]
• [2024/03] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild [paper][code][demo] :heavy_check_mark:
• [2023/01] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers [paper][code][demo] VALL-E :heavy_check_mark:
• [2023/09] PromptTTS 2: Describing and Generating Voices with Text Prompt [paper][code][demo] :heavy_check_mark:
• [2023/09] Matcha-tts: A fast tts architecture with conditional flow matching [paper][code][demo] :heavy_check_mark:
• [2023/09] Voicebox: Text-guided multilingual universal speech generation at scale [paper][demo]
• [2023/09] Voiceflow: Efficient text-to-speech with rectified flow matching [paper][code][demo] :heavy_check_mark:
• [2023/05] Better speech synthesis through scaling [paper][code] tortoise-tts :heavy_check_mark:

Speech Language Models

• WavChat classify all spoken dialogue models based on whether the core language model can directly understand and generate speech representations, dividing them into cascaded and end-to-end categories.

End-to-End (Paradigms) Models

• [2025/12] Fun-ASR-Nano-2512: LLM-based multilingual ASR model [code][model] :heavy_check_mark:
• [2025/11] Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation [paper][code][demo] :heavy_check_mark:
• [2025/10] Neural audio codecs: how to get audio into LLMs [blog][code]
• [2025/08] OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue [paper][code]
• [2025/08] Think Before You Talk: Enhancing Meaningful Dialogue Generation in Full-Duplex Speech Language Models with Planning-Inspired Text Guidance [paper][code][demo] :heavy_check_mark:
• [2025/08] MiDashengLM: Efficient Audio Understanding with General Audio Captions [paper][code][demo] :heavy_check_mark:
• [2025/07] ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching [paper][code][demo] :heavy_check_mark:
• [2025/06] vui [code][demo]
• [2025/06] Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model [paper]
• [2025/06] Towards a Japanese Full-duplex Spoken Dialogue System [paper][code][demo] :heavy_check_mark:
• [2025/06] CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching [paper][demo]
• [2025/06] NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction [paper][code][demo] :heavy_check_mark:
• [2025/05] Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model [paper][demo] Code Comming Soon
• [2025/05] Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space [paper][code] :heavy_check_mark:
• [2025/05] VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model [paper][code] :heavy_check_mark:
• [2025/05] LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis [paper][code][demo] :heavy_check_mark:
• [2025/04] Kimi-Audio Technical Report [paper][code]
• [2025/04] TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling [paper][code][demo] :heavy_check_mark:
• [2025/04] VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation [paper] Code Comming Soon
• [2025/03] Qwen2.5-Omni Technical Report [paper][code] :heavy_check_mark:
• [2025/02] Conversational speech generation [blog][code] :heavy_check_mark:
• [2025/02] Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction [paper][code] :heavy_check_mark:
• [2025/02] Slamming: Training a Speech Language Model on One GPU in a Day [paper][code][demo] :heavy_check_mark:
• [2025/02] ESPnet-SpeechLM: An Open Speech Language Model Toolkit [paper][code] :heavy_check_mark:
• [2025/02] Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction [paper][code] :heavy_check_mark:
• [2025/01] Baichuan-Omni-1.5 Technical Report [paper][code] Baichuan-Omni-1.5 :heavy_check_mark:
• [2025/01] A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming on Your Phone [blog][code] MiniCPM-o :heavy_check_mark:
• [2025/01] MinMo: A Multimodal Large Language Model for Seamless Voice Interaction [paper][demo] Code Comming Soon
• [2025/01] VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction [paper][code] :heavy_check_mark:
• [2025/01] OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios [paper][demo] including ShareChatX dataset
• [2025/01] SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation [paper][demo]
• [2024/12] Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data [paper][demo] Ke-SpeechChat & KE-Omni | Code Comming Soon
• [2024/12] Long-Form Speech Generation with Spoken Language Models [paper][demo] SpeechSSM, Long-Form Generation
• [2024/12] SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training [paper][code][demo] :heavy_check_mark:
• [2024/12] Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners [paper][demo] Flow-Omni, continuous speech tokens
• [2024/02] Paralinguistics-Aware Speech-Empowered LLMs for Natural Conversation [paper][code][demo] learning cross-modal distributional semantics :heavy_check_mark:
• [2024/12] GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot [paper][code] speech interaction model & emotion, intonation, speech rate, and dialect & low latency :heavy_check_mark:
• [2024/11] MooER: Moore-threads Open Omni model for speech-to-speech intERaction [code] Paper Comming Soon
• [2024/11] SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation [paper] Code Comming Soon | free-codec
• [2024/11] Building a Taiwanese Mandarin Spoken Language Model: A First Attempt [paper][code] Code Comming Soon
• [2024/11] hertz-dev [code] :heavy_check_mark:
• [2024/11] Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM [paper][demo][code] frozen llm in training :heavy_check_mark:
• [2024/10] Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities [paper][code] :heavy_check_mark:
• [2024/10] IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities [paper][demo] reducing the length difference between speech and text
• [2024/10] OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [paper][demo] Code Comming Soon
• [2024/09] Westlake-Omni: Open-Source Chinese Emotional Speech Interaction Large Language Model with Unified Discrete Sequence Modeling [code] :heavy_check_mark:
• [2024/09] Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control [paper][demo]
• [2024/09] Moshi: a speech-text foundation model for real time dialogue [paper][code][demo] low delay | only english :heavy_check_mark:
• [2024/09] LLaMA-Omni: Seamless Speech Interaction with Large Language Models [paper][code][demo] only english :heavy_check_mark:
• [2024/09] EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions [paper][demo]
• [2024/08] Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming [paper][code] End-to-End | speech interaction model :heavy_check_mark:
• [2024/08] Speech To Speech: an effort for an open-sourced and modular GPT4-o [code] End-to-End | speech interaction model :heavy_check_mark:
• [2024/08] Language Model Can Listen While Speaking [paper][demo] Full Duplex Modeling | speech interaction model
• [????/??] SpeechGPT2: End-to-End Human-Like Spoken Chatbot [paper][code][demo] paper & Code Comming Soon | speech interaction model
• [2024/01] SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation [paper][demo] Code Comming Soon
• [2023/05] SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities [paper][code][demo] :heavy_check_mark:
• [2024/07] Generative Expressive Conversational Speech Synthesis [paper][code] GPT-Talker ｜ GPT for response and Style, VITS for audio :heavy_check_mark:
• [2024/06] GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities [paper][code][demo] :heavy_check_mark:
• [2024/02] Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities [paper][code][demo] :heavy_check_mark:
• [2024/02] AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling [paper][code][demo] :heavy_check_mark:
• [2024/03] WavLLM: Towards Robust and Adaptive Speech Large Language Model [paper][code] :heavy_check_mark:
• [2024/08] DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidance [paper][demo]
• [2024/08] Style-Talker: Finetuning Audio Language Model and StyleBased Text-to-Speech Model for Fast Spoken Dialogue Generation [paper][code][demo] :heavy_check_mark:
• [2024/04] CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations [paper][code][demo] multi-round dialogue speech generation :heavy_check_mark:

Cascaded (Paradigms) Models

• [2025/03] Mellow: a small audio language model for reasoning [paper][code] :heavy_check_mark:
• [2024/11] A fast multimodal LLM for real-time voice [blog][code][demo] Ultravox :heavy_check_mark:
• [2024/10] Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant [paper][code] :heavy_check_mark:
• [2024/10] Ocean-omni: To Understand the World with Omni-modality [paper][code] Baichuan-Omni :heavy_check_mark:
• [2024/08] VITA: Towards Open-Source Interactive Omni Multimodal LLM [paper][code][demo] :heavy_check_mark:
• [2024/07] Qwen2-Audio Technical Report [paper][code] :heavy_check_mark:
• [2024/05] A Full-duplex Speech Dialogue Scheme Based On Large Language Model [paper] neural finite state machine
• [2023/11] Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models [paper][code] :heavy_check_mark:
• [2023/10] SALMONN: Towards Generic Hearing Abilities for Large Language Models [paper][code] :heavy_check_mark:

Benchmark

• [2025/09] CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation [paper][code] :heavy_check_mark:
• [2025/08] Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models [paper][code]
• [2025/07] FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems [paper][code]
• [2025/06] InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems [paper][code]
• [2025/05] WavReward: Spoken Dialogue Models With Generalist Reward Evaluators [paper][code] Code Comming Soon
• [2025/03] Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [paper][code]
• [2025/03] Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics [paper] Code Comming Soon
• [2025/02] URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models [paper]
• [2023/09] Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech [paper]
• [2024/02] Codec-SUPERB: An In-Depth Analysis of Sound Codec Models [paper][code]
• [2024/07] EMO-Codec: A Depth Look at Emotion Preservation Capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations [paper]
• [2024/06] DASB - Discrete Audio and Speech Benchmark [paper][code] a benchmark for evaluating discrete audio representations
• [2024/12] Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [paper]

Survey

• [2025/06] Discrete Audio Tokens: More Than a Survey! [paper][demo]
• [2025/04] On The Landscape of Spoken Language Models: A Comprehensive Survey [paper]
• [2025/02] Recent Advances in Discrete Speech Tokens: A Review [paper]
• [2024/12] Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey [paper][code]
• [2024/12] Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey [paper][code]
• [2024/11] WavChat: A Survey of Spoken Dialogue Models [paper][code]
• [2024/10] Recent Advances in Speech Language Models: A Survey [paper]
• [2024/10] A Survey on Speech Large Language Models [paper]
• [2024/02] Towards audio language modeling -- an overview [paper]

Some Interesting Models

• [2025/07] Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models [paper][code][demo] :heavy_check_mark:
• [2025/06] Speech-Language Models with Decoupled Tokenizers and Multi-Token Prediction [paper][code][demo] Code Comming Soon
• [2025/06] Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model [paper][code] :heavy_check_mark:
• [2025/06] Ming-Omni: A Unified Multimodal Model for Perception and Generation [paper][code][demo] :heavy_check_mark:
• [2025/05] FLAM: Frame-Wise Language-Audio Modeling [paper][demo]
• [2025/05] ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting [paper][demo]
• [2025/05] FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing [paper][demo]
• [2025/04] Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Human-like Audiobook Generation [paper][demo]
• [2025/03] WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching [paper][code][demo] :heavy_check_mark:
• [2025/03] Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering [paper][code] :heavy_check_mark:
• [2025/03] FilmComposer: LLM-Driven Music Production for Silent Film Clips [paper][code][demo] Code Comming Soon
• [2023/07] WavJourney: Compositional Audio Creation with Large Language Models [paper][code][demo] :heavy_check_mark:
• [2025/03] PodAgent: A Comprehensive Framework for Podcast Generation [paper][code][demo] :heavy_check_mark:
• [2024/09] Text2FX: Harnessing CLAP Embeddings for Text-Guided Audio Effects [paper][code][demo] :heavy_check_mark:
• [2025/02] DisCoder: High-Fidelity Music Vocoder Using Neural Audio Codecs [paper][code][demo] :heavy_check_mark:
• [2024/08] PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation [paper][code][demo] :heavy_check_mark:
• [2025/02] Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment [paper][code][demo] :heavy_check_mark:
• [2024/06] Scaling up masked audio encoder learning for general audio classification [paper][code] DaSheng :heavy_check_mark:
• [2025/02] Metis: A Foundation Speech Generation Model with Masked Generative Pre-training [paper][code][demo] :heavy_check_mark:
• [2025/01] CodecFake-Omni: A Large-Scale Codec-based Deepfake Speech Dataset [paper] Code Comming Soon
• [2024/12] TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization [paper][code][demo] :heavy_check_mark:
• [2024/12] Flow Matching Guide and Code [paper][code]
• [2024/12] OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows [paper][code] :heavy_check_mark:
• [2024/09] SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis [paper][code][demo] :heavy_check_mark:
• [2024/11] FLowHigh: Towards efficient and high-quality audio super-resolution with single-step flow matching [code][demo] :heavy_check_mark:
• [2024/11] O1 Replication Journey: A Strategic Progress Report -- Part 1 [paper][code] :heavy_check_mark:
• [2024/11] LLaMA-O1: Open Large Reasoning Model Frameworks For Training, Inference and Evaluation With PyTorch and HuggingFace [code] :heavy_check_mark:
• [2024/07] SenseVoice: Multilingual speech understanding with ASR, emotion recognition, and audio event detection [paper][code][model] :heavy_check_mark:
• [2024/07] Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation [paper][code][demo] :heavy_check_mark:
• [2024/07] Stable Audio Open [paper] [code] :heavy_check_mark:
• [2024/05] EmoLLM(心理健康大模型) [code][demo] :heavy_check_mark:
• [2023/02] Improving and generalizing flow-based generative models with minibatch optimal transport [paper][code] TorchCFM | Tutorials :heavy_check_mark:
• [2022/10] Flow Matching for Generative Modeling [paper] Conditional Flow Matching
• [2022/09] Rectified Flow: A Marginal Preserving Approach to Optimal Transport [paper][code] :heavy_check_mark:

Music Generation

• [2025/08] Live Music Models [paper][code]
• [2025/06] LeVo: High-Quality Song Generation with Multi-Preference Alignment [paper][code][demo] :heavy_check_mark:
• [2025/05] ACE-Step: A Step Towards Music Generation Foundation Model [paper][code][demo :heavy_check_mark:
• [2025/04] Versatile Framework for Song Generation with Prompt-based Control [paper][demo]
• [2025/04] MusFlow: Multimodal Music Generation via Conditional Flow Matching [paper][demo] Code Comming Soon
• [2025/04] A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives: Data, Methods, and Challenges [paper]
• [2025/03] Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation [paper][demo] MusiCoT
• [2025/03] DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion [paper][code][demo] :heavy_check_mark:
• [2025/02] SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [paper][code][demo] :heavy_check_mark:
• [2025/02] TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching [paper][code][demo] :heavy_check_mark:
• [2025/02] CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages [paper][code][demo] :heavy_check_mark:
• [2025/02] YuE: Open Music Foundation Models for Full-Song Generation [paper][code][demo] :heavy_check_mark:
• [2025/01] InspireMusic: A Unified Framework for Music, Song, Audio Generation [paper][code][demo] :heavy_check_mark:
• [2024/12] SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor [paper][demo]
• [2024/12] MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models [paper][code] :heavy_check_mark:
• [2024/10] MusicFlow: Cascaded Flow Matching for Text Guided Music Generation [paper] Code Comming Soon | Similar to MaskGCT
• [2024/09] FLUX that Plays Music [paper][code][melodio] KunLun :heavy_check_mark:
• [2024/09] Seed-Music: A Unified Framework for High Quality and Controlled Music Generation [paper][demo] tech-report
• [2024/05] QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation [paper][code][demo] :heavy_check_mark:
• [2024/05] Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning [paper][code][demo] Instruction Tuning :heavy_check_mark:
• [2023/06] Simple and Controllable Music Generation [paper][code] Prompt Control | AudioCraft :heavy_check_mark:

Speech DataSet

• [2025/09] SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding [paper][code] :heavy_check_mark:
• [2025/09] SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models [paper]
• [2025/06] HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset [paper][dataset]
• [2025/04] DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue [paper][code][demo] synthetic data :heavy_check_mark:
• [2025/04] SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning [paper]
• [2025/03] Scaling Rich Style-Prompted Text-to-Speech Datasets [paper][code][demo] :heavy_check_mark:
• [2025/02] CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition [paper]
• [2025/02] Audio-FLAN: A Preliminary Release [paper][code][dataset] :heavy_check_mark:
• [2024/08] SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description [paper][code][demo] :heavy_check_mark:
• [2024/07] Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation [paper][code][demo][dataset] :heavy_check_mark:
• [2024/06] WenetSpeech4TTS: A 12,800-hour Mandarin TTS corpus for large speech generation model benchmark [paper][demo][dataset] :heavy_check_mark:
• [2020/10] Didispeech: A large scale Mandarin speech corpus [paper][code][demo][dataset]

Some Interesting knowledge

Blog & Courses

• Anthropic courses [github]
• LLM101n: Let's build a Storyteller [github]
• Build a Large Language Model (From Scratch) [github]
• build nanoGPT from Karpathy [github]

Minor Points of Concern

GitHub

• ChatTTS: https://github.com/2noise/ChatTTS/tree/main
• OpenVoice: https://github.com/myshell-ai/OpenVoice
• GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
• Bert-vits2-NoBug: https://github.com/ywh-my/Bert-vits2-NoBug
• VoiceCraft: https://github.com/jasonppy/VoiceCraft
• YourTTS: https://github.com/Edresson/YourTTS
• Coqui: https://github.com/coqui-ai/TTS
• ebook2audiobookXTTS: https://github.com/DrewThomasson/ebook2audiobookXTTS
• MARS5-TTS: https://github.com/Camb-ai/MARS5-TTS
• edge-tts: https://github.com/rany2/edge-tts
• metavoice-src: https://github.com/metavoiceio/metavoice-src
• StyleTTS2: https://github.com/yl4579/StyleTTS2
• open-tts-tracker: https://github.com/Vaibhavs10/open-tts-tracker
• Amphion: https://github.com/open-mmlab/Amphion
• CTranslate2: https://github.com/OpenNMT/CTranslate2
• CFM: https://github.com/atong01/conditional-flow-matching
• speech-trident: https://github.com/ga642381/speech-trident
• bark: https://github.com/suno-ai/bark
• LangGPT: https://github.com/langgptai/LangGPT (提示词工程)
• composio: https://github.com/ComposioHQ/composio
• torchdiffeq: https://github.com/rtqichen/torchdiffeq
• podlm: https://github.com/lihuithe/podlm-public (NoteBookLM 的平替)
• NotebookLlama: https://github.com/meta-llama/llama-recipes/tree/main/recipes/quickstart/NotebookLlama (类似 NoteBookLM)
• playnote: https://play.ai/playnote (类似 NotebookLM)
• podcastfy: https://github.com/souzatharsis/podcastfy (类似 NotebookLM)
• dify: https://github.com/langgenius/dify (开源的 LLM 应用开发平台)
• Awesome-Dify-Workflow: https://github.com/svcvit/Awesome-Dify-Workflow
• LiblibAI: https://www.liblib.art (AI创作平台)

Nice Tool

• pytorch-OpCounter: https://github.com/Lyken17/pytorch-OpCounter
• rich: https://github.com/Textualize/rich
• argbind: https://github.com/pseeth/argbind/
• audiotools: https://github.com/descriptinc/audiotools
• hydra: https://github.com/facebookresearch/hydra
• joblib: https://github.com/joblib/joblib
• einops: https://github.com/arogozhnikov/einops
• safetensors: https://github.com/huggingface/safetensors
• OpenDiloco: https://github.com/PrimeIntellect-ai/OpenDiloco
• WeTextProcessing: https://github.com/wenet-e2e/WeTextProcessing
• zed: https://github.com/zed-industries/zed
• weekly: https://github.com/ljinkai/weekly
• tinygrad: https://github.com/tinygrad/tinygrad
• ffmpeg-normalize: https://github.com/slhck/ffmpeg-normalize
• kohya_ss: https://github.com/bmaltais/kohya_ss
• Lora-Training-in-Comfy: https://github.com/LarryJane491/Lora-Training-in-Comfy
• ComfyUI-Manager: https://github.com/ltdrdata/ComfyUI-Manager
• ComfyUI: https://github.com/comfyanonymous/ComfyUI
• comfyui-workspace-manager: https://github.com/11cafe/comfyui-workspace-manager
• CosyVoice+ComfyUI: https://github.com/AIFSH/CosyVoice-ComfyUI
• ComfyUI-wiki: https://github.com/602387193c/ComfyUI-wiki
• ZHO: https://github.com/ZHO-ZHO-ZHO
• tmux: https://github.com/tmux/tmux
• LoRAlib: https://github.com/microsoft/LoRA
• codespaces: https://github.com/codespaces
• Foliate(PDF): https://johnfactotum.github.io/foliate/
• Okular(PDF): https://okular.kde.org/zh-cn/
• audioFlux: https://github.com/libAudioFlux/audioFlux
• PyWavelets: https://github.com/PyWavelets/pywt
• 智能体或工作流平台: https://ai-bot.cn/ai-agent-development-platform/
• open-webui: https://github.com/open-webui/open-webui
• qwen-2.5-code-interpreter: https://github.com/cfahlgren1/qwen-2.5-code-interpreter
• ollama: https://github.com/ollama/ollama; https://ollama.com/
• vllm: https://github.com/vllm-project/vllm
• anythingLLM: https://github.com/Mintplex-Labs/anything-llm
• Windsurf: https://codeium.com/windsurf
• cursor: https://www.cursor.com/
• docling: https://github.com/DS4SD/docling
• TEN-Agent: https://github.com/TEN-framework/TEN-Agent

Reference

• [ZhiHu]别慌！一文教你看懂GPT-4o背后的语音技术
• [ZhiHu]百花齐放的Audio Codec: 语音合成利器
• [InterSpeech2024]InterSpeech2024 Speech Processing Using Discrete Speech Units : https://www.wavlab.org/activities/2024/Interspeech2024-Discrete-Speech-Unit-Challenge/ : https://huggingface.co/discrete-speech : arxiv 2024 : [paper]
• [Slides]Challenges in Developing Spoken Language Models slides
• [GitHub]speech-trident : Awesome speech/audio LLMs, representation learning, and codec models
• [GitHub]Awesome-Speech-Language-Model : Paper, Code and Resources for Speech Language Model and End2End Speech Dialogue System