archinetai/audio-ai-timeline

A timeline of the latest AI models for audio generation, starting in 2023!

1,903

59 commits

updated Jan 4, 2024

See the code

README

Audio AI Timeline

Here we will keep track of the latest AI models for waveform based audio generation, starting in 2023!

2023

Date	Release [Samples]	Paper	Code	Trained Model
14.11	Mustango: Toward Controllable Text-to-Music Generation	arXiv	GitHub	Hugging Face
13.11	Music ControlNet: Multiple Time-varying Controls for Music Generation	arXiv	-	-
02.11	E3 TTS: Easy End-to-End Diffusion-based Text to Speech	arXiv	-	-
01.10	UniAudio: An Audio Foundation Model Toward Universal Audio Generation	arXiv	GitHub	-
24.09	VoiceLDM: Text-to-Speech with Environmental Context	arXiv	GitHub	-
05.09	PromptTTS 2: Describing and Generating Voices with Text Prompt	arXiv	-	-
14.08	SpeechX: Neural Codec Language Model as a Versatile Speech Transformer	arXiv	-	-
10.08	AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining	arXiv	GitHub	Hugging Face
09.08	JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models	arXiv	-	-
03.08	MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies	arXiv	GitHub	-
14.07	Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts	arXiv	-	-
10.07	VampNet: Music Generation via Masked Acoustic Token Modeling	arXiv	GitHub	-
22.06	AudioPaLM: A Large Language Model That Can Speak and Listen	arXiv	-	-
19.06	Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale	PDF	GitHub	-
08.06	MusicGen: Simple and Controllable Music Generation	arXiv	GitHub	Hugging Face Colab
06.06	Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias	arXiv	-	-
01.06	Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis	arXiv	GitHub	-
29.05	Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation	arXiv	-	-
25.05	MeLoDy: Efficient Neural Music Generation	arXiv	-	-
18.05	CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training	arXiv	-	-
18.05	SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities	arXiv	GitHub	-
16.05	SoundStorm: Efficient Parallel Audio Generation	arXiv	GitHub (unofficial)	-
03.05	Diverse and Vivid Sound Generation from Text Descriptions	arXiv	-	-
02.05	Long-Term Rhythmic Video Soundtracker	arXiv	GitHub	-
24.04	TANGO: Text-to-Audio generation using instruction tuned LLM and Latent Diffusion Model	PDF	GitHub	Hugging Face
18.04	NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers	arXiv	GitHub (unofficial)	-
10.04	Bark: Text-Prompted Generative Audio Model	-	GitHub	Hugging Face Colab
03.04	AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models	arXiv	-	-
08.03	VALL-E X: Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling	arXiv	-	-
27.02	I Hear Your True Colors: Image Guided Audio Generation	arXiv	GitHub	-
08.02	Noise2Music: Text-conditioned Music Generation with Diffusion Models	arXiv	-	-
04.02	Multi-Source Diffusion Models for Simultaneous Music Generation and Separation	arXiv	GitHub	-
30.01	SingSong: Generating musical accompaniments from singing	arXiv	-	-
30.01	AudioLDM: Text-to-Audio Generation with Latent Diffusion Models	arXiv	GitHub	Hugging Face
30.01	Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion	arXiv	GitHub	-
29.01	Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models	PDF	-	-
28.01	Noise2Music	-	-	-
27.01	RAVE2 [Samples RAVE1]	arXiv	GitHub	-
26.01	MusicLM: Generating Music From Text	arXiv	GitHub (unofficial)	-
18.01	Msanii: High Fidelity Music Synthesis on a Shoestring Budget	arXiv	GitHub	Hugging Face Colab
16.01	ArchiSound: Audio Generation with Diffusion	arXiv	GitHub	-
05.01	VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers	arXiv	GitHub (unofficial) (demo)	-

artificial-intelligence

audio-generation

machine-learning

Contributors

43 commits

13 commits

1 commits

1 commits

Similar repos

affige/genmusic_demo_list

a list of demo websites for automatic music generation research

799

EmulationAI/awesome-large-audio-models

Collection of resources on the applications of Large Language Models (LLMs) in Audio AI.

741

ga642381/speech-trident

Awesome speech/audio LLMs, representation learning, and codec models

1,248

AI-Guru/music-generation-research

A straightforward collection of Music Generation research resources.

605

LqNoob/Neural-Codec-and-Speech-Language-Models

Awesome Neural Codec Models, Text-to-Speech Synthesizers & Speech Language Models

248

BinWang28/audio-ai-hub

The hub for audio AI research: papers, open models, benchmarks & datasets across audio LLMs, speech…

958

kadirnar/awesome-tts-architectures

A visual catalog of text-to-speech architectures, with model diagrams, concise descriptions, and…

haoheliu/audioldm2

Text-to-Audio/Music Generation

2,644

archinetai/audio-ai-timeline

A timeline of the latest AI models for audio generation, starting in 2023!

1,903

59 commits

updated Jan 4, 2024

See the code

README

Audio AI Timeline

Here we will keep track of the latest AI models for waveform based audio generation, starting in 2023!

2023

Date	Release [Samples]	Paper	Code	Trained Model
14.11	Mustango: Toward Controllable Text-to-Music Generation	arXiv	GitHub	Hugging Face
13.11	Music ControlNet: Multiple Time-varying Controls for Music Generation	arXiv	-	-
02.11	E3 TTS: Easy End-to-End Diffusion-based Text to Speech	arXiv	-	-
01.10	UniAudio: An Audio Foundation Model Toward Universal Audio Generation	arXiv	GitHub	-
24.09	VoiceLDM: Text-to-Speech with Environmental Context	arXiv	GitHub	-
05.09	PromptTTS 2: Describing and Generating Voices with Text Prompt	arXiv	-	-
14.08	SpeechX: Neural Codec Language Model as a Versatile Speech Transformer	arXiv	-	-
10.08	AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining	arXiv	GitHub	Hugging Face
09.08	JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models	arXiv	-	-
03.08	MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies	arXiv	GitHub	-
14.07	Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts	arXiv	-	-
10.07	VampNet: Music Generation via Masked Acoustic Token Modeling	arXiv	GitHub	-
22.06	AudioPaLM: A Large Language Model That Can Speak and Listen	arXiv	-	-
19.06	Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale	PDF	GitHub	-
08.06	MusicGen: Simple and Controllable Music Generation	arXiv	GitHub	Hugging Face Colab
06.06	Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias	arXiv	-	-
01.06	Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis	arXiv	GitHub	-
29.05	Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation	arXiv	-	-
25.05	MeLoDy: Efficient Neural Music Generation	arXiv	-	-
18.05	CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training	arXiv	-	-
18.05	SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities	arXiv	GitHub	-
16.05	SoundStorm: Efficient Parallel Audio Generation	arXiv	GitHub (unofficial)	-
03.05	Diverse and Vivid Sound Generation from Text Descriptions	arXiv	-	-
02.05	Long-Term Rhythmic Video Soundtracker	arXiv	GitHub	-
24.04	TANGO: Text-to-Audio generation using instruction tuned LLM and Latent Diffusion Model	PDF	GitHub	Hugging Face
18.04	NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers	arXiv	GitHub (unofficial)	-
10.04	Bark: Text-Prompted Generative Audio Model	-	GitHub	Hugging Face Colab
03.04	AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models	arXiv	-	-
08.03	VALL-E X: Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling	arXiv	-	-
27.02	I Hear Your True Colors: Image Guided Audio Generation	arXiv	GitHub	-
08.02	Noise2Music: Text-conditioned Music Generation with Diffusion Models	arXiv	-	-
04.02	Multi-Source Diffusion Models for Simultaneous Music Generation and Separation	arXiv	GitHub	-
30.01	SingSong: Generating musical accompaniments from singing	arXiv	-	-
30.01	AudioLDM: Text-to-Audio Generation with Latent Diffusion Models	arXiv	GitHub	Hugging Face
30.01	Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion	arXiv	GitHub	-
29.01	Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models	PDF	-	-
28.01	Noise2Music	-	-	-
27.01	RAVE2 [Samples RAVE1]	arXiv	GitHub	-
26.01	MusicLM: Generating Music From Text	arXiv	GitHub (unofficial)	-
18.01	Msanii: High Fidelity Music Synthesis on a Shoestring Budget	arXiv	GitHub	Hugging Face Colab
16.01	ArchiSound: Audio Generation with Diffusion	arXiv	GitHub	-
05.01	VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers	arXiv	GitHub (unofficial) (demo)	-

artificial-intelligence

audio-generation

machine-learning

Similar repos

affige/genmusic_demo_list

a list of demo websites for automatic music generation research

799

EmulationAI/awesome-large-audio-models

Collection of resources on the applications of Large Language Models (LLMs) in Audio AI.

741

ga642381/speech-trident

Awesome speech/audio LLMs, representation learning, and codec models

1,248

AI-Guru/music-generation-research

A straightforward collection of Music Generation research resources.

605

LqNoob/Neural-Codec-and-Speech-Language-Models

Awesome Neural Codec Models, Text-to-Speech Synthesizers & Speech Language Models

248

BinWang28/audio-ai-hub

The hub for audio AI research: papers, open models, benchmarks & datasets across audio LLMs, speech…

958

kadirnar/awesome-tts-architectures

A visual catalog of text-to-speech architectures, with model diagrams, concise descriptions, and…

haoheliu/audioldm2

Text-to-Audio/Music Generation

2,644

Contributors

43 commits

13 commits

1 commits

1 commits