Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis

Audio samples | Paper [abs] [pdf]

Vocos is a fast neural vocoder designed to synthesize audio waveforms from acoustic features. Trained using a Generative Adversarial Network (GAN) objective, Vocos can generate waveforms in a single forward pass. Unlike other typical GAN-based vocoders, Vocos does not model audio samples in the time domain. Instead, it generates spectral coefficients, facilitating rapid audio reconstruction through inverse Fourier transform.

Project maintenance

Vocos remains available for research and production use, and the maintainers are continuing to review focused, backward-compatible improvements. Current priorities include documentation, project infrastructure, issue and pull request triage, and making the contribution path clearer. If you use Vocos downstream, feedback and small, well-scoped contributions are welcome.

Installation

To use Vocos only in inference mode, install it using:

python -m pip install vocos

The training entry point, configs, and metrics live in the source repository. To train a model, clone the repository and install it with the training dependencies:

git clone https://github.com/gemelo-ai/vocos.git
cd vocos
python -m pip install "setuptools<80"
python -m pip install -e ".[train]"

The checked-in training stack pins PyTorch Lightning 1.8.6, which still uses pkg_resources; keeping Setuptools below version 80 in the training environment preserves that compatibility.

Usage

Reconstruct audio from mel-spectrogram

import torch

from vocos import Vocos

vocos = Vocos.from_pretrained("charactr/vocos-mel-24khz")

mel = torch.randn(1, 100, 256)  # B, C, T
audio = vocos.decode(mel)

Copy-synthesis from a file:

import torchaudio

audio_path = "path/to/audio.wav"
y, sr = torchaudio.load(audio_path)
if y.size(0) > 1:  # mix to mono
    y = y.mean(dim=0, keepdim=True)
y = torchaudio.functional.resample(y, orig_freq=sr, new_freq=24000)
y_hat = vocos(y)

Reconstruct audio from EnCodec tokens

Additionally, you need to provide a bandwidth_id which corresponds to the embedding for bandwidth from the list: [1.5, 3.0, 6.0, 12.0].

vocos = Vocos.from_pretrained("charactr/vocos-encodec-24khz")

audio_tokens = torch.randint(low=0, high=1024, size=(8, 200))  # 8 codebooks, 200 frames
features = vocos.codes_to_features(audio_tokens)
bandwidth_id = torch.tensor([2])  # 6 kbps

audio = vocos.decode(features, bandwidth_id=bandwidth_id)

Copy-synthesis from a file: It extracts and quantizes features with EnCodec, then reconstructs them with Vocos in a single forward pass.

audio_path = "path/to/audio.wav"
y, sr = torchaudio.load(audio_path)
if y.size(0) > 1:  # mix to mono
    y = y.mean(dim=0, keepdim=True)
y = torchaudio.functional.resample(y, orig_freq=sr, new_freq=24000)

y_hat = vocos(y, bandwidth_id=bandwidth_id)

Integrate with 🐶 Bark text-to-audio model

See example notebook.

Pre-trained models

Training

Prepare a filelist of audio files for the training and validation set:

find "$TRAIN_DATASET_DIR" \( -type f -o -type l \) -name '*.wav' -print > filelist.train
find "$VAL_DATASET_DIR" \( -type f -o -type l \) -name '*.wav' -print > filelist.val

This includes symlinked .wav files without following symlinked directories.

Fill a config file, e.g. vocos.yaml, with your filelist paths and start training with:

python train.py -c configs/vocos.yaml

The checked-in training code and configs target PyTorch Lightning 1.8.6. Refer to the PyTorch Lightning 1.8.6 documentation for details about customizing that training pipeline.

Contributing

Bug reports, documentation improvements, and focused fixes are welcome. Please read CONTRIBUTING.md before opening a pull request, and include enough detail for maintainers to understand and reproduce the change.

Citation

If this code contributes to your research, please cite our work:

@article{siuzdak2023vocos,
  title={Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis},
  author={Siuzdak, Hubert},
  journal={arXiv preprint arXiv:2306.00814},
  year={2023}
}

License

The code in this repository is released under the MIT license as found in the LICENSE file.