MelTok: A Frequency-Aware Audio Tokenizer for Audio Language Modeling

Audio samples | Paper

MelCap is a neural audio codec that compresses complex audio into a single codebook and reconstructs it with high fidelity.

Installation

To use MelCap only in inference mode, install it using:

pip install cosmos-tokenizer
pip install vocos

If you wish to train the model, install it with additional dependencies:

pip install cosmos-tokenizer
pip install vocos[train]

Usage

First Stage: Mel-spectrogram Reconstruction

find $TRAIN_DATASET_DIR -name *.wav > filelist.train
find $VAL_DATASET_DIR -name *.wav > filelist.val

Train with only reconstruction loss:

python recon/train_96.py

Train with reconstruction loss and vgg loss:

python recon/train_96_vgg.py

If you want to fint-tuen with GM Loss:

python recon/train_96_post_gm.py

Second Stage: From Discrete Mel Tokens to Waveform

• In configs/vocos_96_com.yaml, set the vae_ckpt_path to point to the checkpoint from your stage one training.

• In vocos/feature_extractors.py, ensure the parameters in the feature extractor are consistent with the configuration used in train_96.py during stage one.

python train.py -c configs/vocos_96_com.yaml

Inference

In infer.py, set the config_path and ckpt_path to point to the checkpoint from your stage two training. Additionally, in infer.py, ensure the params are consistent with the configuration used in train_96.py during the first stage.

python infer.py 

Pre-trained models

Released at huggingface

Acknowledgments

We would like to acknowledge the following projects where parts of the codes in MelCAp are derived from: https://github.com/NVIDIA/Cosmos-Tokenizer https://github.com/gemelo-ai/vocos

License

The code in this repository is released under the MIT license as found in the LICENSE file.