Vietnamese Automated Lyrics Transcription (ALT)

This project aims to perform automatic lyrics transcription on Vietnamese songs, the pre-trained model used for this task is Whisper model from Robust Speech Recognition via Large-Scale Weak Supervision.

Our fine-tuned Whisper model for Vietnamese lyrics transcription is now available on Hugging Face 🤗

• xyzDivergence/whisper-small-vietnamese-lyrics-transcription
• xyzDivergence/whisper-medium-vietnamese-lyrics-transcription
• xyzDivergence/whisper-large-v2-vietnamese-lyrics-transcription

Setup

To install the dependencies, run the following command:

pip install -r requirements.txt

Resource

• GPU x2 RTX 3090 24GB
• RAM 128 GB

Training Data

The training dataset consists of 7,000 Vietnamese songs, in total of roughly 550 hours of audio, across various Vietnamese music genres, dialects and accents. Due to IP concerns, the data is not publicly available. However, songs and lyrics scraping guide and preprocessing code can be found in data_processing, along with the songs and links used for training and testing, which are in folder data_processing/in train and val csv files. Each song includes lyrics along with corresponding line-level timestamps, enabling precise mapping of audio segments to their respective lyrics based on the provided timestamp information.

Preparation for Model training

To train the Whisper model, change your directory to the train/ folder:

cd train/

Data folder structure

├── train                                               # Training pipeline
│   ├── generate_audio_for_whisper.py                   # Pre-generate chunks of 30-second audio
│   ├── whisper_train.py                                # Whisper training script
│   ├── ds_config.json                                  # DeepSpeed configuration file
│   ├── training_audio                                  # Training folder
│   │   ├── song1.mp3                                   # Song in .mp3 format
│   │   └── song1.origin.lrc                            # The lyric file of the song
│   └── validation_audio                                # Validation folder
│       ├── song2.mp3                                   # Song in .mp3 format
│       └── song2.origin.lrc                            # The lyric file of the song

Preprocess audio into format for Whisper fine-tuning

Generate both training and validation preprocessed audio lists required for Whisper training.

The following command preprocesses the raw audio and lyrics by chunking the original audio into 30-second segments and aligning the timestamp of the lyrics with its audio speech:

python generate_audio_for_whisper.py --train_dir 'training_audio' \
                                     --test_dir 'validation_audio' \
                                     --audio_arr_dir_name "audio_8k_pp"

This will generate two metadata files: "train_preprocessed_audio_8k_pp_audio_list.npy" and "test_preprocessed_audio_8k_pp_audio_list.npy", which are used for Whisper fine-tuning. The original 30s audio chunks are also generated and stored in the audio_arr_dir_name folder.

Whisper Fine-Tuning

The Whisper model can be fine-tuned using the following parameters:

Arguments

• experiment_name: The name of the experiment to keep track. The checkpoint will be saved under this folder.
• model_name: The name of the model, it should be one of the whisper variant form HuggingFace e.g. "openai/whisper-small"
• train_audio_list: The preprocessed training audio list from the previous step.
• test_audio_list: The preprocessed testing audio list from the previous step.
• gpu_no: The GPU device to use for training. Use "0" for the first GPU, "1" for the second GPU, etc.

Example

Train with single GPU

python train/whisper_train.py --experiment_name "whisper-large-v2-7k-1k-pp" \
                              --model_name "openai/whisper-large-v2" \
                              --train_audio_list "train_preprocessed_audio_8k_pp_audio_list.npy" \
                              --test_audio_list "test_preprocessed_audio_8k_pp_audio_list.npy" \

Train with multiple GPUs using DeepSpeed (Recommended)

deepspeed --num_gpus=2 whisper_train.py --experiment_name "whisper-large-v2-7k-1k-pp" \
                                        --model_name "openai/whisper-large-v2" \
                                        --train_audio_list "train_preprocessed_audio_8k_pp_audio_list.npy" \
                                        --test_audio_list "test_preprocessed_audio_8k_pp_audio_list.npy"

Whisper Evaluation

Whisper Inference

To perform Vietnamese automatic lyrics transcription, you can use the Transformers pipeline below.

Using Transformers Pipeline

To generate the transcription for a song, we can use the Transformers pipeline. Chunking is enabled by setting chunk_length_s=30 when instantiating the pipeline. With chunking enabled, the pipeline can be run with batched inference. It can also be extended to predict sequence level timestamps by passing return_timestamps=True. In the following example we are passing return_timestamps="word" that provides precise timestamps for when each individual word in the audio starts and ends.

>>> from transformers import pipeline
>>> asr_pipeline = pipeline(
>>>    "automatic-speech-recognition",
>>>    model="xyzDivergence/whisper-large-v2-vietnamese-lyrics-transcription", chunk_length_s=30, device='cuda',
>>>    tokenizer="xyzDivergence/whisper-large-v2-vietnamese-lyrics-transcription"
>>> )
>>> transcription = asr_pipeline("sample_audio.mp3", return_timestamps="word")

Streamlit Demo

For the streamlit demo app, go into streamlit_demo folder and follow its README.md.

Contribution

This project was made through equal contributions from:

• Kevin Soh
• Bernard Cheng Zheng Zhuan
• Nguyen Quoc Anh