Foundation models for World Music Tagging

Perform automatic tagging on diverse music datasets using Probing, Supervised Fine-Tuning and Few-shot Learning with foundation models.

Reference

Universal Music Representations? Evaluating Foundation Models on World Music Corpora, ISMIR 2025.

• Charilaos Papaioannou, Emmanouil Benetos, and Alexandros Potamianos

Requirements

• Python 3.10 or later
• To set up the environment and install the necessary packages, run:

python -m venv venv
source venv/bin/activate    # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Data preparation

Follow the steps described in the sections Datasets and Preprocessing from the ccml repository for data preparation. This includes downloading the required datasets, processing audio features, and organizing the data in a format compatible with this work.

Models

Foundation models as backbones

• MERT-95M & MERT-330M
• CLAP-Music & CLAP-Music-and-Speech
• Qwen2-Audio

Training

Train a Probe

To train a probe on top of a backbone foundation model, use the train_probe.py script. Below is a detailed explanation of the command-line arguments that can be used to configure the training process:

Arguments

• --dataset:
  The name of the dataset to use for training.
  Options: magnatagatune, fma, makam, lyra, hindustani, carnatic
  Default: magnatagatune

• --data_dir:
  The directory where the audios and split directories are expected to be found. If not specified, the script will look for the magnatagatune dataset in the default data directory.
  Default: os.path.join(DATA_DIR, 'magnatagatune')

• --model:
  The backbone foundation model to be used for feature extraction.
  Options: mert_95 (MERT-95M), mert_330 (MERT-330M), clap_music (CLAP-Music), clap_music_speech (CLAP-Music-and-Speech), qwen (Qwen2-Audio) Default: mert_95

• --probe:
  The probing model to be used for automatic tagging.
  Options: MLP (Multilayer Perceptron) Default: MLP

• --run_idx:
  Define a run index to vary the random seed used for different runs. This is useful for performing multiple training runs with different seeds.
  Options: 1, 2, 3, 4, 5
  Default: 1

• --device:
  Specify the device to use for training. Set this to cpu if no GPU is available, or choose a specific GPU by specifying the CUDA index (e.g., cuda:0, cuda:1).
  Default: cuda:0

Example command

Train an MLP probe on top of MERT-95M model on Lyra dataset:

python train_probe.py --dataset "lyra" --data_dir "/__path_to__/lyra" --model "mert_95" --run_idx "1" --device "cuda:0"

Naming Convention of the Saved Models

The models will be saved in the saved_models/probing/{run_idx}/{dataset} directory using the following naming convention:

• {model}_MLP.pth

Supervised Fine-Tuning (SFT)

To run SFT on a foundation model, use the fine_tune_model.py script. The command-line arguments are the following (the descriptions that are the same as in "Train a Probe" section, are omitted):

Arguments

• --dataset

• --data_dir

• --model

• --num_layers:
  The number of the last layers of the foundation model to be fine-tuned.
  Default: 1

• --load_trained_probe:
  Whether to initialize the weights of the MLP probe, used on top of the foundation model, from a stored state.
  Default: True

• --run_idx

• --device

Example command

Run SFT using the Qwen2-Audio model and the MagnaTagATune dataset, initializing MLP from a previously trained Probe model:

python fine_tune_model.py --dataset "magnatagatune" --data_dir "/__path_to__/magnatagatune" --model "qwen" --num_layers 1 --run_idx "1" --device "cuda:0"

Naming Convention of the Saved Models

The models will be saved in the saved_models/sft/{run_idx}/{dataset} directory using the following naming convention:

• {model}_sft{num_layers}.pth

Evaluation

Probing and SFT models

To evaluate MLP Probe or Supervised Fine-Tuned models, use the evaluate.py script. Below is a detailed explanation of the command-line arguments:

Arguments

• --dataset:
  The name of the dataset to use for training.
  Options: magnatagatune, fma, makam, lyra, hindustani, carnatic
  Default: magnatagatune

• --data_dir:
  The directory where the audios and split directories are expected to be found. If not specified, the script will look for the magnatagatune dataset in the default data directory.
  Default: os.path.join(DATA_DIR, 'magnatagatune')

• --trained_model:
  The trained model to be evaluated. The naming conventions followed in the training processes are expected (e.g., clap_music_speech_MLP, qwen_sft1, etc.). Default: mert_95_MLP

• --run_idx:
  Define a run index to load the respective trained model and vary the random seed.
  Options: 1, 2, 3, 4, 5
  Default: 1

• --device:
  Specify the device to use for evaluation. Set this to cpu if no GPU is available, or choose a specific GPU by specifying the CUDA index (e.g., cuda:0, cuda:1).
  Default: cuda:0

Example command

Evaluate a supervised fine-tuned CLAP-Music&Speech model on the Hindustani dataset:

python evaluate.py --dataset "hindustani" --data_dir "/__path_to__/hindustani" --trained_model "clap_music_speech_sft1" --run_idx "1" --device "cuda:0"

Naming Convention of the Evaluation Results

The evaluation results are saved in the evaluation/probing/{run_idx}/{dataset} directory for the Probing models and evaluation/sft/{run_idx}/{dataset} directory for the SFT ones, with the naming convention: {model}.txt.

Each evaluation file contains:

• ROC-AUC, PR-AUC as well as Macro-F1 and Micro-F1 scores
• A per-tag classification report for the model

ML-FSL evaluation

For the Multi-Label Few-Shot Learning evaluation, use the few_shot_evaluate.py script. The command-line arguments are the following (the descriptions that are the same as in Probing/SFT evaluation, are omitted):

Arguments

• --dataset

• --data_dir

• --model:
  The model to be used for the ML-FSL evaluation. It can be a publicly available pre-trained model (use: mert_95, mert_330, clap_music, clap_music_speech, qwen), a trained MLP Probe on top of a model (e.g., clap_music_MLP), or a SFT model (e.g., mert_95_sft2).
  Default: mert_95

• --type:
  Specifies whether to evaluate on "standard" tags or "extended" set of tags.
  Options:

  • standard: Use the well-represented classes.
  • extended: Use the extended set of tags. Default: extended

• --K:
  The number of support items (examples) per label (K-shot).
  Default: 3

• --dist:
  The distance metric to use for prototype-based classification.
  Options:

  • l2: Euclidean distance
  • cos: Cosine distance
    Default: cos

• --run_idx

• --device

Example commands

Evaluate an MLP-Probe trained on top of Qwen2-Audio on the Turkish-makam dataset. Use the extended set of tags and 3 examples per tag:

python few_shot_evaluate.py --dataset "makam" --data_dir "/__path_to__/makam" --model "qwen_MLP" --type "extended" --K 3 --run_idx "1" --device "cuda:0"

Naming Convention of the Evaluation Results

The evaluation results are saved in the evaluation/mlfsl/{run_idx}/{dataset} directory using the following naming convention:

• {N}_way_{type}_{model}.txt: where N is the number of tags (classes) in the extended set.

Each evaluation file contains:

• Macro-F1 and Micro-F1 scores
• A per-tag classification report for the model
• Process insights:
  • The number of LC-Prototypes formed
  • The number of unique LC-Prototypes in terms of their representation (see the reference paper for more information)
  • The optimization ratio
  • The number of unique items in the support set and the query set
  • The mean ground truth and predicted labels per item
  • Total execution time

License

This project is licensed under the MIT License. See the LICENSE file for details.