V(P)AD scoring model

Description

This repository contains a weights of a scoring model for V(P)AD (Valence, Arousal, and Dominance) emotional state estimation.

The model uses fine-tuned Whisper based encoder to extract audio features and a simple CNN and MLP head to predict VAD scores means and standard deviations.

This repository consists only the CNN-MLP head weights, the encoder weights available at laion/BUD-E-Whisper.

The source code for the model is available at this repository.

Inteneded use

The model is intended to be used for estimating VAD scores from audio data, which can be useful in various applications such as emotion recognition, affective computing, and human-computer interaction.

Data

The model was trained with freezed encoder weights on the Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset. The dataset does not have VAD scores provided directly, so to obtain them the following steps were taken:

1. Extract emotion categorical labels from the dataset file names.
2. Use Gemma3 27B model to select closest label by meaning from 157 labels described in the original paper.
3. Use selected labels to obtain VAD scores from the table in the paper.

The rare labels were removed from the training mix and the equal number of samples for each label was used.

Limitations

1. Model was trained and tested on single-speaker audio data, so it may not perform well on multi-speaker audio.
2. Since VAD scores are not directly provided in the dataset, the model's performance may be limited by the quality of the label selection process and the accuracy of the VAD scores obtained from the original paper.
3. Because Whisper usese both semantic and acoustic features, the models performance may be affected by the speaker language if the language is not directly supported by Whisper.

Credit

1. LaionAI for the Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset as well as the BUD-E Whisper model.