EmotionRankCLAP

Bridging natural language speaking styles and ordinal speech emotion via Rank-N-Contrast Interspeech 2025 · arXiv:2505.23732

Emotion-CLAP models usually align speech with a text prompt and stop there, which treats "slightly annoyed" and "furious" as merely different — two labels that happen not to match. Emotions are ordered, and that ordering is the signal these models throw away.

EmotionRankCLAP learns a joint embedding space in which similarity decays monotonically with distance in the valence-arousal plane, by training with a Rank-N-Contrast objective over continuous emotion annotations instead of category matches. The result is a space you can walk along: turn up the arousal in a caption, and the speech it retrieves gets more aroused too.

from emotionrankclap import EmotionRankCLAP

model = EmotionRankCLAP.from_pretrained()
text  = model.embed_text(["a calm, gentle voice", "furious shouting"])
audio = model.embed_audio(["utterance.wav"])
print(model.similarity(text, audio))

Results

Ordinality on MSP-Podcast test1 (~35k utterances): captions sweeping one attribute are embedded, each retrieves its nearest utterance, and Kendall's tau is computed between the requested values and the retrieved utterances' annotations. Mean over 100 lists per attribute.

Reproduce with one command once you have the corpus — see Reproducing the paper. The original run's per-list taus are in reference_results/, so the comparison can be a diff rather than a squint: the two distributions agree at every quartile, with 96/100 (arousal) and 99/100 (valence) order statistics within 0.05.

Install

git clone https://github.com/Kodhandarama/EmotionRankCLAP
cd EmotionRankCLAP
pip install -e .              # inference
pip install -e ".[train]"     # + training (PyTorch Lightning, pandas)

Python ≥ 3.9. A GPU is optional for inference and expected for training.

Getting the weights

Two files are needed, and only one of them is small.

1. EmotionRankCLAP itself — checkpoints/emotionrankclap_justAV_epoch06.pt (3.7 MB), shipped with this repository. It holds only what was learned: the two projectors and the loss temperature, 918,529 parameters. The text encoder is loaded from the public Hugging Face model at runtime, because training never changed it — verified tensor by tensor, see docs/MODEL.md.

sha256  5cd81067be9f07b9d42dce8b94ae84cbf83b48194c0b12f0f67cea526f901007

2. The SER audio backbone — WavLM-large, 1.2 GB, downloaded on first use from 3loi/SER-Odyssey-Baseline-WavLM-Multi-Attributes. To pin it locally:

huggingface-cli download 3loi/SER-Odyssey-Baseline-WavLM-Multi-Attributes \
    --local-dir checkpoints/ser_backbone
python -c "from emotionrankclap import EmotionRankCLAP; \
           EmotionRankCLAP.from_pretrained(ser_backbone='checkpoints/ser_backbone')"

Text-only work never loads it — the audio branch is built on first use.

Quickstart

python examples/quickstart.py            # text side only: pulls the 330 MB text encoder, not the SER backbone
python examples/quickstart.py speech/*.wav         # rank real utterances against captions

Embed a corpus for downstream use:

python scripts/embed.py --audio "corpus/*.wav" --out audio.npz --device cuda:0
python scripts/embed.py --text captions.txt --out text.npz

Embeddings are L2-normalised, so text @ audio.T is cosine similarity. They are also non-negative — both projectors end in a ReLU, so scores live in [0, 1] and a completely unrelated pair still scores above zero. Rank with them; don't threshold them as if they were probabilities.

Reproducing the paper

Needs MSP-Podcast (see docs/DATA.md); the evaluation prompts are in this repository.

python scripts/evaluate_ordinality.py \
    --msp-csv metadata/LLD_HLD_v2_Test1.csv \
    --audio-dir /path/to/MSP-PODCAST-Publish-1.12/Audios \
    --save-pool pool_test1.npz --device cuda:0

# later runs reuse the embedded pool
python scripts/evaluate_ordinality.py --pool pool_test1.npz

metric      mean     std           published    delta
AOC        0.560   0.139     0.552 +/- 0.12    +0.008
VOC        0.616   0.132     0.616 +/- 0.13    -0.000

Pool size changes the metric — fewer candidates means fewer near-misses — so quote it alongside any number you report.

Training

cp configs/train.yaml configs/mine.yaml   # edit the three data paths
python scripts/train.py --config configs/mine.yaml
python scripts/export_checkpoint.py \
    --checkpoint train_logs/emotionrankclap/version_0/checkpoints/best-epoch=06-*.ckpt \
    --out checkpoints/my_model.pt

configs/train.yaml carries the hyper-parameters of the released run, read back out of its checkpoint rather than from memory: AdamW, lr 1e-3, batch 64, gradient clipping at 10.0, best epoch by validation loss (epoch 06, step 9,842). Because both encoders are frozen, a run costs one forward pass of WavLM per utterance and fits on a single GPU.

What is easy to get wrong

Reproducing this model is mostly a matter of six details that do not announce themselves — the joint space being non-negative, text and audio being normalised differently, the audio backbone living outside the checkpoint, the audio length differing between training and evaluation. Each is documented, with the evidence, in docs/MODEL.md. Worth reading before changing anything in emotionrankclap/inference.py.

Repository layout

emotionrankclap/
├── inference.py        EmotionRankCLAP — load, embed, rank (no Lightning needed)
├── model.py            EmotionRankCLAPModule — the trainable LightningModule
├── losses.py           Rank-N-Contrast InfoNCE (published loop + vectorised)
├── data.py             MSP-Podcast + caption dataset and collation
├── metrics.py          ordinality evaluation (AOC / VOC)
├── utils.py            audio loading, config, checkpoint discovery
└── backbones/          frozen text (DistilRoBERTa) and audio (WavLM SER) encoders
scripts/
├── train.py            training entry point
├── evaluate_ordinality.py   reproduce AOC / VOC
├── embed.py            embed a corpus or a caption file
├── export_checkpoint.py     research checkpoint -> compact release checkpoint
└── generate_captions.py     regenerate captions / evaluation sweeps
configs/train.yaml      hyper-parameters of the released run
data/ordinality_prompts/  the evaluation sweeps (100 lists x 14 captions x 2 attributes)
reference_results/      per-list taus from the original evaluation
docs/                   MODEL.md (architecture + pitfalls), DATA.md (corpus + captions)
tests/                  loss equivalence, model contract, prompt-list integrity

Citation

@inproceedings{chandra2025emotionrankclap,
  title     = {EmotionRankCLAP: Bridging Natural Language Speaking Styles and
               Ordinal Speech Emotion via Rank-N-Contrast},
  author    = {Suresh Chandra, Shreeram and Goncalves, Lucas and Lu, Junchen and
               Busso, Carlos and Sisman, Berrak},
  booktitle = {Interspeech},
  year      = {2025}
}

Acknowledgements

• Emo-CLIM / CLIMUR (Stewart et al., ICASSP 2024) — the training scaffolding this codebase grew out of. MIT licensed; the copyright notice is retained.
• Odyssey 2024 SER baseline (Goncalves et al.) — the frozen audio encoder, 3loi/SER-Odyssey-Baseline-WavLM-Multi-Attributes, MIT.
• j-hartmann/emotion-english-distilroberta-base — the frozen text encoder.
• Rank-N-Contrast (Zha et al., NeurIPS 2023) — the objective this work lifts into the cross-modal setting.
• MSP-Podcast (Lotfian and Busso) — the corpus, licensed separately by UT Dallas and not redistributed here.

Released under the MIT License; see LICENSE. Model weights inherit the terms of the upstream encoders they build on.