Multimodal Emotion Recognition (EmoMV)

Code for multimodal emotion recognition on user-generated music videos (EmoMV), combining audio, video (face), lyrics, and comments.

Scope of this repo: you prepare the data; we provide feature extractors, the unified embedding/CSV formats, and the multimodal trainer.

python multimodal_classifier.py --mode train ...

This repository supports the Interspeech 2026 work Beyond Audio-Visual: Unlocking the Value of Metadata for Emotion Recognition in User-Generated Content.

What this repo does / does not ship

Assume train/val/test media (and optional lyrics/comment text) are already on disk with labels. Then run extractors → manifests → train.

Environments (two are enough)

pip install -e . only registers local packages for imports — it does not install third-party deps. Run it once inside each env.

# --- Env 1: VER (video + train) ---
conda env create -f VER/environment.yml
conda activate VER
cd /path/to/ER && pip install -e .
python VER/extract_EmoMV.py ...
python multimodal_classifier.py --mode train ...

# --- Env 2: MER (audio + text embeddings) ---
conda create -n MER python=3.10 -y && conda activate MER
cd /path/to/ER
pip install -r MER/requirements.txt && pip install -e .
python MER/extract_EmoMV.py ...
python TER/text_embedding_extract.py ...
python TER/extract_comment_embeddings.py ...

Notes:

• VER includes torch / pandas / wandb plus the face stack (dlib, emotiefflib, opencv, …). Using it for training avoids a third env.
• MER is for HuggingFace audio/text (torchaudio, transformers). Face extraction will not work there.

Preparing your data

Bring your own splits. Extractors and the trainer only need consistent keys (clip ids) and labels.

Raw inputs (before extraction)

EmoMV emotion labels (5-way): exciting, fear, tense, sad, relax (indices 0–4).

After extraction — CSV manifests the trainer reads

.pt files use EmbeddingSchema v1 (utils/embedding_schema.py). Full details:

• CSV_MANIFEST_SPECIFICATION.md
• CSV_HEADER_SPEC.py
• EMBEDDING_FORMAT.md

Quick start (classifier)

Features already extracted; activate VER:

conda activate VER
pip install -e .   # once per env

python multimodal_classifier.py --mode train \
  --video-root-dir VER/EmoMV_features_-1_10/Dataset1 \
  --audio-root-dir MER/EmoMV_features/mert/Dataset1 \
  --lyrics-root-dir TER/csvs \
  --comments-root-dir TER/comment_features \
  --fusion-type late_concat \
  --output-base outputs/multimodal_avt \
  --batch-size 16 --epochs 50 --allow-missing

Useful flags:

• --fusion-type — late_concat, cross_attention_mean, video_query_cross_attention, or unimodal audio_only / video_only / lyrics_only / comments_only
• --no-video / --no-audio / … — ablations
• --cross-attn-q/k/v — query / key / value modalities
• --use-wandb — Weights & Biases

Audio-only baseline:

python multimodal_classifier.py --mode train \
  --fusion-type audio_only \
  --audio-root-dir MER/EmoMV_features/mert/Dataset1 \
  --video-root-dir VER/EmoMV_features_-1_10/Dataset1 \
  --allow-missing --output-base outputs/audio_only \
  --batch-size 16 --epochs 50

More examples: docstring at the top of multimodal_classifier.py.

python test_fusion_api.py   # fusion unit tests, no data needed

Repository layout

multimodal_classifier.py   # ★ main train / eval entry
models/                    # encoders + fusion
data/multimodal.py         # join modalities by key
utils/embedding_schema.py  # EmbeddingSchema v1
VER/                       # video extraction (+ environment.yml)
MER/                       # audio extraction (+ requirements.txt)
TER/                       # lyrics / comment embedding extraction
moede/                     # optional MoEDE face experts (--model-name moede)
scripts/analyze_best_results.py
CSV_MANIFEST_SPECIFICATION.md
EMBEDDING_FORMAT.md

Pipeline (assuming data is ready)

1. Audio (MER env) — MER/extract_EmoMV.py
2. Video (VER env) — VER/extract_EmoMV.py
3. Text (MER env) — TER/text_embedding_extract.py / TER/extract_comment_embeddings.py on your prepared text files
4. Train (VER env) — multimodal_classifier.py

Citation

If you use this code, please cite the associated Interspeech 2026 paper:

Beyond Audio-Visual: Unlocking the Value of Metadata for Emotion Recognition in User-Generated Content.