Monash Research: Tone Bias

Building on the paper "Bias Beneath the Tone: Empirical Characterisation of Tone Bias in LLM-Driven UX Systems" (UISE 2026, arXiv:2512.19950)

Speech emotion recognition (SER) evaluation on eight cleaned corpora, running inference on emotion2vec and Empathic-Insight-Voice (EIV), and then fine-tuning the better model for our speech corpus. Our-speech human labels come from HEET (heet_dataset_clean.csv).

Setup (uv)

Requires uv and Python 3.12+.

uv sync                    # create .venv and install dependencies
cp .env.example .env       # optional: Gemini API key to run the Gemini pipeline.
uv run python src/inference/emotion2vec.py --dry-run # test the pipeline for the first 10 rows.

Run any script with uv run python <path>. Evaluators in src/evaluators/ are run the same way (see below).

For GPU inference (EIV / emotion2vec), install a CUDA build of PyTorch separately if needed, e.g. uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124.

Dependencies

Declared in pyproject.toml. Full stack used by this project:

funasr also pulls modelscope for model hosting. Cleaners and manifest loaders need only the pandas/numpy stack.

Corpora

Eight cleaned sets under corpora_cleaned/ (not in git):

Data files

Project layout

monash-research-tone-bias/
├── README.md
├── heet_dataset.csv
├── heet_dataset_clean.csv
├── empathic_insight_voice.py          # EIV valence-head only (legacy local eval)
├── our_speech_emotion_variants_colab.py
├── manifests/
├── src/
│   ├── gemini_response_pipeline.py    # Generate our-speech audio via Gemini Live API
│   ├── cleaners/                      # Build corpora_cleaned/ subsets
│   ├── manifest_loaders/              # HF metadata → manifests/
│   ├── inference/                     # GPU inference (writes predictions + metrics)
│   └── evaluators/                    # Summary tables, diagrams, re-aggregation
└── corpora_cleaned/                   # Audio (local only)

Results are written to ../results/ (sibling of this repo).

Inference

emotion2vec

src/inference/emotion2vec.py — pretrained emotion2vec_plus_large, 3-way valence (pos/neu/neg).

uv run python src/inference/emotion2vec.py --dry-run
uv run python src/inference/emotion2vec.py
uv run python src/inference/emotion2vec.py --from-cache --workers 8

Output: ../results/emotion2vec/<corpus>/predictions.csv, metrics.json

Empathic-Insight-Voice (36 emotion heads)

src/inference/empathic_insight_voice_all36_kaggle.py — Whisper + Valence MLP + 36 EMONET emotion heads. Three valence methods:

uv run python src/inference/empathic_insight_voice_all36_kaggle.py --dry-run
uv run python src/inference/empathic_insight_voice_all36_kaggle.py

Output: ../results/empathic_insight_voice_all36/<corpus>/predictions.csv, metrics.json

Evaluators

Run after inference. All live in src/evaluators/.

uv run python src/evaluators/emotion2vec_summary_table.py
uv run python src/evaluators/emotion2vec_all_emotions.py
uv run python src/evaluators/eiv_summary_table.py
uv run python src/evaluators/eiv_all_emotions.py

Our-speech summary rows:

• *_prompted — vs filename / synthesis target valence
• *_human_GT — vs HEET human labels on the shared n=229 set (human-labelled ∩ emotion2vec-mappable)

Corpus builders

Manifest loaders (src/manifest_loaders/)

See src/manifest_loaders/README.md for usage.

Cleaners (src/cleaners/)

Other scripts

Results layout

../results/
├── emotion2vec/
│   ├── summary_metrics_table.csv
│   ├── <corpus>/predictions.csv, metrics.json
│   ├── all_emotions/
│   └── result_diagrams/
└── empathic_insight_voice_all36/
    ├── summary_metrics_table_{valence_head,top1,mass}.csv
    ├── summary_comparison.csv
    ├── metrics_all.json
    ├── <corpus>/predictions.csv, metrics.json
    ├── all_emotions/
    └── result_diagrams/

References

• emotion2vec_plus_large
• Empathic-Insight-Voice-Small
• EmoWhisper-AnS-Small