Overview

Most speech-emotion demos are a convolutional network trained on a few hours of acted studio recordings. They report 95% accuracy on their own test split and then fall apart the moment a real person speaks into a real microphone.

This project is built the other way around. A speech-emotion foundation model does the classification, a second model trained on spontaneous conversation votes alongside it, a third supplies learned arousal and valence, and a neural voice-activity detector segments your speech at natural pauses so the classifier sees complete utterances — the condition it was trained on.

Every layer in that stack earned its place by measurement. The repository ships its own two-corpus evaluation harness (eval/), and several plausible-sounding ideas were tested and rejected because the numbers said so. See How it was made.

Audio never leaves your machine. The browser streams your microphone to a local FastAPI server, which analyzes it on the fly and records nothing.

The eight emotions

The classifier maps the acoustic signature of your speech onto eight emotional states, each with a confidence score updated continuously as you talk.

Capabilities

Live microphone analysis. Audio streams from the browser over a WebSocket. The stream is segmented into utterances at natural speech pauses — the pattern production streaming systems use. Provisional interim predictions update roughly twice per second while you speak; when a pause closes the utterance, a final prediction runs on the complete utterance and carries the most weight. Two utterances never blur together in one analysis window.

Neural voice activity detection. Silero VAD segments speech at 300 ms pauses. Silence, fan hum, and keyboard clicks produce no predictions at all — and quiet, flat deliveries are still detected, which the original energy-heuristic VAD missed roughly 30% of the time on sad speech.

Two-model ensemble. Finalized utterances are scored by emotion2vec+ and the Odyssey 2024 WavLM baseline together. The two are trained on different distributions — largely performed speech versus spontaneous podcast speech — so their errors decorrelate where it matters most.

Learned arousal and valence. A wav2vec2 model trained on natural podcast speech supplies continuous arousal and valence, replacing hand-tuned pitch-and-energy heuristics. Emotions whose expected activation contradicts the measured delivery are downweighted, so a flat, low-energy "I'm happy" is not scored as happy.

Speaker calibration. Each session learns the speaker's resting register from their own neutral utterances and recenters the fusion against it — bounded and label-guided, so a genuinely angry session can never normalize itself away.

Reliability gating. Predictions from under three seconds of voiced speech — the model's documented hallucination regime — get reduced smoothing weight and are never reported as high-certainty. The classes emotion2vec recognizes poorly only headline when they clearly lead.

Honest confidence. Probabilities are calibrated and smoothed over time. Unstable results are flagged as stabilizing instead of flashing 99.9%.

Orb assistant. An animated glass orb with eyes that idles, blinks, listens, thinks, and reacts to your voice volume, glowing in the color of the detected emotion.

Architecture

flowchart TD
    A["Browser — Web Audio API<br/>raw Float32 PCM over WebSocket"] --> B["FastAPI — utterance buffer<br/>segmented at speech pauses, 12 s cap"]
    B --> C["Silero VAD<br/>neural segmentation at 300 ms pauses"]
    C --> D["emotion2vec+ base<br/>42,500 h foundation model"]
    C --> E["Odyssey 2024 WavLM<br/>MSP-Podcast, natural speech"]
    C --> F["audeering wav2vec2<br/>learned arousal / valence"]
    C --> G["faster-whisper<br/>transcript, display only"]
    D --> H["Ensemble blend<br/>finalized utterances"]
    E --> H
    H --> I["Calibration and affect fusion<br/>temperature, label smoothing,<br/>arousal / valence reweighting"]
    F --> I
    I --> J["Speaker calibration<br/>per-session resting baseline"]
    J --> K["Temporal smoothing<br/>finals dominate interims"]
    K --> L["JSON to UI<br/>emotion, confidence, probabilities, prosody"]
    G --> L

    classDef primary fill:#c8a97e,stroke:#ab8a5c,color:#352e24
    classDef model fill:#9fafca,stroke:#7d90ad,color:#352e24
    classDef fusion fill:#e8dcc8,stroke:#837562,color:#352e24
    class A,B,L primary
    class D,E,F,G model
    class C,H,I,J,K fusion

The fusion layer is the interesting part. Each emotion carries an expected arousal level — how activated the delivery should sound — and an expected valence. Both are measured by a model trained on real conversational speech, then emotions whose expectations contradict the measured delivery are downweighted before the result is reported. The classifier's spectral opinion is checked against the actual energy and positivity of the delivery, and "calm" is synthesized from the neutral mass at genuinely low arousal.

Tech stack

Every model has an automatic fallback. Without network access on first run, the app still starts and serves predictions from the bundled CNN.

How it was made

The project began as the standard approach: a convolutional network trained on RAVDESS MFCCs, scoring about 95% on a held-out RAVDESS split. On live microphone speech it was unconvincing — the classic symptom of a model that memorized 24 actors reading fixed sentences in a studio.

Fixing that meant treating it as a measurement problem rather than a model problem.

First, an evaluation harness. Nothing could improve without a way to tell improvement from noise. eval/ downloads two corpora and scores the exact production code path on both: CREMA-D for acted studio speech and MELD for spontaneous television conversation. They answer different questions — CREMA-D catches regressions on clean audio, MELD approximates the live microphone reality — so they are always reported separately.

Then, one change at a time, each one measured. The results, including the ideas that did not survive:

Two of nine candidate improvements were rejected on evidence, and the ensemble blend weight turned out to be a domain dial rather than a single best value — weighting the natural-speech model more helps conversation and costs performed speech. It is set for live microphone use and documented in emotion_engine.py next to the measurement that chose it.

Every constant that was tuned has its measurement recorded beside it in the source, so the reasoning survives longer than the memory of the session that produced it.

Accuracy, honestly

Measured by eval/evaluate.py on the fused pipeline:

That second number is not a defect. State-of-the-art systems reach roughly 0.31–0.40 macro-F1 on natural eight-class speech (Odyssey 2024 challenge), and the same foundation models that score in the seventies on acted corpora drop to the twenties on spontaneous speech. Emotion recognition from tone alone on real conversation is a genuinely unsolved problem, and any project advertising 90%-plus is reporting acted-corpus numbers.

What the surrounding pipeline buys is the removal of the worst failure modes: predictions on silence, labels that contradict the delivery, wild confidence spikes, dropped quiet speech, and confidently-wrong rare classes. The certainty flag tells you when a result is stable enough to trust.

Reproduce it:

venv/bin/python eval/download_eval_set.py    # fetches both corpora
venv/bin/python eval/evaluate.py             # scores the production path
venv/bin/python eval/evaluate_sessions.py    # per-speaker adaptation A/B

Results are written to eval/results/ as timestamped JSON so any two pipeline versions can be compared directly.

Getting started

Requirements: Python 3.12 — TensorFlow does not yet support 3.13 or 3.14.

./start.sh        # macOS / Linux
start.bat         # Windows, double-click also works

The launcher creates a virtual environment on first run, installs dependencies, starts the server, waits for the models to load, and opens http://localhost:8000.

python3.12 -m venv venv
./venv/bin/pip install -r requirements.txt
./venv/bin/uvicorn main:app --port 8000

On first start, model checkpoints are downloaded from Hugging Face automatically — roughly 3.5 GB in total. Without network access the app still runs on the bundled CNN fallback.

API

File upload. Returns 422 if the file contains no detectable speech.

curl -X POST -F "file=@audio.wav" http://localhost:8000/predict

{
  "Emotion": "happy",
  "Confidence": 62.4,
  "Probabilities": { "angry": 4.2, "calm": 3.1, "...": 0 },
  "Prosody": { "arousal": 0.61, "valence": 0.72, "pitch_hz": 254.7 },
  "VoicedSeconds": 3.4,
  "Transcript": "that went really well actually",
  "Engine": "emotion2vec+ base + Odyssey WavLM",
  "Status": "Success"
}

Live stream. Send one JSON text frame {"sampleRate": 48000, "transcribe": true}, then binary Float32 PCM chunks. transcribe is optional (default true) and can be flipped mid-session with another config frame — transcription is English-only, so speakers of other languages can turn it off, which also makes final results arrive faster.

{
  "status": "ok",
  "segment": "final",
  "emotion": "happy",
  "confidence": 62.4,
  "certainty": "high",
  "probabilities": { "angry": 4.2, "...": 0 },
  "prosody": {
    "pitch_hz": 254.7, "pitch_var": 0.64, "energy_mod": 0.39,
    "speech_rate": 3.4, "arousal": 0.61, "valence": 0.72
  },
  "voiced_seconds": 3.4,
  "transcript": "that went really well actually"
}

segment is "interim" — provisional, computed on the utterance so far — or "final", computed once on the complete utterance after a pause closes it. Finals dominate the smoothed result. Frames with no detected speech return {"status": "silence"}.

Project structure

.
├── main.py                  # FastAPI app: routes + live WebSocket pipeline
├── analysis.py              # Silero VAD, prosody, calibration, affect fusion
├── emotion_engine.py        # emotion2vec+ and WavLM engines, ensemble, headline policy
├── dimensional.py           # learned arousal / valence (audeering wav2vec2)
├── wavlm_ser.py             # native port of the Odyssey 2024 WavLM baseline
├── text_emotion.py          # ASR (faster-whisper) + text-emotion branch
├── speaker_session.py       # per-session speaker calibration
├── extract_feature.py       # MFCC extraction for the fallback CNN
├── eval/
│   ├── download_eval_set.py # fetches CREMA-D, MELD, per-actor sessions
│   ├── evaluate.py          # scores the production path, per corpus
│   ├── evaluate_sessions.py # per-speaker adaptation A/B
│   └── results/             # timestamped JSON, diffable across versions
├── static/
│   ├── index.html           # landing page
│   └── app.html             # live experience: orb, waveform, analysis
├── orb/                     # original React orb app (reference, not served)
├── voice_emotion_model.h5   # CNN trained on RAVDESS (fallback)
├── label_encoder.pkl        # emotion label encoder (fallback)
├── requirements.txt
├── start.sh                 # one-command launcher (macOS / Linux)
└── start.bat                # one-command launcher (Windows)

Roadmap

The off-the-shelf improvements are exhausted; the pipeline now scores within the published state-of-the-art band on natural speech. One substantial avenue remains:

Fine-tune a classifier head on frozen features. Extract emotion2vec embeddings once over merged corpora — RAVDESS, CREMA-D, TESS, SAVEE, plus a natural-speech set — with reverberation and pitch-shift augmentation, then train a small head matched to this project's exact label space. Reverberation and pitch shifting are the two highest-value augmentations reported for speech emotion generalization, at roughly +3.4% and +3.2% respectively, and they simulate precisely what differs between studio corpora and a laptop microphone in a room.

Worth doing when a specific, repeatable failure appears — the evaluation harness will confirm whether it fixed it.

Credits

• emotion2vec+ — speech emotion foundation model, ACL 2024
• Odyssey 2024 SER Challenge baseline — WavLM trained on MSP-Podcast, MIT
• audeering w2v2 MSP-Dim — dimensional arousal / valence, CC-BY-NC-SA, research use
• Silero VAD — neural voice activity detection, MIT
• RAVDESS — training data for the fallback CNN
• CREMA-D and MELD — evaluation corpora
• Orb assistant — concept from the Google AI Studio orb starter app (orb/), ported to dependency-free CSS/JS and restyled