VoiceEmoji

A project for analyzing the emotional state based on audio recordings and generating psycho-emotional maps.
Visit our website
Supported formats: MP3, OGG, WAV.

Features

• Audio analysis using the Whisper-large-v3 model for emotion recognition.
• Visualization of results:
  • Emotional map (t-SNE) — shows the distribution of emotions in space.
  • Mental Map (PCA) — displays psychological characteristics.
  • Time chart — dynamics of emotions by audio segments.
  • UMAP projection — is an alternative visualization while maintaining the global structure.
  • Heat map — intensity of emotions by category.
• Intuitive graphical interface (GUI) based on Tkinter.

How it works

1. Audio is divided into segments of 10 seconds each.
2. For each segment, an emotion is determined using a pre-trained model.
3. Based on the received emotions, we build:

• 2D visualization using t-SNE and UMAP to group similar states.
• PCA analysis for interpreting psychological patterns.
• Timeline with color-coded emotions.
• Heat map to assess the frequency of emotions.

Some help

1. Emotion Map (t-SNE)

• Visualizes emotional patterns in 2D space:
  • Close points: similar emotional states
  • Colors: different emotion categories
  • Use legend for identification

2. Mental Map (PCA)

• Shows principal components of emotional data:
  • X-axis: Primary variation pattern
  • Y-axis: Secondary variation pattern
  • Labels show dominant segment emotion

3. Temporal Chart

• Displays emotion dynamics over time:
  • Band width: segment duration (10 sec)
  • Color: current emotion
  • Vertical axis: sequence of segments

4. UMAP Projection

• Alternative multidimensional visualization:
  • Preserves both global and local structure
  • Clusters: emotionally similar groups

5. Heatmap

• Shows emotion intensity:
  • Bar length: relative frequency
  • Color gradient: weak (light) to strong (dark)
  • Percentages show emotion share

Tips

1. Compare different projections for full picture.
2. Note clusters and outliers.
3. Use temporal chart to track changes.

Screenshots

Models used

speech-emotion-recognition-with-openai-whisper-large-v3 — classification of emotions.
bert-base-uncased — vector representation of the text.
whisper-medium — transcribing audio.
— advice generation.

Possible errors

• Error when processing audio: Make sure that the file is not corrupted and corresponds to the supported formats.
• Dependency issues on Windows: Install Microsoft Visual C++ Build Tools.

Requirements

• Python 3.8.10
• Dependencies:

  pip install -r requirements.txt

• Install ffmpeg (for pydub):
  • sudo apt-get install ffmpeg (Linux)
  • or via the official website (Windows/Mac)
• Install python3-tk (for GUI):
  • sudo apt-get install python3-tk (Linux)

Usage

git clone https://github.com/leksik-phew/VoiceEmoji.git
cd VoiceEmoji
pip install -r requirements.txt
python main.py

Feedback: tg @bez_organov