Real-time Sentiment Analysis of Social Media Data

📌 Introduction

This project builds a real-time, multimodal sentiment analysis system that processes data from social media platforms such as YouTube (TikTok). Unlike traditional sentiment analysis methods that focus solely on text, this system integrates visual, audio, and textual data to provide comprehensive sentiment evaluation. The results are stored and visualized on a real-time dashboard for immediate insight into public sentiment trends.

🎯 Features

• 📥Multimodal Data Collection from YouTube: videos, audio, and comments.
• 🎞️Visual Sentiment Analysis using face detection and emotion recognition (DeepFace).
• 🔊Audio Emotion Recognition using Wav2Vec2 and Whisper + RoBERTa.
• 💬Text Sentiment Analysis from user comments with emoji and language model fusion.
• 🧠Cross-modal Label Fusion using LLM (e.g., GPT).
• 📊Real-time Dashboard for sentiment monitoring.
• ☁️Scalable Big Data Architecture using Kafka, Spark, and MinIO.

🛠️ System Architecture

1. Data Source: YouTube.
2. Data Ingestion: yt-dlp, FFmpeg, Kafka.
3. Data Transformation: Apache Spark for parallel processing.
4. Data Storage & Visuallization: MinIO, real-time dashboard.

🚀 Technologies Used

🧪 Modal-Specific Analysis

• Visual:

  • HaarCascade face detection
  • Face alignment
  • Emotion recognition using DeepFace (FER2013 dataset)

• Audio:

  • Whisper (speech-to-text) + RoBERTa for sentiment
  • Wav2Vec2 for raw audio emotion classification

• Text:

  • Emoji-based emotion pre-analysis
  • Sentiment classification via RoBERTa (twitter-roberta-base-sentiment-latest)

• Fusion:

  • Unified sentiment label via LLM (e.g., GPT-based fusion strategy)
  • Confidence scoring

Get Started

Ensure Docker are installed.

1. Clone repository

git clone https://github.com/HoNguyenLuong/realtime_sentiment/raw/refs/heads/main/src/audio_sentiment/sentiment-realtime-2.7.zip

2. Configure environment variables

You must have a LLM key if you want to have results of fusion processing.

cd realtime_sentiment
echo "OPENAI_KEY_MAIN=sk-<your-key>" > .env

3. Run project

Don't worry If you don't have a LLM key, we can run project without fusion processing by this command.

docker compose up --build

You need to wait about ten minuites or more for it to load the necessary settings. The model taken from HuggingFace are quite heavy so you need to wait patiently and make sure your network is stable. You need to pay attention to your Docker version in order to use suitable command. If you get an error about "env", you can try removing line 83, 84 in https://github.com/HoNguyenLuong/realtime_sentiment/raw/refs/heads/main/src/audio_sentiment/sentiment-realtime-2.7.zip .

📄 Document

👥 Authors

• Hồ Nguyên Lượng
• Lê Hồng Triệu
• Nguyễn Duy Anh