DeepMeet Guard is a dual-purpose AI security research platform targeting one of the most critical emerging threats in remote hiring:
AI-powered audio fraud during live online interviews.
It simultaneously demonstrates how AI-driven audio spoofing attacks operate in real-time,
and deploys a multi-layer detection system to identify and report fraudulent audio to the interviewer.

📖 Setup Guide · 🚀 Quick Start · 🏗️ Architecture · 📡 API Reference · 🤝 Contributors

📋 Table of Contents

• Overview
• Motivation
• Key Features
• System Workflow
• System Architecture
• Project Structure
• Backend — Server
• Frontend — Client
• API Reference
• Setup Instructions
• Running the Project
• Environment Variables
• Technologies Used
• Troubleshooting
• Future Improvements
• Contributors
• License

🔍 Overview

DeepMeet Guard is a graduation research platform engineered with two deeply integrated AI subsystems:

This dual-sided architecture makes DeepMeet Guard both a security research tool and a corporate fraud prevention platform.

💡 Motivation

The rapid commoditization of voice synthesis and large language models has made it trivially easy for bad actors to impersonate candidates in remote interviews. A technically sophisticated actor can now:

• Clone any target voice from a short audio sample
• Deploy an LLM that answers domain-specific questions in real-time
• Route synthesized audio through virtual audio drivers — completely undetected

Existing interview platforms offer no protection against this attack vector.

DeepMeet Guard was built to:

1. Demonstrate the full attack surface through a working, end-to-end simulation
2. Defend against it through a real-time, multi-model audio integrity analysis pipeline
3. Inform organizations about the maturity and accessibility of this threat

✨ Key Features

🎭 Simulation Engine (Attacker Side)

• 🎙️ Real-time Speech-to-Text (STT) — Transcribes the interviewer's spoken question using a Vosk-based offline model
• 🧠 Multi-Provider LLM Response Generation — Supports OpenAI, Gemini, Claude, Cohere, and Ollama, grounded in the candidate's knowledge base via LangChain
• 🔊 Neural Text-to-Speech (TTS) — Synthesizes natural speech using NeuCodec-based NeuTTS with eSpeak-NG phonemization
• 🧬 Zero-Shot Voice Cloning — Applies the candidate's enrolled voice profile to TTS output for seamless impersonation
• 📚 Knowledge Base Integration — Candidate-defined knowledge documents serve as the ground truth for all LLM answers
• ⚡ Low-latency Pipeline — Optimized for near-real-time performance in live interview conditions

🛡️ Detection Engine (Defender Side)

• 🔬 4-Layer Ensemble Detection — Combines Spectra0, ViT, RawNet2, and liveness detection models with configurable weighted voting
• 📊 Confidence Scoring — Returns a weighted probability verdict: REAL / AI-GENERATED
• 🚨 Interviewer Reporting — Delivers period-by-period detection results directly to the interviewer's interface
• 🔄 Continuous Session Monitoring — Captures and analyzes audio in configurable time periods across the full interview

🌐 Platform

• 🖥️ Full-Stack Web Application: — Unified Next.js interface for both simulation and detection workflows
• 🔌 RESTful API — Clean, documented FastAPI endpoints with automatic Swagger UI
• 🤝 Multi-Agent LangGraph Workflow — Coordinated pipeline of specialized AI agents handling each stage of simulation

🔄 System Workflow

Simulation Pipeline

Interviewer speaks
       │
       ▼
  [STT Module — Vosk]
  Transcribes speech to text offline
       │
       ▼
  [LangChain + LangGraph Agent]
  Retrieves relevant context from candidate's knowledge base
  Generates a tailored answer via configured LLM provider
       │
       ▼
  [NeuTTS + eSpeak-NG Phonemizer]
  Converts text to natural speech with phoneme-level control
       │
       ▼
  [Voice Cloning Module]
  Applies candidate's reference voice to synthesized audio
       │
       ▼
  Fake audio streamed back to the interviewer

Detection Pipeline

Audio stream captured from interviewee
       │
       ▼
  [Audio Capture — sounddevice / soundcard]
  Segments audio into configurable time periods
       │
       ▼
  [Parallel 4-Layer Detection]
  ┌─────────────┬──────────────┬───────────────┬──────────────────┐
  │  Spectra0   │    ViT       │   RawNet2     │ Behaviour/Live.  │
  │ (w=0.42)    │ (w=0.26)    │  (w=0.172)    │  (w=0.148)       │
  └─────────────┴──────────────┴───────────────┴──────────────────┘
       │
       ▼
  [Weighted Ensemble Scorer]
  Aggregates layer verdicts into final confidence score
       │
       ▼
  [Reporting Module]
  Result + confidence delivered to interviewer dashboard per period

🏗️ System Architecture

graph TD
    subgraph CLIENT ["🌐 Web Application (Next.js 16 / React 19)"]
        UI[Candidate Interface\nSimulation Controls]
        Dashboard[Interviewer Dashboard\nLive Detection Feed]
    end

    subgraph SERVER ["⚙️ FastAPI Backend (Python 3.11)"]
        API[FastAPI Gateway\nUvicorn ASGI]

        subgraph SIM ["🎭 Simulation Engine"]
            STT[Vosk STT\nOffline Speech-to-Text]
            LLM[LangChain / LangGraph\nMulti-Provider LLM Agent]
            KB[Knowledge Base\nDocument Store]
            TTS[NeuTTS + eSpeak-NG\nNeural TTS Engine]
            VC[Voice Cloning\nNeuCodec / Perth]
        end

        subgraph DET ["🛡️ Detection Engine (4-Layer Ensemble)"]
            CAP[sounddevice / soundcard\nAudio Capture]
            L1[Layer 1 — Spectra0\nweight 0.42]
            L2[Layer 2 — ViT\nweight 0.26]
            L3[Layer 3 — RawNet2\nweight 0.172]
            L4[Layer 4 — Liveness\nweight 0.148]
            SCORE[Weighted Ensemble\nVerdictScorer]
        end

        REPORT[Reporting Module\nPeriod-based JSON Reports]
    end

    UI -->|REST /deepmeet/simulator/*| API
    Dashboard -->|REST /deepmeet/detector/*| API

    API --> STT
    STT --> LLM
    LLM --> KB
    LLM --> TTS
    TTS --> VC
    VC -->|Synthesized Audio Response| API

    API --> CAP
    CAP --> L1 & L2 & L3 & L4
    L1 & L2 & L3 & L4 --> SCORE
    SCORE --> REPORT
    REPORT -->|Verdict + Confidence| Dashboard

📁 Project Structure

DeepMeet-Gaurd/
├── LICENSE
├── README.md
└── src/
    ├── main.py                        # FastAPI application entry point
    ├── requirements.txt               # Python dependencies
    ├── .env.example                   # Environment variable template
    │
    ├── assets/
    │   ├── simulator_assets/          # Candidate voice samples & knowledge base storage
    │   └── detector_assets/           # Meeting session audio & detection reports
    │
    ├── client/                        # Next.js 16 Frontend
    │   ├── app/                       # App Router pages & layouts
    │   ├── components/                # Reusable React components (shadcn/ui + Radix UI)
    │   ├── hooks/                     # Custom React hooks
    │   ├── lib/                       # Utility functions & API client
    │   ├── styles/                    # Global CSS styles
    │   ├── public/                    # Static assets
    │   ├── package.json
    │   ├── tsconfig.json
    │   └── next.config.mjs
    │
    └── server/                        # Python FastAPI Backend
        ├── app_defaults/              # Default reference audio & text for TTS
        ├── controllers/               # Request handling logic
        ├── helpers/                   # Configuration loaders & shared utilities
        ├── infrastructure/            # Model files: STT (Vosk), liveness detection
        ├── models/
        │   ├── enums/                 # Shared enumerations
        │   ├── interfaces/            # Abstract base classes
        │   └── schemas/               # Pydantic request/response schemas
        ├── routers/                   # FastAPI route definitions
        │   ├── health.py              # GET /health
        │   ├── data.py                # POST /deepmeet/simulator/data/*
        │   ├── setup.py               # POST /deepmeet/simulator/setup/*
        │   ├── communication.py       # POST /deepmeet/simulator/communication/*
        │   └── detection.py           # POST|GET /deepmeet/detector/*
        ├── services/
        │   ├── simulator.py           # Simulator service singleton
        │   └── detector.py            # Detector service singleton
        ├── usecases/                  # Business logic layer
        ├── utilities/                 # Audio processing, session management helpers
        └── views/                     # Streamlit debug/monitoring views

🐍 Backend — Server

The server is a Python 3.11 application built on FastAPI, orchestrating two specialized AI subsystems.

Simulation Engine Components

Detection Engine — 4-Layer Ensemble

Additional libraries: Jabberjay 0.0.11, librosa 0.11, torchaudio ≥2.11, ONNX Runtime 1.23, transformers, scikit-learn 1.8, xgboost

Server Configuration (FastAPI)

# main.py
app = FastAPI(title="DeepMeet Guard API", version="1.0.0")

# CORS origins
allow_origins = ["http://localhost:3000", "http://localhost:3001", "http://127.0.0.1:3000"]

🌐 Frontend — Client

The client is a Next.js 16 application written in TypeScript 5.7 with React 19, providing:

• Candidate View — Simulation interface: info upload, voice enrollment, knowledge base upload, and simulation control
• Interviewer Dashboard — Real-time audio monitoring with live detection verdict and confidence score display
• Session Management — Interview session creation, cloning, and full lifecycle control

Frontend Technology Stack

📡 API Reference

All endpoints are served by the FastAPI backend. Interactive Swagger docs are available at http://localhost:8000/docs when the server is running.

🔧 Health

🎭 Simulator — Data Upload

Prefix: /deepmeet/simulator/data

⚙️ Simulator — Setup

Prefix: /deepmeet/simulator/setup

🗣️ Simulator — Communication

Prefix: /deepmeet/simulator/communication

🛡️ Detector

Prefix: /deepmeet/detector

⚙️ Setup Instructions

Prerequisites

🐍 Server Setup

Step 1 — Clone & Navigate

git clone https://github.com/3bdelmoemn/DeepMeet-Gaurd.git
cd DeepMeet-Gaurd/src

Step 2 — Create Python Virtual Environment

Requirements :

• python 3.11

Install Python Using "Miniconda" :

1. Download and install Miniconda From >>> Her

2. Create A New Conda-Enviroment Using The Followed Command :

conda create -n dmg python=3.11

3. Activate Custom Conda-Enviroment Using The Followed Command:

conda activate dmg

Step 3 — Install eSpeak-NG (TTS Dependency)

⚠️ Windows only. Linux users: sudo apt install espeak-ng

winget install -e --id eSpeak-NG.eSpeak-NG

Set required environment variables (run as Administrator):

setx PHONEMIZER_ESPEAK_LIBRARY "C:\Program Files\eSpeak NG\libespeak-ng.dll"
setx PHONEMIZER_ESPEAK_PATH "C:\Program Files\eSpeak NG"

Verify installation:

espeak-ng --version

Verify Python integration:

python -c "from phonemizer import phonemize; print(phonemize('hello world', language='en-us'))"
# Expected: h ə l oʊ  w ɜː l d

💡 Note: The repository already includes all required NeuTTS / NeuCodec-related files. Do not clone or download any additional TTS repositories.

Step 4 — Download the STT Model

Download the Vosk STT model from Google Drive:

📥 Download STT Model — vosk-model-en-us-0.22

After downloading:

1. Extract the archive
2. Place the extracted folder at:

src/server/infrastructure/stt/vosk-model-en-us-0.22/

The path must match STT_MODEL_PATH in your .env file.

Step 5 — Configure Environment Variables

cp .env.example .env

Open .env and fill in all required values. See Environment Variables for a complete reference.

Step 6 — (Optional) Ollama LLM via Google Colab

If using LLM_MODE=local with Ollama tunneled via Colab:

1. Upload the notebook src/server/notebooks/ollama_setup.ipynb to Google Colab
2. Run all cells and insert your ngrok API key when prompted
3. Copy the generated public tunnel URL
4. Paste it into .env as OLLAMA_BASE_URL

Step 7 — Install Python Dependencies

pip install -r requirements.txt

✅ All required packages are listed in requirements.txt. Do not install additional packages manually.

Step 8 — Configure Hugging Face Authentication

Some models require a Hugging Face access token before they can be downloaded.

1. Visit: hugging-face 🤗
2. Create a new token with Read permissions.
3. Copy your token and run:

hf auth login

4. Paste your token when prompted and press Enter.

That's it — the token will be stored locally and used automatically when downloading Hugging Face models.

🌐 Client Setup

Step 1 — Navigate to Client Directory

cd src/client

Step 2 — Install Node.js Dependencies

npm install

Step 3 — Configure Client Environment

cp .env.example .env.local

Set the backend API URL:

NEXT_PUBLIC_API_URL=http://localhost:8000

🚀 Running the Project

Start the Backend Server

# From the src/ directory with venv activated
uvicorn main:app --reload --port 8000

Start the Frontend

# From src/client/
npm run dev

Build for Production

cd src/client
npm run build
npm start

🔐 Environment Variables

Copy src/.env.example to src/.env and fill in all values.

App & Storage

LLM Configuration

TTS Configuration

STT Configuration

Detection Configuration

Refer to .env.example for a fully annotated reference with inline documentation.

🛠️ Technologies Used

Backend

Frontend

Infrastructure & AI Models

🛠️ Troubleshooting

setx PHONEMIZER_ESPEAK_LIBRARY "C:\Program Files\eSpeak NG\libespeak-ng.dll"
setx PHONEMIZER_ESPEAK_PATH "C:\Program Files\eSpeak NG"

src/server/infrastructure/stt/vosk-model-en-us-0.22/

STT_MODEL_PATH=server/infrastructure/stt/vosk-model-en-us-0.22

python --version   # Must output Python 3.11.x

py -3.11 -m venv venv

POST http://localhost:8000/deepmeet/detector/end

🔮 Future Improvements

• WebRTC Integration — Replace REST audio uploads with real-time streaming via WebRTC for sub-second latency
• Multi-Language Support — Extend STT, LLM, and TTS components beyond English
• Browser Extension — Package detection as a Chrome extension for Google Meet, Zoom, and Teams
• Session Analytics Dashboard — Post-session reports with timeline-annotated detection events and charts
• Docker + Docker Compose — One-command containerized setup for all services
• CI/CD Pipeline — Automated testing and deployment via GitHub Actions
• Mobile Client — React Native companion app for on-device monitoring
• Enterprise Webhook API — Integration for ATS and HR platforms
• GPU Acceleration — CUDA-optimized inference paths for all 4 detection layers
• Real-time WebSocket Feed — Push-based detection results instead of polling

👥 Contributors

📄 License

This project is licensed under the Apache License 2.0.
See the LICENSE file for full details.

DeepMeet Guard · Built to expose the threat. Engineered to stop it.

For research and educational purposes. Use responsibly.