BhashaBridge 🎙️🌉

BhashaBridge is an advanced, high-performance Speech-to-Text (ASR) transcription, translation, and AI-assisted post-processing pipeline. It is specifically optimized for bilingual Malayalam-English and English audio processing, delivering hyper-accurate, readable subtitle segments using state-of-the-art deep learning models.

The system consists of a robust FastAPI backend (incorporating Demucs, Silero VAD, Hugging Face Whisper pipelines, and native llama.cpp LLM inference) paired with a responsive Flutter UI frontend.

---

Key Features & Pipeline Architecture

BhashaBridge runs a highly optimized, multi-stage processing pipeline on CUDA-capable GPUs:

[Audio Input] 
     │
     ▼
┌──────────────┐
│ Stage 1:     │ ──► Demucs separates vocals from background noise/music
│ Demucs       │
└──────────────┘
     │
     ▼
┌──────────────┐
│ Stage 2:     │ ──► Silero VAD crops active dialogue segments
│ Silero VAD   │
└──────────────┘
     │
     ▼
┌──────────────┐
│ Stage 2.5:   │ ──► Consistent audio levels across crops
│ Peak Norm    │
└──────────────┘
     │
     ▼
┌──────────────┐
│ Stage 3 & 4: │ ──► whisper-medium-ml (Malayalam) or whisper-large-v3 (English)
│ ASR Pipeline │
└──────────────┘
     │
     ▼
┌──────────────┐
│ Stage 5:     │ ──► Accurate chunk start/end reconstruction
│ Align Timings│
└──────────────┘
     │
     ▼
┌──────────────┐
│ Stage 6:     │ ──► Gemma-4-E4B-it corrects text (Chillu / Lipi structure)
│ Gemma LLM    │     while keeping exact segment structures
└──────────────┘

1. Demucs Vocal Separation

Separates vocals from backing tracks, background noise, or sound effects using the htdemucs model. Ensures Whisper only hears clean dialogue.

2. Voice Activity Detection (VAD)

Uses Silero VAD to accurately isolate dialogue chunks in-memory. This prevents Whisper from hallucinating during long silent gaps.

3. Dynamic Peak Normalization

Normalizes speech crops to a peak amplitude of 0.95 before ASR. This boosts quiet speakers and balances loud ones, improving transcription accuracy.

4. Robust Bilingual ASR Routing

• Malayalam to Malayalam (mal_to_mal): Transcribed via tomalex04/whisper-medium-ml-bf16-v2 with Flash Attention 2.
• Malayalam to English (mal_to_eng): Translated via openai/whisper-large-v3.
• English to English (eng_to_eng): Transcribed via openai/whisper-large-v3.

5. Local LLM Correction Engine

Utilizes gemma-4-E4B-it (running locally via llama-cpp-python with full CUDA offloading). It operates on subtitle segments using deterministic Malayalam grammatical and orthographic rules:

• Chillu Conversion: Fixes ZWNJ/ZWJ letter outputs to proper chillus (e.g. ല്+ZWNJ ➔ ൽ, ര്+ZWNJ ➔ ർ).
• Modern Lipi Alignment: Standardizes prefix vowel signs.
• Structure Preservation: Strict schema verification ensures the output segments exactly match input timestamps and indices.

---

Prerequisites

• Operating System: Linux (tested on Ubuntu)
• Hardware: CUDA-compatible Nvidia GPU (VRAM ≥ 6GB recommended for full pipeline offloading)
• System Tools: ffmpeg (for audio transcoding)
• Environment Manager: Conda (Miniconda/Anaconda)

---

Backend Installation & Setup

1. Environment Setup

Create and activate the conda environment using the provided environment.yml file:

# Clone the repository
git clone <your-repo-url>
cd BhashaBridge

# Create the environment from file
conda env create -f environment.yml

# Activate the environment
conda env activate bhashabridge

Alternatively, to install manual dependencies via requirements.txt:

conda create -n bhashabridge python=3.10 -y
conda activate bhashabridge

# Install PyTorch with CUDA support
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install packaging, ninja (required for compiling custom attention layers)
pip install packaging ninja

# Build and install llama-cpp-python from source with CUDA support
CMAKE_ARGS="-GGUIDE=OFF -DGGML_CUDA=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir

# Install rest of requirements
pip install -r requirements.txt

2. Configure Environment Variables

Create a .env file in the project root:

# Hugging Face Auth Token (for downloading gated repositories/models)
HF_TOKEN=your_hugging_face_write_token_here

# Server Bind Settings
SERVER_HOST=0.0.0.0
SERVER_PORT=8000

# Optional directory to save intermediate audio crops for diagnostics
DEBUG_AUDIO_CLIPS_PATH=audio_clips

---

Running the Backend Server

Start the FastAPI server:

python server_gemma4.py

On first startup, BhashaBridge will automatically download and cache all necessary models (Gemma-4 GGUF quant, Whisper weights, Demucs, Silero VAD). Ensure you have a fast, stable internet connection.

API Documentation (Endpoints)

Once running, interactive API docs are available at http://localhost:8000/docs.

• POST /process_audio: Upload audio file with transcription_mode (mal_to_mal, mal_to_eng, or eng_to_eng). Returns a task_id.
• GET /task_status/{job_id}: Retrieves the status of a queued or running pipeline.
• GET /result/{job_id}: Gets final formatted subtitle segments.
• DELETE /cancel/{job_id}: Cancels a pending queued task.
• POST /post-process: Submits raw segments to the local Gemma model along with a prompt to clean/format/correct the output.

---

Frontend Installation & Setup (Flutter)

The UI is built with Flutter, offering cross-platform support (Linux Desktop, macOS, Windows, Web, Android).

1. Prerequisites

• Flutter SDK installed and configured on your path.

2. Run the App

Navigate to the bhashabridge_ui directory and launch the app:

cd bhashabridge_ui

# Fetch packages
flutter pub get

# Run application (Make sure a target device is connected/running)
flutter run

---

Development & Diagnostics

To run VRAM diagnostics or test local model loading thresholds:

python vram_test.py

This script evaluates how Demucs, Silero VAD, Whisper, and Gemma offload to CUDA without overflowing available GPU memory.