Real-Time Transcription and Translation System

A sophisticated real-time transcription and translation application with separated concerns, featuring voice-to-text conversion, multi-language translation, and Named Entity Recognition (NER) capabilities.

🏗️ Architecture Overview

This project follows a microservices architecture with clear separation of concerns:

┌─────────────────┐
│   Frontend      │  React App (Voice → Text via Chrome Web Speech API)
│  (Port 80)      │
└────────┬────────┘
         │ WebSocket
         │
┌────────▼────────┐
│   Backend       │  Django Channels (Translation + NER Processing)
│  (Port 8000)    │
└────────┬────────┘
         │
┌────────▼────────┐
│     Redis       │  Channel Layer & Message Broker
│  (Port 6379)    │
└─────────────────┘

📋 Features

• Real-Time Voice Transcription: Uses Google Chrome's Web Speech API for client-side speech-to-text conversion
• Multi-Language Translation: Supports translation between multiple languages using Google Translate API
• Named Entity Recognition (NER): Identifies and highlights entities (persons, organizations, locations, etc.) in both source and target languages
• WebSocket Communication: Real-time bidirectional communication between frontend and backend
• Dockerized Deployment: Containerized services for easy deployment and scaling

🎯 Component Responsibilities

Frontend (Real-time-translator/)

Responsibility: Voice input capture and text display

• Captures audio input using Chrome's Web Speech API (react-speech-recognition)
• Converts speech to text in real-time
• Sends transcribed text to backend via WebSocket
• Displays original transcript and translated text with entity highlighting
• Provides language selection and control panel

Technologies:

• React 18
• Chakra UI
• react-speech-recognition (Chrome Web Speech API wrapper)
• WebSocket client

Backend (back2/)

Responsibility: Translation and NER processing

• Receives transcribed text from frontend via WebSocket
• Performs translation using Google Translate API
• Executes Named Entity Recognition on both source and target languages
• Returns translation and entity information to frontend

Technologies:

• Django 4.x
• Django Channels (WebSocket support)
• spaCy (NER models for multiple languages)
• googletrans (Translation service)
• Redis (Channel layer)

Supported Languages for NER:

• English (en_core_web_sm)
• Spanish (es_core_news_sm)
• Italian (it_core_news_sm)
• Arabic (xx_ent_wiki_sm)

Nginx Backend Server (nginx_backend_server/)

Responsibility: Additional NER model serving (optional)

• Contains advanced NER models using Flair and Transformers
• Can serve as a separate NER processing service
• Supports multiple language models for enhanced entity recognition

🚀 Getting Started

Prerequisites

• Docker and Docker Compose
• Google Chrome browser (for voice recognition)
• At least 4GB RAM (for NER models)

Installation

1. Clone the repository

   git clone <repository-url>
   cd RTT_deploy
   

2. Start the services

   docker-compose up --build
   

   This will start:

   • Redis server (port 6379)
   • Django backend (port 8000)
   • React frontend (port 80)
   • Nginx backend server (port 8000)

3. Access the application

   • Open your browser and navigate to http://localhost
   • Important: Use Google Chrome for voice recognition support

📁 Project Structure

RTT_deploy/
├── Real-time-translator/          # Frontend React application
│   ├── src/
│   │   ├── components/            # React components
│   │   │   ├── voiceInput.js      # Voice capture component
│   │   │   ├── translationBoxes.js
│   │   │   └── ...
│   │   ├── pages/
│   │   │   └── translationPage.js # Main translation page
│   │   └── connection/
│   │       └── wsConnection.js    # WebSocket connection
│   ├── Dockerfile
│   └── package.json
│
├── back2/                         # Backend Django application
│   ├── app/
│   │   ├── consumers.py           # WebSocket consumers (Translation + NER)
│   │   ├── routing.py             # WebSocket routing
│   │   └── views.py
│   ├── back2/
│   │   ├── settings.py
│   │   └── asgi.py                # ASGI configuration
│   ├── requirements.txt
│   └── Dockerfile
│
├── nginx_backend_server/          # NER model server
│   ├── NER_models.py              # Advanced NER models
│   └── Dockerfile
│
└── docker-compose.yml             # Service orchestration

🔄 Data Flow

1. Voice Input: User speaks into microphone → Chrome Web Speech API converts to text
2. Frontend Processing: React app captures transcript and sends to backend via WebSocket
3. Backend Processing:
   • Receives transcript with source/target language codes
   • Translates text using Google Translate
   • Performs NER on source language text
   • Performs NER on translated text
4. Response: Backend sends back:
   • Translated text
   • Source language entities (with labels)
   • Target language entities (with labels)
5. Display: Frontend highlights entities in both original and translated text

🛠️ Development

Backend Development

cd back2
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
python manage.py migrate
daphne back2.asgi:channel_layer --port 8000 -b 0.0.0.0

Frontend Development

cd Real-time-translator
npm install
npm start

🔧 Configuration

Environment Variables

The application uses default configurations. For production, consider setting:

• REDIS_URL: Redis connection string
• ALLOWED_HOSTS: Django allowed hosts
• CORS_ORIGINS: Frontend origin URLs

Language Support

The system supports translation between any languages supported by Google Translate. However, NER is currently optimized for:

• English
• Spanish
• Italian
• Arabic

📝 API/WebSocket Protocol

WebSocket Connection

• Endpoint: ws://localhost:8000/ws/translate/
• Protocol: JSON messages

Message Format (Client → Server)

{
  "transcript": "Hello, my name is John.",
  "fromLanguage": "en",
  "toLanguage": "es"
}

Response Format (Server → Client)

{
  "translation": "Hola, mi nombre es John.",
  "speechHighlitedWords": {
    "label": ["PERSON"],
    "entity": ["John"]
  },
  "highlightedWords": {
    "label": ["PERSON"],
    "entity": ["John"]
  }
}

🐳 Docker Services

• redis: Message broker and channel layer
• backend: Django application with WebSocket support
• frontend: React application served via Nginx
• nginx_backend_server: Additional NER processing service

🔍 Troubleshooting

Voice Recognition Not Working

• Ensure you're using Google Chrome browser
• Check microphone permissions in browser settings
• Verify HTTPS is enabled (required for Web Speech API in production)

Translation Issues

• Check internet connection (Google Translate API requires internet)
• Verify language codes are correct (ISO 639-1 format)

NER Not Working

• Ensure spaCy models are downloaded (handled automatically in Docker)
• Check that the language is supported for NER

📄 License

[Add your license information here]

🤝 Contributing

[Add contribution guidelines here]

👥 Authors

[Add author information here]