🌍 AI Video Translator (Local)

Break language barriers with cinema-quality video translation — privately, on your own hardware.

Transform any video into a professional multilingual production with natural voice cloning, lip-sync, and on-screen text translation. No cloud APIs, no subscriptions, no data leaving your machine.

📝 Enjoying the project? Please star it ⭐️! It helps me gauge interest and keep working on new features.

🎬 Demo

🎥 Click the image above to watch the full demo on YouTube!

✨ Why This Project?

Traditional dubbing is expensive, time-consuming, and requires professional studios. AI Video Translator democratizes video localization by bringing Hollywood-grade technology to your desktop:

• 🎬 Content Creators: Expand your audience globally without hiring voice actors
• 🎓 Educators: Make training content accessible in any language
• 📰 Journalists & Documentarians: Localize footage for international audiences
• 🎮 Game Developers: Dub cutscenes and trailers cost-effectively
• 🏢 Businesses: Translate corporate videos, presentations, and webinars
• 🔒 Privacy-Focused Users: Keep sensitive content 100% local

🎯 What It Does

Upload a video, select your target language, and let the AI handle everything:

📹 Input Video (English) → 🤖 AI Pipeline → 📹 Output Video (French, with cloned voice & synced lips)

The full pipeline includes:

1. Vocal Separation — Isolates speech from music/sound effects
2. Transcription — Converts speech to text with word-level precision
3. Translation — Translates text using local LLMs or Google Translate
4. Voice Cloning — Regenerates speech in the target language with the original speaker's voice
5. Lip-Sync — Adjusts mouth movements to match the new audio
6. Visual Text Translation — Detects and replaces on-screen text (subtitles, signs, etc.)
7. Audio Enhancement — Cleans and restores generated speech for broadcast quality

🚀 Key Features

Audio Intelligence

Translation Engine

All local models support context-aware mode using full-transcript context for superior coherence.

Voice Synthesis (TTS)

Visual Enhancements

🌐 Supported Languages

Video Translator supports a wide range of languages for both source and target translation.

Production-Ready

• 🖥️ Friendly Gradio UI — Easy drag-and-drop interface
• 🎛️ Fine-Grained Control — Beam size, VAD settings, voice selection, and more
• 👤 LivePortrait Lip-Sync — State-of-the-art lip synchronizer with TensorRT acceleration support
• 🖼️ Visual Text Translation — Detects, translates, and seamlessly replaces text in video frames (cached for speed)
• 📝 Auto-Generated Subtitles — Exports .srt files alongside translated videos
• 🔄 Smart Segment Merging — Combines choppy phrases into natural sentences
• ⏳ Real-time Progress & ETA — Track detailed progress with estimated time remaining
• 🧹 VoiceFixer Enhancement — Restores and cleans up generated audio for studio quality
• ⚡ GPU Optimized — One-model-at-a-time policy for maximum VRAM efficiency
• 🛡️ Global CPU Fallback — Automatically switches to CPU if GPU fails

🎬 Use Cases

🎥 YouTube & Social Media Creators

"I have 50 English tutorials and want to reach Spanish speakers."

Upload each video, select English → Spanish, and export professional dubs with your cloned voice. No re-recording needed!

🎓 Corporate Training & E-Learning

"Our compliance training is in English but we have offices in 12 countries."

Batch-translate training videos while maintaining the presenter's voice for authenticity. Export with or without subtitles.

🎞️ Film & Documentary Localization

"I want my indie film to premiere at international festivals."

Use LivePortrait (HD) lip-sync for cinema-quality dubbing that doesn't look like a bad overdub.

LivePortrait GPU Acceleration

📢 Marketing & Advertising

"We need our 30-second ad in French, German, and Japanese by tomorrow."

Process multiple language versions simultaneously with local LLM translation for brand-appropriate messaging.

🔐 Sensitive Content Translation

"Our video contains confidential product demos."

Everything runs locally — no data leaves your machine. Perfect for legal teams, medical content, or proprietary information.

🛠️ Prerequisites

winget install ffmpeg
# Restart terminal after installation

sudo apt install ffmpeg

brew install ffmpeg

📦 Installation

🚀 Quick Install (Recommended)

The smart installer auto-detects your platform, GPU, and CUDA version:

# Clone the repository
git clone https://github.com/overcrash66/video-translator.git
cd video-translator

# Windows:
scripts\install.bat

# Linux / macOS:
chmod +x scripts/install.sh
./scripts/install.sh

🔧 Manual Install

py -3.10 -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt

python3.10 -m venv venv
source venv/bin/activate
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install -r deploy/docker/requirements.docker.txt

python3.10 -m venv venv
source venv/bin/activate
pip install torch torchvision torchaudio
pip install -r requirements.txt

# Full install with CUDA 12.4 (default):
pip install -e ".[all]"

# Full install with CUDA 12.8 (RTX 50 series):
pip install -e ".[all-cu128]" --extra-index-url https://download.pytorch.org/whl/cu128

# CPU only:
pip install -e ".[all-cpu]"

# Minimal (core + dev tools only):
pip install -e ".[dev]"

# Selective install (pick what you need):
pip install -e ".[tts,ocr,lipsync]"

✅ Verify Installation

# Run smoke tests to validate your install
pytest tests/test_smoke_install.py -m smoke -v

# Or use the installer's verify mode
python scripts/install.py --verify

Optional Components

🐳 Docker Deployment (GPU)

You can run the application in a Docker container with NVIDIA GPU support.

Prerequisites:

• NVIDIA Driver (compatible with CUDA 12.x)
• Docker Desktop (Windows) or Docker Engine (Linux)
• NVIDIA Container Toolkit (for GPU access inside Docker)

Build and Run (Recommended)

1. Build the image (run from the project root):

   docker build -f deploy/docker/Dockerfile -t video-translator .
   

2. Run the container:

   # For PowerShell:
   docker run --gpus all -p 7860:7860 -v ${PWD}/output:/app/output --name video-translator video-translator
   
   # For Command Prompt (CMD):
   docker run --gpus all -p 7860:7860 -v %cd%/output:/app/output --name video-translator video-translator
   

   Note: If you encounter a "Ports are not available" error (common on Windows with Hyper-V), try mapping to a different port like 7950:

   docker run --gpus all -p 7950:7860 -v ${PWD}/output:/app/output --name video-translator video-translator
   

3. Access the App: Open your browser to http://localhost:7860 (or http://localhost:7950 if you used the alternate port).

🔧 Troubleshooting

"Ports are not available" / "Access is denied" On Windows, Hyper-V or WinNAT often reserves large ranges of ports (including 7860).

• Solution: Use a different host port (like 7950 or 8080) as shown in the note above.
• Check reserved ranges: Run netsh interface ipv4 show excludedportrange protocol=tcp to see which ports are blocked.

🖥️ Usage

Quick Start

# Activate environment
.\venv\Scripts\activate

# Launch the application
python app.py

Open your browser to http://127.0.0.1:7860

Step-by-Step Translation

1. Upload Video — Drag & drop MP4, MKV, or MOV files
2. Select Languages — Source (or Auto-detect) → Target
3. Choose Models:
   • Translation: Google (fast) / Llama 3.1 (quality) / ALMA-R (best)
   • TTS: Edge (online) / F5-TTS (fast cloning) / XTTS (emotion control)
4. Enable Features (optional):
   • ✅ Speaker Diarization — Multi-speaker videos
   • ✅ Lip-Sync — Select quality level (Fast/HD/Cinema)
   • ✅ Visual Text Translation — Replace on-screen text
   • ✅ Audio Enhancement — VoiceFixer post-processing
5. Click "Process Video" and monitor progress

⚙️ Configuration

Directory Structure

video-translator/
├── temp/           # Intermediate files (auto-cleaned)
├── output/         # Final translated videos
├── models/         # Downloaded model weights
└── .env            # Environment variables (HF_TOKEN, etc.)

Environment Variables

HF_TOKEN=your_huggingface_token  # Required for Llama 3.1, NeMo

🧩 Pipeline Architecture

flowchart TD
    Video[Input Video] --> Extract[Extract Audio via FFmpeg]
    Extract --> Separator{"Audio Separator<br/>(HDemucs)"}
    
    Separator -->|Vocals| Vocals[Vocal Track]
    Separator -->|Accompaniment| Background[Background Track]
    
    Vocals --> VAD{"VAD Preprocessing<br/>(Silero VAD)"}
    VAD --> Transcribe{"Transcribe<br/>(Faster-Whisper Turbo)"}
    Transcribe --> Segments[Text Segments]
    Segments --> Merge{Smart Segment Merging}
    
    Vocals -.-> Diarize{"Diarize<br/>(NeMo / SpeechBrain)"}
    Diarize -.-> SpeakerProfiling[Speaker Profiling]
    
    Merge --> Translate{"Translate<br/>(Llama 3.1 / ALMA / HY-MT)"}
    Translate --> SRT[Export .SRT Subtitles]
    
    Translate --> TTS{"Neural TTS<br/>(F5-TTS / XTTS / Edge)"}
    SpeakerProfiling -.-> TTS
    TTS --> TTSAudio[Generated Speech Clips]
    
    TTSAudio --> EQ{EQ Spectral Matching}
    EQ --> Sync{"Synchronize<br/>(PyRubberband)"}
    Sync --> MergedSpeech[Merged Speech Track]
    
    MergedSpeech -.-> VoiceFixer{"Voice Enhancement<br/>(VoiceFixer)"}
    VoiceFixer -.-> Mix
    MergedSpeech --> Mix{Mix Audio}
    Background --> Mix
    
    Mix --> FinalAudio[Final Audio Track]
    
    Video --> VisualTrans{"Visual Translation<br/>(PaddleOCR / EasyOCR)"}
    VisualTrans --> LipSync{"Lip-Sync<br/>(LivePortrait / Wav2Lip / GFPGAN)"}
    MergedSpeech -.-> LipSync
    
    LipSync --> Mux{"Merge with Video<br/>(FFmpeg)"}
    FinalAudio --> Mux
    
    Mux --> Output[Translated Output Video]

🤝 Contributing

Contributions are welcome! Areas where help is appreciated:

• Additional language support and voice models
• Performance optimizations
• Bug fixes and stability improvements
• Documentation and tutorials

🏷️ Citation

If you find this project useful, please cite it as follows:

@software{video_translator,
  author = {Wael Sahli},
  email = {[email protected]},
  title = {AI Video Translator},
  url = {https://github.com/overcrash66/video-translator},
  version = {1.0.0},
  date = {2026-06-15}
}

📄 License

This project is for educational and personal use. Please respect the licenses of underlying models and technologies.