OmniPlay

AI-Powered Video Dubbing OmniPlay Vision: The Future of Media Consumption for all Platforms

Features

Core Dubbing Capabilities

• Audio Extraction: Advanced vocal separation using Spleeter, Librosa, and FFmpeg
• Speech Transcription: High-accuracy transcription with speaker diarization using Whisper
• Voice Cloning: AI-powered voice synthesis maintaining speaker characteristics
• Multi-Language Dubbing: Support for 100+ languages via Google Translate
• Background Preservation: Maintains original background music and ambiance
• Multi-Track Output: Creates videos with switchable original/dubbed audio tracks
• Subtitle Generation: Automatic SRT subtitle creation with timing synchronization

Quick Start

GUI Mode (Recommended)

OmniPlay aims to revolutionize how global audiences consume video content by eliminating language barriers through real-time AI dubbing, while providing creators and platforms with advanced media protection and analytics capabilities.

Phase 1: Current Desktop Engine (Active)

What we have now:

• Complete offline dubbing pipeline
• Multi-language support with voice cloning
• Professional-quality output with background preservation
• User-friendly GUI and CLI interfaces Future Real-Time Media Platform**

A complete automated solution for dubbing videos using advanced AI technologies. Transform videos with original language audio into professionally dubbed versions with background music preservation and multi-track audio support.

Future Vision: OmniPlay aims to become a web-based embeddable player that performs real-time dubbing with advanced media protection features, revolutionizing how content is consumed across languages and platforms.

Future Roadmap (OmniPlay Web Platform)

• Web-Based Player: Embeddable HTML5 player with real-time dubbing capabilities
• Real-Time Processing: Live dubbing during video playback without pre-processing
• Media Protection: Advanced DRM, watermarking, and content security features
• API Integration: RESTful APIs for seamless platform integration
• Cloud Processing: Scalable cloud infrastructure for high-volume processing
• Cross-Platform: Support for web, mobile, and smart TV platforms
• Customizable UI: White-label solutions for content providers
• Analytics Dashboard: Detailed usage analytics and performance metrics

OmniPlay Vision: The Future of Media Consumption

Mission Statement

OmniPlay aims to revolutionize how global audiences consume video content by eliminating language barriers through real-time AI dubbing, while providing creators and platforms with advanced media protection and analytics capabilities.

Phase 1: Current Desktop Engine (Active)

What we have now:

• Complete offline dubbing pipeline
• Multi-language support with voice cloning
• Professional-quality output with background preservation
• User-friendly GUI and CLI interfaces

Phase 2: Web-Based Real-Time Player (In Development)

Target: Q4 2025

• Embeddable HTML5 Player: Drop-in solution for any website
• Real-Time Dubbing: Live processing during video playback
• WebRTC Integration: Low-latency streaming with dubbing
• Progressive Web App: Mobile-first responsive design
• CDN Integration: Global content delivery network support

<!-- Simple embed example -->
<omniplay-player 
  src="https://example.com/video.mp4"
  target-language="es"
  enable-dubbing="true"
  enable-subtitles="true">
</omniplay-player>

Phase 3: Advanced Media Protection (2026)

Enterprise-Grade Security Features:

• Dynamic Watermarking: Invisible, user-specific watermarks
• DRM Integration: Support for Widevine, PlayReady, FairPlay
• Screen Recording Protection: Anti-piracy measures
• Geo-Blocking: Region-specific content access
• Session Analytics: Detailed viewing behavior tracking
• Content Fingerprinting: Automatic copyright protection

Phase 4: Cloud Platform & API Ecosystem (2026-2027)

Scalable Infrastructure:

• Auto-Scaling Servers: Handle millions of concurrent users
• Edge Computing: Process dubbing at edge locations for minimal latency
• API Marketplace: Third-party integrations and plugins
• White-Label Solutions: Custom-branded players for enterprises
• Advanced Analytics: AI-powered content insights and recommendations

System Requirements

Hardware Requirements

• RAM: Minimum 8GB, Recommended 16GB+
• Storage: 5GB+ free space for models and processing
• GPU: CUDA-compatible GPU recommended (optional, will fallback to CPU)
• CPU: Multi-core processor recommended for faster processing

Software Requirements

• OS: Windows 10/11, macOS 10.14+, or Linux (Ubuntu 18.04+)
• Python: 3.8, 3.9, or 3.10 (multiple versions supported in different environments)
• FFmpeg: Required for audio/video processing
• PyQt5: For GUI interface

Installation Guide

Prerequisites Installation

1. Install Python Versions

# Windows - Download from python.org (install 3.8, 3.9, and 3.10)
# macOS
brew install python@3.8 python@3.9 python@3.10

# Ubuntu/Debian
sudo apt update
sudo apt install python3.8 python3.9 python3.10
sudo apt install python3.8-venv python3.9-venv python3.10-venv

2. Install FFmpeg

# Windows - Download from ffmpeg.org or use chocolatey
choco install ffmpeg

# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt install ffmpeg

3. Install PyQt5 (for GUI)

pip install PyQt5

Environment Setup

The project uses 4 separate Python environments for optimal isolation:

Multi-environment architecture ensuring dependency isolation

Root Environment (Python 3.10) - Video Mixing

python -m venv envf
# Windows
envf\Scripts\activate
# macOS/Linux  
source envf/bin/activate
pip install moviepy==1.0.3 pysrt numpy

VideoSplitter Environment (Python 3.8)

cd VideoSplitter
python3.8 -m venv env388
# Windows
env388\Scripts\activate
# macOS/Linux
source env388/bin/activate
pip install -r requirements_splitter.txt

VoiceTranscriber Environment (Python 3.9)

cd VoiceTranscriber
python3.9 -m venv env309
# Windows
env309\Scripts\activate
# macOS/Linux
source env309/bin/activate
pip install -r requirements_transcriber.txt

VoiceCloner Environment (Python 3.10)

cd VoiceCloner
python3.10 -m venv env310
# Windows
env310\Scripts\activate
# macOS/Linux
source env310/bin/activate
pip install -r requirements_cloner.txt

Pipeline Overview

4-Step Automated Process

Step 1: Audio Extraction

Input: video.mp4
Process: VideoSplitter/extract_v0.py (Python 3.8)
Output: temp/step1/
├── input_raw_audio.wav    # Original full audio
├── input_vocals.wav       # Extracted vocals only
└── input_background.wav   # Background music/ambiance

Technologies Used:

• Spleeter: Professional-grade source separation
• Librosa: Advanced audio analysis and processing
• FFmpeg: Fallback audio processing

Step 2: Speech Transcription

Input: temp/step1/input_vocals.wav
Process: VoiceTranscriber/v4.py (Python 3.9)
Output: temp/step2/
├── output.json           # Transcription with timestamps & speakers
└── samples/              # Individual speaker voice samples
    ├── speakerSPEAKER_00_sample.wav
    └── speakerSPEAKER_01_sample.wav

Features:

• Whisper Integration: OpenAI's speech recognition
• Speaker Diarization: Automatic speaker identification and separation
• Timestamp Precision: Accurate timing for perfect lip-sync
• Multi-Language Detection: Automatic source language identification

Step 3: Voice Cloning & Translation

Input: temp/step2/output.json + speaker samples
Process: VoiceCloner/v7.py (Python 3.10)
Output: temp/step3/
├── final_mix.wav         # Complete dubbed audio track
└── output_clips/         # Individual translated segments
    ├── clip_00_SPEAKER_00.wav
    ├── clip_01_SPEAKER_01.wav
    └── ...

Step 4: Video Assembly

Input: Original video + temp/step3/final_mix.wav + temp/step1/input_background.wav
Process: mixer.py (Python 3.10)
Output: C:/Dubbed/video_TIMESTAMP/
├── video_multi_audio.mp4  # Final video with 2 audio tracks
└── subtitles.srt          # Generated subtitles

Multi-Track Features:

• Track 1: Original Audio (preserved exactly)
• Track 2: Dubbed Audio (TTS voices + background mix)
• Background Preservation: Original music and ambiance maintained

Project Structure

OmniPlay/                        # Main project directory
├── run.py                       # Main dubbing pipeline orchestrator
├── gui_launcher.py              # PyQt5 GUI interface  
├── mixer.py                     # Final video assembly
├── README.md                    # This comprehensive guide
├── envf/                        # Root Python environment (3.10)
│
├── VideoSplitter/               # Audio extraction module
│   ├── extract_v0.py           # Multi-algorithm vocal separation
│   ├── requirements_splitter.txt
│   └── env388/                 # Python 3.8 environment
│
├── VoiceTranscriber/            # Speech-to-text module
│   ├── v1.py - v5.py           # Various transcription approaches
│   ├── v4.py                   # Production transcriber (pipeline default)
│   ├── requirements_transcriber.txt
│   └── env309/                 # Python 3.9 environment
│
├── VoiceCloner/                 # Voice synthesis module
│   ├── v7.py                   # Production voice cloner
│   ├── requirements_cloner.txt
│   ├── voice_cloner_model.pkl  # Pre-trained AI model
│   └── env310/                 # Python 3.10 environment
│
└── 📂 temp/                     # Pipeline processing data
    ├── step1/                  # Audio extraction outputs
    ├── step2/                  # Transcription outputs
    └── step3/                  # Voice cloning outputs

Advanced Configuration

Audio Mixing Settings

Edit mixer.py for custom audio balance:

# Audio volume controls
background_volume = 0.3  # Background music (0.0-1.0)
tts_volume = 0.9        # Dubbed voices (0.0-1.0)

# Audio quality settings
audio_bitrate = '192k'   # Output audio bitrate
sample_rate = 44100      # Audio sample rate

Language Configuration

Supported languages (ISO 639-1 codes):

SUPPORTED_LANGUAGES = {
    'en': 'English',     'es': 'Spanish',    'fr': 'French',
    'de': 'German',      'zh': 'Chinese',    'ja': 'Japanese',
    'ko': 'Korean',      'hi': 'Hindi',      'ar': 'Arabic',
    'ru': 'Russian',     'pt': 'Portuguese', 'it': 'Italian',
    'nl': 'Dutch',       'sv': 'Swedish',    'no': 'Norwegian',
    'da': 'Danish',      'fi': 'Finnish',    'pl': 'Polish',
    # ... 80+ more languages supported
}

Voice Cloning Parameters

Customize in VoiceCloner/v7.py:

# Voice similarity threshold (0.0-1.0)
SIMILARITY_THRESHOLD = 0.85

# Speaking rate adjustment
SPEECH_RATE = 1.0  # 1.0 = normal, 0.8 = slower, 1.2 = faster

# Voice characteristics preservation
PRESERVE_PITCH = True
PRESERVE_TONE = True
PRESERVE_ACCENT = False  # Set True to maintain original accent

Performance Optimization

Hardware Acceleration

# Enable GPU acceleration (if available)
export CUDA_VISIBLE_DEVICES=0  # Use first GPU
export TORCH_CUDA_ARCH_LIST="7.5;8.0;8.6"  # Your GPU architecture

# For CPU-only processing
export CUDA_VISIBLE_DEVICES=""

Processing Speed Tips

For Faster Processing:

1. Use SSD Storage: Store temp files on SSD
2. Increase RAM: Close unnecessary applications
3. GPU Acceleration: CUDA-enabled GPU significantly speeds up processing
4. Parallel Processing: The pipeline automatically uses multiple CPU cores

Estimated Processing Times:

Contribution Guidelines

Code Style

• Follow PEP 8 style guidelines
• Use type hints where appropriate
• Add docstrings for all functions
• Include unit tests for new features

Pull Request Process

1. Fork the repository
2. Create feature branch (git checkout -b feature/amazing-feature)
3. Write tests for new functionality
4. Ensure all tests pass (python -m pytest)
5. Update documentation if needed
6. Commit changes (git commit -m 'Add amazing feature')
7. Push to branch (git push origin feature/amazing-feature)
8. Open Pull Request with detailed description

Bug Reports

Include in your bug report:

• Operating system and version
• Python versions used
• Complete error message and traceback
• Steps to reproduce the issue
• Sample input file (if shareable)
• Expected vs actual behavior

Areas for Contribution

• Performance Optimization: Improve processing speed
• Additional Languages: Add support for new languages
• UI/UX Improvements: Enhance GUI interface
• Quality Metrics: Add audio/video quality assessment
• Batch Processing: Support for multiple file processing
• Web Platform Development: Build the real-time web player
• API Development: Create robust APIs for third-party integration
• Media Protection: Implement advanced DRM and watermarking features

License

This project is licensed under the MIT License - see the LICENSE file for details.

Third-Party Licenses

• OpenAI Whisper: MIT License
• Spleeter: MIT License
• MoviePy: MIT License
• PyQt5: GPL v3 / Commercial License

Acknowledgments

Core AI Technologies

• OpenAI Whisper: Revolutionary speech recognition
• Spleeter by Deezer: Professional audio source separation
• Google Translate: Comprehensive language translation
• Hugging Face Transformers: Advanced NLP models

Essential Libraries

• MoviePy: Powerful video editing capabilities
• PyQt5: Professional GUI framework
• NumPy: Fundamental numerical computing
• Librosa: Advanced audio analysis
• FFmpeg: Comprehensive multimedia processing

👥 Community Contributors

Special thanks to all contributors who have helped improve OmniPlay:

• Bug reports and feature requests
• Code contributions and optimizations
• Documentation improvements
• Testing and feedback

Support & Community

Getting Help

• Documentation: Check this README and inline code comments
• Issues: Report bugs on GitHub Issues
• Discussions: Ask questions in GitHub Discussions
• Wiki: Additional tutorials and guides in the project wiki

Show Your Support

If OmniPlay helps you, please:

• Star the repository
• Fork and contribute
• Share with others
• Email feedback: akarsh91140@gmail.com

Stay Connected

• GitHub: github.com/akarsh911/OmniPlay
• Email: akarsh91140@gmail.com

---

Ready to Get Started with OmniPlay?

Quick Start Checklist:

• Install Python 3.8, 3.9, and 3.10
• Install FFmpeg
• Clone the OmniPlay repository
• Set up the 4 Python environments
• Install dependencies in each environment
• Test with a short video sample

First Video Dubbing:

# GUI Mode (Recommended for beginners)
python gui_launcher.py

# CLI Mode (For advanced users)
python run.py "path/to/your/video.mp4"

Expected Output:

C:/Dubbed/video_20250709_123456/
├── video_multi_audio.mp4    # Final video with dual audio tracks
└── subtitles.srt           # Generated subtitles

Congratulations! You're now ready to transform videos with OmniPlay's AI-powered dubbing!

Transform any video into a professionally dubbed, multi-language experience while preserving the original audio quality and background music. The future of video localization is here!

---

What's Next?

• Follow our progress as we build the web-based real-time player
• Join our community to influence the roadmap and features
• Share your feedback to help us build the platform you need
• Stay tuned for the OmniPlay web platform launch in 2025!