video-dub

Automated video dubbing using ASR, MT and Zero-shot voice cloning

The project implements a prototype of an automated video dubbing system from English. The process includes the following stages: selecting and pre-processing videos, extracting audio tracks, segmenting and converting audio to a format suitable for automatic speech recognition (ASR). An open-source ASR model is used to obtain transcription. Next, the obtained English texts are translated into target language using open-source machine translation (MT). At the final stage, the XTTS-v2 (Coqui) model is used to generate audio with the translated text in the original speaker's voice (Zero-shot voice cloning). The generated audio is synchronized and superimposed on the original video, forming a full-fledged dubbing.

The main focus is on studying the architecture of the XTTS model, its capabilities in the context of zero-shot dubbing, as well as the quality of speech recognition and translation. The project demonstrates the potential of open-source solutions for high-quality multilingual dubbing.

Usage

1. Clone repository git clone https://github.com/gkatenin/video-dub

2. cd video-dub

3. Install the requirements pip install -r requirements.txt

4. Install XTTS-v2 package:

   • git clone https://github.com/idiap/coqui-ai-TTS
   • cd coqui-ai-TTS
   • pip install -e .

5. Copy XTTS-v2 model weights and confiag git clone https://huggingface.co/coqui/XTTS-v2

6. Run the program python -m video_dub -i input.mp4 -o output.mp4 -l ru --verbose

Here, 'input' is the source video file, 'output' is the resulting video file, '-l' is the target two-letter language (currently supported are ru for Russian an zh for Chinese). Turn on verbose output by adding the --verbose (-v) option.