lipsync_pipeline

Movie-dubbing-grade lip sync: given a video and a replacement audio track (a dub, a voiceover, a translated line reading), it detects the speaking face, syncs the mouth to the new audio with LatentSync (diffusion-based, Whisper-conditioned), and restores/sharpens the result. Rewritten from the original lipsync_pro.sh / lipsync0.2_pro.sh bash scripts as a proper Python package — same pipeline, four bugs fixed.

What it does

1. Pre-processes video (25fps H.264) and audio (16kHz mono, loudness-normalized).
2. Extracts Whisper encoder embeddings from the audio — this is what actually conditions LatentSync's diffusion UNet on mouth shape, not the raw waveform. WhisperX forced alignment runs alongside it as a diagnostic (non-fatal if it fails).
3. Detects every face in the video, tracks them across frames (IoU matching), and either auto-picks the speaking face (by mouth-region motion variance) or lets you pick interactively.
4. Crops just that face, runs LatentSync on the crop, and Poisson-blends the synced result back into the original frame — everyone else in frame is untouched.
5. CodeFormer face restoration + Real-ESRGAN sharpening.
6. Final mux with cinema-grade audio (48kHz stereo AAC 320k).

Install

bash setup.sh              # creates .venv, installs everything correctly
source .venv/bin/activate

Usage

python -m lipsync_pipeline video.mp4 dub.wav out.mp4
python -m lipsync_pipeline video.mp4 dub.wav out.mp4 --face pick     # interactive
python -m lipsync_pipeline video.mp4 dub.wav out.mp4 --face 1        # force face index
python -m lipsync_pipeline video.mp4 dub.wav out.mp4 --skip-restore --skip-sr

Run python -m lipsync_pipeline --help for the full flag list.

Web UI

python app.py

Opens a local Gradio UI (http://127.0.0.1:7860): upload a video and an audio track, click Analyze to see the detected faces (auto-picked by mouth motion, or pick a different one from the radio list), then Run lip sync. Same pipeline as the CLI, just point-and-click.

Fixed vs. the original scripts

A note on use

This does face-region video editing — the same underlying capability as face-swap/deepfake tools, just aimed at dubbing. It has no consent or provenance checks built in; that's on you. If you're distributing output or attaching your name to it, it's worth: using it on your own or licensed footage with willing subjects, and considering a metadata tag or watermark on output so it's clear the audio was replaced. None of that is enforced here — it's just the responsible-use context that's easy to skip when you're just trying to get a pipeline running.