AI Talking Video Generation

A local pipeline for generating AI talking-head videos using open-source lip-sync and portrait animation models. Built as part of ongoing FYP research and development into AI-generated video content.

Overview

This project sets up and runs open-source AI models that generate synchronized talking-head videos from a face image/video and an audio or driving-video input. It's an early-stage exploration into AI video generation techniques, with the goal of eventually fine-tuning a model for a specific use case as part of a larger AI video content platform.

What's Implemented

• Environment setup: Isolated Python 3.10 environment with GPU-accelerated PyTorch (CUDA 12.1), configured for local inference on consumer GPU hardware (tested on NVIDIA GTX 1050, 4GB VRAM)
• Wav2Lip: Audio-driven lip-sync generation — takes a face video and audio clip, outputs a video with lips synced to the audio
• LivePortrait: Portrait animation — takes a source face image and a driving video, transfers the driving video's motion (head pose, expressions, blinking) onto the source face, producing higher visual quality and more natural motion than Wav2Lip
• Debugging/compatibility fixes:
  • Patched compatibility issues between the original (2020) Wav2Lip codebase and modern library versions (librosa's melspectrogram API change)
  • Resolved corrupted/incomplete auto-downloaded checkpoint (face detection model) by clearing cache and re-downloading
  • Resolved dependency conflicts between Wav2Lip's and LivePortrait's requirements (numpy/torch version pinning)
  • Migrated from deprecated huggingface-cli to the current hf CLI for model weight downloads

Tech Stack

• Python 3.10
• PyTorch 2.5.1 (CUDA 12.1)
• Wav2Lip (open-source audio-driven lip-sync model)
• LivePortrait (open-source portrait animation model, KwaiVGI)
• ffmpeg (audio/video processing)
• librosa, OpenCV, insightface (audio/image/face processing)
• Hugging Face Hub (hf CLI) for model weight downloads

Setup

Common environment

1. Clone this repo
2. Create a Python 3.10 virtual environment: python -m venv venv
3. Activate it: venv\Scripts\activate (Windows)
4. Install PyTorch with CUDA support:

   pip install torch==2.5.1+cu121 torchvision==0.20.1+cu121 torchaudio==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu121
   

Wav2Lip

1. cd Wav2Lip
2. Install dependencies: pip install librosa opencv-contrib-python opencv-python tqdm numba
3. Download the pretrained checkpoint (wav2lip_gan.pth) and place it in Wav2Lip/checkpoints/

LivePortrait

1. cd LivePortrait
2. Install dependencies: pip install -r requirements.txt
3. Install the Hugging Face CLI: pip install -U "huggingface_hub[cli]"
4. Download pretrained weights:

   hf download KwaiVGI/LivePortrait --local-dir pretrained_weights
   

Usage

Wav2Lip (audio-driven lip-sync)

python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth --face input_face.mp4 --audio input_audio.wav

Output is saved to results/result_voice.mp4

LivePortrait (motion transfer / portrait animation)

python inference.py -s assets/examples/source/my_face.jpg -d assets/examples/driving/d13.mp4

• -s — source face image (photo or short video)
• -d — driving video (provides the motion/expressions to transfer onto the source)

Output is saved to animations/, including a side-by-side comparison video (*_concat.mp4) and the final result (*.mp4)

Status / Next Steps

This is an early proof-of-concept stage, actively expanding. Completed and planned work:

• Wav2Lip — working end-to-end, audio-driven lip-sync
• LivePortrait — working end-to-end, higher-quality motion transfer, tested with custom source photo
• CogVideoX — text-to-video scene generation, for non-human/background content
• Fine-tune a model on a custom dataset for a specific niche use case (e.g., regional language support)
• Build an automated content pipeline (LLM-generated topic/script → speech synthesis → scene + talking-head generation → stitched final video)
• Develop evaluation metrics to measure output quality against baseline (e.g., LSE-D/LSE-C lip-sync accuracy scores, user studies)
• Agentic scheduling layer for autonomous, recurring content generation

Notes

This project currently uses pretrained, open-source models for initial experimentation and technical proof-of-concept. Fine-tuning and original model/pipeline contributions are planned as the next phase of development, forming the core research contribution of the FYP.