Advanced Video Inpainting Platform

A modular, production-grade, cloud-native video inpainting and enhancement platform featuring state-of-the-art AI models for object removal, background replacement, and video restoration.

🔥 Latest: Next-Generation SOTA Integration (CVPR 2025+ Level)

We've implemented four revolutionary approaches representing the cutting-edge of video inpainting research:

🎯 3D-Aware Scene Coherence

• Research: MICCAI 2024 "Depth-Aware Endoscopic Video Inpainting" + CVPR 2020 3D Photo Inpainting
• Features: Monocular depth estimation, geometric consistency, perspective-aware inpainting
• Performance: 2% PSNR improvement over 2D methods
• Use Cases: Architectural scenes, depth-dependent content, geometric objects

🎵 Audio-Visual Coherence

• Research: UnAvgLip + Environmental Audio-Visual Synchronization
• Features: High-fidelity lip-syncing, identity preservation, 40ms sync window
• Formats: WAV, MP3, FLAC, M4A, OGG
• Use Cases: Talking heads, dubbing, audio-synchronized content

🎮 Advanced Interactive Controllability

• Research: BlobGEN-Vid + IM-Zero approaches
• Features: Compositional scene control (8 regions), zero-shot motion, trajectory animation
• Controls: Static, linear, bezier, spline, custom motion types
• Use Cases: Interactive editing, motion graphics, complex scene composition

⚡ Efficiency Optimization

• Features: 4 performance profiles, advanced quantization (8-bit/4-bit), memory optimization
• Performance: 1-4x speed improvement, 50-75% memory reduction
• Profiles: Max Quality (24GB), Balanced (16GB), Max Speed (8GB), Memory Efficient (4GB)
• Use Cases: Resource-constrained environments, real-time processing

---

Legacy SOTA Models (Still Available):

FloED (Flow-guided Enhanced Diffusion)

FloED combines optical flow estimation with diffusion models to generate seamless, temporally consistent results. Our implementation features:

• Dual-branch architecture: Separate flow completion and diffusion inpainting branches
• Flow-guided feature propagation: Uses completed flow fields to guide the diffusion process
• Text-conditional generation: CLIP-based text conditioning for controlled inpainting
• Memory-efficient processing: Supports chunked processing for long videos
• FP16 optimization: Reduced memory footprint with half-precision computation

Key benefits of FloED:

• Better temporal consistency with flow-guided diffusion
• Reduced flickering between consecutive frames
• Higher fidelity for complex motions
• More realistic textures in inpainted regions
• Controllable generation: Supports text-guided inpainting via diffusion conditioning

ProPainter (Transformer-based with Flow Priors)

ProPainter uses a transformer-based architecture with a dual-branch design for flow completion and content generation.

Key benefits of ProPainter:

• Long-range dependencies captured by transformer attention
• Reference frame utilization for improved consistency
• Sliding window processing for handling longer videos
• Flow-guided propagation for coherent motion
• Transformer architecture: Uses self-attention for capturing spatial context
• Memory-efficient processing: Supports chunked processing for long videos
• Half-precision support: FP16 mode for lower memory usage

🚀 Features

• Modular Pipeline Architecture: Mix and match segmentation, inpainting, and enhancement models
• Plugin System: Easily extend with new AI models and algorithms
• SOTA AI Integration:
  • Grounding DINO + SAM: Text-guided object detection and segmentation
  • FloED and ProPainter: State-of-the-art video inpainting models
  • AnimateDiff: For enhanced temporal coherence
• Production-Ready: Error handling, monitoring, and cloud deployment
• Web UI: Advanced annotation, job management, and result visualization
• Flow-guided processing: Leverages optical flow for temporal consistency

🛠️ Architecture

The platform uses a modular, plugin-based architecture with these core components:

1. Segmentation: Text-guided object detection (Grounding DINO) and segmentation (SAM)
2. Inpainting: Fill masked regions with content-aware generation
3. Enhancement: Upscaling, face restoration, and other enhancements
4. Coherence: Temporal consistency filtering and optimization

📚 Examples

Text-guided Segmentation

We provide an example script demonstrating text-guided segmentation using Grounding DINO with SAM:

# Download the required models
python download_grounding_dino_sam.py

# Run the example
cd examples
python text_guided_segmentation.py --input <image_path> --prompt "person" --visualize

See the examples directory for more detailed instructions and additional examples.

🚀 Getting Started

Quick Start with Docker (Recommended)

The easiest way to get started is using our Docker deployment:

# Clone the repository
git clone https://github.com/yourusername/video-inpainting-platform.git
cd video-inpainting-platform

# One-command deployment (Linux/macOS)
chmod +x deploy.sh
./deploy.sh full

# Windows users
deploy.bat full

This will:

1. Build the Docker image with all dependencies
2. Download all required models (Grounding DINO, SAM, FloED, ProPainter)
3. Start the web interface at http://localhost:5000

See our comprehensive Docker Deployment Guide for detailed instructions.

Native Linux Installation (Recommended for Performance)

For optimal performance and direct system access on Linux:

# Clone the repository
git clone https://github.com/yourusername/video-inpainting-platform.git
cd video-inpainting-platform

# Run the automated installer (supports Ubuntu, CentOS, Debian, Arch)
chmod +x install_linux.sh
./install_linux.sh

# Start the platform
./start_server.sh

Linux Requirements:

• Ubuntu 18.04+ / CentOS 7+ / Debian 10+ / Arch Linux
• NVIDIA GPU with CUDA support (RTX 3060+ recommended)
• 16GB+ RAM (32GB+ recommended for HD processing)
• 50GB+ free storage space
• Python 3.8-3.11

The installer will:

1. Install system dependencies (CUDA, FFmpeg, etc.)
2. Set up Python virtual environment
3. Install PyTorch with CUDA support
4. Download all AI models
5. Create startup scripts and configuration

For detailed Linux installation instructions, troubleshooting, and advanced configuration, see LINUX_NATIVE_SETUP.md.

RunPod Cloud GPU Installation (Recommended for Cloud)

For cloud GPU deployment on RunPod:

# Deploy on RunPod (no sudo required)
cd /workspace
git clone https://github.com/yourusername/video-inpainting-platform.git
cd video-inpainting-platform
chmod +x install_runpod.sh
./install_runpod.sh

# Start the platform
./start_runpod.sh

RunPod Requirements:

• RTX 3060+ GPU (RTX 4090 recommended)
• PyTorch 2.1+ template
• Network volume for persistent storage
• Exposed ports: 5000, 7860, 8080

RunPod Features:

• ✅ No sudo access required
• ✅ Persistent storage via network volumes
• ✅ GPU-optimized configurations
• ✅ RunPod proxy URLs for easy access
• ✅ Cost-effective spot instances

Access via: https://<pod-id>-5000.proxy.runpod.net

For detailed RunPod setup instructions, see RUNPOD_SETUP.md.

Manual Installation (Cross-platform)

1. Clone the repository:

git clone https://github.com/yourusername/video-inpainting-platform.git
cd video-inpainting-platform

2. Install dependencies:

pip install -r requirements.txt

3. Download models:

# Download all models (recommended)
python download_all_models.py

# Or download specific models
python download_grounding_dino_sam.py  # For text-guided segmentation
python download_floed_models.py        # For FloED inpainting
python download_propainter_models.py   # For ProPainter inpainting

Usage

Web Interface (Recommended)

# Docker deployment
./deploy.sh start  # Linux/macOS
deploy.bat start   # Windows

# Manual installation
cd web
python app.py

Then open your browser at http://localhost:5000

Command Line Interface

python run_inpainting.py --input <video_path> --remove "person" --inpaint "background"

📦 Requirements

System Requirements

• Operating System: Linux (Ubuntu 20.04+), macOS, or Windows 10/11
• GPU: NVIDIA GPU with 8GB+ VRAM (16GB+ recommended for HD videos)
• RAM: 16GB+ system RAM
• Storage: 50GB+ free disk space (for models and temporary files)

Software Requirements

Docker Deployment (Recommended)

• Docker Engine 20.10+
• Docker Compose
• NVIDIA Container Toolkit (for GPU support)

Manual Installation

• Python 3.8+
• PyTorch 1.10+
• CUDA 11.3+ (for GPU acceleration)
• See requirements.txt for all dependencies

🐳 Docker Deployment

We provide comprehensive Docker support for easy deployment:

• Dockerfile: Production-ready container with all dependencies
• docker-compose.yml: Multi-service orchestration
• deploy.sh / deploy.bat: Automated deployment scripts
• DEPLOYMENT.md: Comprehensive deployment guide

Quick Commands

# Linux/macOS
./deploy.sh [command]

# Windows
deploy.bat [command]

# Available commands:
# full      - Complete setup (build + download + start)
# build     - Build Docker image
# download  - Download models
# start     - Start platform
# stop      - Stop platform
# status    - Show status
# logs      - Show logs
# shell     - Interactive shell
# update    - Update platform
# cleanup   - Cleanup containers

🔧 Configuration

The platform uses YAML configuration files for pipeline customization:

ProPainter Configuration

Key settings in resources/propainter_config_example.yaml:

# Model paths
model_path: 'models/propainter/propainter_m9.pth'
flow_model_path: 'models/raft/raft_things.pth'

# Memory optimization
max_sub_video_length: 80  # Reduce if facing OOM issues
fp16_mode: true          # Use half-precision for memory efficiency  

# Processing parameters
resize_ratio: 1.0        # Resize ratio for input frames

FloED Configuration

Key settings in resources/floed_config_example.yaml:

# Model paths
model_path: 'models/floed/model.ckpt'
flow_model_path: 'models/raft/raft_things.pth'

# Diffusion parameters
num_inference_steps: 50
guidance_scale: 7.5

See resources/floed_config_example.yaml and resources/propainter_config_example.yaml for complete examples.

Memory Optimization Tips

For processing large videos with limited GPU memory:

1. Enable half-precision mode (fp16_mode: true)
2. Reduce max_sub_video_length to process fewer frames at once
3. Set resize_ratio below 1.0 to downscale input resolution
4. For ProPainter, reduce neighbor_length and increase ref_stride

🧩 Extending the Platform

Adding Custom Plugins

1. Create a new class that extends the base class for your component
2. Register it with the plugin registry
3. Use it in your pipeline configuration

Example:

# 1. Create custom inpainter
class MyCustomInpainter(VideoInpainter):
    def __init__(self, model_loader):
        super().__init__(model_loader)
        # Your initialization code...
    
    def inpaint_batch(self, frames, masks, prompt, ...):
        # Your implementation...

# 2. Register with plugin registry
from core.plugin_registry import plugin_registry
plugin_registry.register('inpainting', 'my_custom', MyCustomInpainter)

# 3. Use in configuration
# pipeline.plugins.inpainting: "my_custom"

📋 Roadmap

See PROJECT_ROADMAP.md for the full development roadmap.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• ProPainter implementation: sczhou/ProPainter
• RAFT optical flow: princeton-vl/RAFT