MVSC: Multi-Modal Video Semantic Compression

A semantic video compression framework that leverages Foundation Models (CLIP, CogVideoX, Flux) to enhance compressed video quality through multi-modal guidance including text captions, generative image/video enhancement, and audio-visual alignment.

Overview

MVSC integrates semantic understanding into the video compression pipeline. Instead of relying solely on traditional rate-distortion optimization, it uses:

• Video Captioning via CogVLM2 to generate text descriptions of video content
• Image Enhancement via Flux + ControlNet to reconstruct high-quality frames from compressed latents guided by text prompts
• Video Enhancement via CogVideoX for temporal-consistent video-level reconstruction
• Audio-Visual Guidance via wav2clip / AudioCLIP to align compressed frames with audio semantics

The compression backbone is Scale-Space Flow (SSF2020) from CompressAI, optimized with a combined loss of MSE, BPP (bits per pixel), LPIPS (perceptual loss), and optional audio guidance losses (CLIP loss, direction loss).

Architecture

Input Video
    │
    ├──► Video Captioning (CogVLM2) ──► Text Prompt
    │
    ├──► Video Compression (SSF2020) ──► Compressed Frames (x_rec)
    │                                        │
    │                                        ├──► Image Enhancement (Flux + ControlNet)
    │                                        │        guided by Text Prompt
    │                                        │
    │                                        └──► Video Enhancement (CogVideoX)
    │                                                 guided by Text Prompt
    │
    └──► Audio Embedding (wav2clip / AudioCLIP) ──► Audio-Visual Loss

Project Structure

MVSC/
├── src/
│   ├── train.py                 # Training script
│   ├── eval_model.py            # Evaluation and inference
│   ├── baseline/
│   │   └── ssf2020.py           # Scale-Space Flow model
│   ├── utils/
│   │   ├── generate_text.py     # Video captioning (CogVLM2)
│   │   ├── generate_image.py    # Image generation (Stable Diffusion)
│   │   ├── generate_image2.py   # Image enhancement (Flux + ControlNet)
│   │   ├── generate_video.py    # Video generation (CogVideoX)
│   │   ├── losses.py            # CLIP loss, direction loss, AudioCLIP loss
│   │   ├── audioclip.py         # AudioCLIP model wrapper
│   │   ├── format_utils.py      # PIL <-> Tensor conversion
│   │   └── dataprepare.py       # Dataset preparation
│   └── metrics/
│       ├── calculate_lpips.py   # LPIPS perceptual metric
│       └── calculate_fvd.py     # Frechet Video Distance
├── CLIPVQA/                     # CLIP-based Video Quality Assessment
│   ├── main.py                  # VQA training/testing entry point
│   ├── models/                  # Spatial/temporal attention models
│   ├── configs/                 # VQA configuration files
│   └── README.md
├── common_metrics_on_video_quality/  # Video quality metrics (FVD, SSIM, LPIPS, PSNR)
│   ├── demo.py
│   └── README.md
├── configs/
│   ├── train/config.yaml        # Training configuration
│   └── test/config.yaml         # Testing configuration
├── tools/
│   ├── train.sh                 # Training launch script
│   └── test.sh                  # Testing launch script
└── requirements.txt

Requirements

• Python >= 3.10
• CUDA-capable GPU (recommended: 24GB+ VRAM for generative models)

Dependencies

torch >= 2.5.0
torchvision >= 0.20.0
diffusers >= 0.32.1
transformers >= 4.46.2
accelerate >= 1.1.1
compressai == 1.2.6
numpy == 1.26.0
SwissArmyTransformer >= 0.4.12
moviepy >= 2.0.0
scikit-video >= 1.1.11
imageio >= 2.35.1
imageio-ffmpeg >= 0.5.1
wav2clip
lpips
pytorch-msssim
decord

Installation

git clone <repo-url>
cd MVSC
pip install -r requirements.txt

Pre-trained Models

The framework uses the following Foundation Models (downloaded automatically or specify local paths):

Datasets

• Training: Vimeo Septuplet — 7-frame video sequences
• Testing: UVG — raw YUV420 video sequences

Usage

Training

Edit configs/train/config.yaml:

model: ssf2020
dataset: /path/to/vimeo_septuplet
epochs: 100
learning_rate: 0.0001
lmbda: 0.01
batch_size: 16
patch_size: 256
cuda: True

Run training:

bash tools/train.sh
# or
CUDA_VISIBLE_DEVICES=0 python src/train.py \
    -d /path/to/vimeo_septuplet \
    -m ssf2020 \
    --lambda_mse 0.01 \
    --lambda_lpips 0.01 \
    --checkpoint /path/to/checkpoint.pth.tar  # optional, resume training

Evaluation

Using pre-trained CompressAI models:

CUDA_VISIBLE_DEVICES=0 python src/eval_model.py pretrained \
    /path/to/UVG/yuv \
    /path/to/output \
    -a ssf2020 \
    -q 1,2,3,4,5,6,7,8 \
    --v2v_model /path/to/CogVideoX-5B \
    --controlnet_path /path/to/Flux.1-dev-Controlnet-Upscaler \
    --basenet_path /path/to/FLUX.1-dev

Using a custom checkpoint:

CUDA_VISIBLE_DEVICES=0 python src/eval_model.py checkpoint \
    /path/to/UVG/yuv \
    /path/to/output \
    -a ssf2020 \
    -p /path/to/checkpoint.pth.tar

Evaluation Metrics

The evaluation outputs per-sequence JSON files containing:

Loss Function

The training objective is a Rate-Distortion Loss with multi-modal guidance:

L = λ_mse * MSE(x_hat, x) + BPP + λ_lpips * LPIPS(x_hat, x)
    + λ_audio * CLIP_loss(x_hat, audio_embed)
    + λ_direction * Direction_loss(x_hat, audio_embed)

• MSE: Pixel-level reconstruction fidelity
• BPP: Bitrate cost from entropy model
• LPIPS: Perceptual quality
• CLIP loss: Audio-visual semantic alignment
• Direction loss: Temporal consistency between consecutive frames guided by audio

Sub-modules

CLIPVQA

A CLIP-based Video Quality Assessment model that uses semantic descriptors ("Excellent", "Good", "Poor") to predict video quality scores via spatial and temporal attention. See CLIPVQA/README.md.

Common Metrics on Video Quality

Evaluation toolkit for video generation/compression quality with support for FVD (PyTorch & TensorFlow), SSIM, LPIPS, and PSNR. See common_metrics_on_video_quality/README.md.

License

Based on CompressAI — see license headers in source files for details.