Introduction

This repository implements a three-stage video object removal workflow.

• Part 1 is a classical baseline built around YOLOv8-Seg, sparse optical flow, mask cleanup, temporal background borrowing, and OpenCV inpainting.
• Part 2 is a higher-quality pipeline built around SAM2 mask propagation and ProPainter video inpainting.
• Part 3 refines Part 2 masks with SAM3 and supports generative fallback for persistent occlusions before temporal inpainting.

The previous long-form README has been preserved as README_Outdated.md.

Current Status

• Part 1 is implemented and validated.
• Part 2 is implemented and validated.
• Part 3 is implemented and validated for the main tracked sequences in this repo.
• DAVIS sequences parkour and dance-twirl are integrated into the same CLI workflow as bmx-trees and tennis.

Environment

Create conda environment named project3

conda activate project3
pip install -r requirements.txt

Expected model locations:

• models/yolo_v8_seg/yolov8x-seg.pt
• models/sam2/checkpoints/
• models/ProPainter/weights/
• models/sam3/checkpoints/sam3.pt
• models/controlnet/

Dataset Keys

Supported sequence keys are defined in configs/common.yaml.

• bmx-trees
• tennis
• parkour
• dance-twirl
• direct DAVIS sequence names resolved from data/raw/davis/DAVIS/JPEGImages/480p/

Supported aliases:

• bmx_trees -> bmx-trees
• dance_twirl -> dance-twirl

Part 1

Model

Part 1 combines:

• YOLOv8-Seg for candidate instance masks
• sparse Lucas-Kanade optical flow for dynamic/static filtering
• temporal propagation for background recovery
• OpenCV inpainting as fallback filling

Structure

• Entrypoint: src/part1_baseline/pipeline_part1.py
• Detection: src/part1_baseline/mask_extraction_yolo.py
• Motion filtering: src/part1_baseline/dynamic_judgment.py
• Inpainting: src/part1_baseline/inpaint_traditional.py
• Shared utilities: src/common/

Configuration

• Shared dataset, naming, and output settings: configs/common.yaml
• Part 1 phase config: configs/part1_baseline.yaml
• Default target classes: person, bicycle
• Key tunable blocks: dynamic_filter, postprocess, inpainting, visualization

Outputs

• Masks: results/masks/part1/{sequence}/
• Video: results/videos/part1/{sequence}_part1.mp4
• Visualizations: results/visualizations/part1/{sequence}/
• Metrics: results/metrics/part1/{sequence}/

Run Commands

Run the default Part 1 sequence:

python src/part1_baseline/pipeline_part1.py \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Run a specific sequence:

python src/part1_baseline/pipeline_part1.py \
  --sequence parkour \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Quick smoke test:

python src/part1_baseline/pipeline_part1.py \
  --sequence bmx-trees \
  --max-frames 3 \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Wrapper script:

bash scripts/run_part1.sh --sequence bmx-trees --max-frames 3

Evaluation

Standard evaluation:

python scripts/evaluate_metrics.py \
  --phase-config configs/part1_baseline.yaml \
  --sequence parkour

Optional short evaluation:

python scripts/evaluate_metrics.py \
  --phase-config configs/part1_baseline.yaml \
  --sequence parkour \
  --max-frames 3

If --reference-frames-dir is not provided, PSNR and SSIM are computed in background_preservation mode.

Visualization

Each sequence visualization folder contains:

• motion_scores/
• mask_overlays/
• comparisons/

These assets are used for debugging and report figures.

Part 2

Model

Part 2 combines:

• SAM2 for prompt-based sequence mask propagation
• lightweight mask postprocessing before export and inpainting
• ProPainter for temporally coherent video restoration

Structure

• Entrypoint: src/part2_sota/pipeline_part2.py
• Prompt handling: src/part2_sota/prompt_utils.py
• SAM2 mask generation: src/part2_sota/mask_sam2.py
• Inpainting: src/part2_sota/inpaint_pro_painter.py

Configuration

• Shared config: configs/common.yaml
• Part 2 phase config: configs/part2_sota.yaml
• Sequence-specific prompts live under the prompts: block
• DAVIS prompts for parkour and dance-twirl are already configured

Outputs

• Object masks: results/masks/part2/{sequence}/objects/
• Combined masks: results/masks/part2/{sequence}/combined/
• Video: results/videos/part2/{sequence}_inpainted.mp4
• Visualizations: results/visualizations/part2/{sequence}/
• Metrics: results/metrics/part2/{sequence}/

Run Commands

Run the default Part 2 sequence:

python src/part2_sota/pipeline_part2.py \
  --common-config configs/common.yaml \
  --phase-config configs/part2_sota.yaml

Run a specific sequence:

python src/part2_sota/pipeline_part2.py \
  --sequence dance-twirl \
  --common-config configs/common.yaml \
  --phase-config configs/part2_sota.yaml

Wrapper script:

bash scripts/run_part2.sh --sequence parkour

Evaluation

python scripts/evaluate_metrics.py \
  --phase-config configs/part2_sota.yaml \
  --sequence dance-twirl

The evaluator automatically resolves the standard Part 2 video name and uses combined/ masks for overall foreground evaluation.

Visualization

Each sequence visualization folder contains:

• motion_scores/ for object overlays
• mask_overlays/
• comparisons/

Part 3

Models

Part 3 combines:

• SAM3 for mask refinement from Part 2 coarse masks
• ProPainter for final temporal inpainting
• ControlNet / Stable Diffusion fallback for persistent occlusion cases

Structure

• Entrypoint: src/part3_exploration/pipeline_part3.py
• Refinement: src/part3_exploration/sam3_upgrade.py
• Generative fallback: src/part3_exploration/diffusion_controlnet.py
• Experimental loop: src/part3_exploration/dynamic_masking_loop.py

Configuration

• Shared config: configs/common.yaml
• Part 3 phase config: configs/part3_exploration.yaml
• Part 3 reuses Part 2 outputs as baseline masks
• Key tunable blocks: refinement, postprocess, visualization, export

Outputs

• Object masks: results/masks/part3/{sequence}/objects/
• Combined masks: results/masks/part3/{sequence}/combined/
• Optional keyframes: results/masks/part3/{sequence}/keyframes/
• Blended debug intermediates: results/masks/part3/{sequence}/sd_blend_frames/ and sd_blend_masks/
• Video: results/videos/part3/{sequence}_part3.mp4
• Visualizations: results/visualizations/part3/{sequence}/
• Metrics: results/metrics/part3/{sequence}/

If --max-frames N is used, Part 3 writes to isolated names such as results/masks/part3/{sequence}_maxframesN/ and results/videos/part3/{sequence}_maxframesN_part3.mp4.

Run Commands

Run the default Part 3 sequence:

python src/part3_exploration/pipeline_part3.py \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Run a specific sequence:

python src/part3_exploration/pipeline_part3.py \
  --sequence parkour \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Short smoke test without overwriting the full run:

python src/part3_exploration/pipeline_part3.py \
  --sequence parkour \
  --max-frames 2 \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Wrapper script:

bash scripts/run_part3.sh --sequence dance-twirl

Evaluation

python scripts/evaluate_metrics.py \
  --phase-config configs/part3_exploration.yaml \
  --sequence parkour

Part 3 uses the same evaluation interface as Part 1 and Part 2.

Visualization

Each sequence visualization folder contains:

• motion_scores/ for refined candidate overlays
• mask_overlays/
• comparisons/

DAVIS-only metric figures for parkour and dance-twirl can be generated with:

python src/common/figure_scripts/generate_metric_figures_davis.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures/davis
```# Video Object Removal and Inpainting

This repository implements a staged video object removal project. The currently implemented pipelines cover Part 1 as a classical baseline built around YOLOv8-Seg, sparse optical flow, temporal background propagation, and traditional image inpainting, plus Part 2 as a higher-quality SOTA pipeline built around SAM2 mask propagation and ProPainter video inpainting.

Part 1 is no longer just a mask extraction demo. It now runs a full baseline pipeline:

1. Detect candidate dynamic classes with YOLOv8-Seg.
2. Use Lucas-Kanade sparse optical flow to decide whether each detected instance is actually moving.
3. Postprocess masks with dilation, hole filling, connected-component cleanup, and short temporal smoothing.
4. Restore the video by borrowing clean background pixels from nearby frames first.
5. Fill any remaining holes with `cv2.inpaint` using Telea or Navier-Stokes.
6. Export masks, restored video, visualization artifacts, and evaluation metrics.

## Current Status

- Part 1 baseline: implemented and validated.
- Part 2 SOTA pipeline: implemented; requires SAM2 and ProPainter assets under `models/sam2` and `models/ProPainter`.
- Part 3 exploration pipeline: interface scaffold only.

## Environment

The commands below assume the Conda environment is named `project3`.

```bash
conda activate project3
pip install -r requirements.txt

Expected Part 1 model weight path:

models/yolo_v8_seg/yolov8x-seg.pt

If the weight file is missing, place the YOLOv8 segmentation checkpoint at that location or update configs/part1_baseline.yaml.

Expected Part 2 model roots:

models/sam2
models/ProPainter

The SAM2 repository should include checkpoints under models/sam2/checkpoints/. ProPainter weights are downloaded into models/ProPainter/weights/ on first use.

Figures

src/common/figure_scripts/ contains the figure-generation entrypoints and shared plotting modules used to convert metric CSVs into publication-style PNG figures.

Overview

• generate_metric_figures.py generates the main figure suite under results/visualizations/figures/
• generate_metric_figures_davis.py generates the DAVIS-only figure suite for parkour and dance-twirl under results/visualizations/figures/davis/
• generate_wild_comparison.py generates the Wild Video qualitative comparison figure
• metric_figure_data.py, metric_figure_plots.py, metric_figure_output.py, and metric_figure_constants.py provide the shared data loading, plotting, output, and naming logic

Prerequisites

Before generating figures, make sure metric CSVs already exist under results/metrics/{part1,part2,part3}/{sequence}/.

Typical workflow:

python scripts/evaluate_metrics.py --phase-config configs/part1_baseline.yaml --sequence parkour
python scripts/evaluate_metrics.py --phase-config configs/part2_sota.yaml --sequence parkour
python scripts/evaluate_metrics.py --phase-config configs/part3_exploration.yaml --sequence parkour

Repeat that for every sequence you want included in the plots.

Main Figure Suite

Generate the overall figure set:

python src/common/figure_scripts/generate_metric_figures.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures

This writes summary CSVs, a generated README manifest, distribution plots, frame-wise trend plots, paired-delta figures, heatmaps, and summary tables into results/visualizations/figures/.

DAVIS Figure Suite

Generate the DAVIS-only figure set:

python src/common/figure_scripts/generate_metric_figures_davis.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures/davis

This writes the same style of metric visualizations, but restricted to parkour and dance-twirl.

Wild Video Comparison

Generate the Wild Video qualitative comparison figure:

python src/common/figure_scripts/generate_wild_comparison.py

Notes

• All figure scripts write PNG outputs.
• The plotting style is centralized in metric_figure_plots.py.
• The scripts resolve output paths from the repository root, so running them from the project root is the safest option.

Representative Results

The figures below are embedded directly from the existing outputs under results/visualizations/ and summarize the final qualitative and quantitative behavior of the repository.

Qualitative Comparison on Wild Video Frames

This figure highlights representative early, middle, and late frames from the unconstrained wild_video_frames sequence, showing the progression from the interpretable Part 1 baseline to the stronger Part 2 and Part 3 restorations.

Stage-Level Restoration Summary

This summary condenses stage-level IoU behavior, sequence-wise IoU, and the IoU-PSNR trade-off across the evaluated sequences.

Cross-Sequence Metric Overview

This overview compares IoU, PSNR, and SSIM across BMX-Trees, Tennis, Parkour, Dance-Twirl, and the Wild Video case.