Introduction

This repository implements a three-stage video object removal workflow.

• Part 1 is a classical baseline built around YOLOv8-Seg, sparse optical flow, mask cleanup, temporal background borrowing, and OpenCV inpainting.
• Part 2 is a higher-quality pipeline built around SAM2 mask propagation and ProPainter video inpainting.
• Part 3 refines Part 2 masks with SAM3 and supports generative fallback for persistent occlusions before temporal inpainting.

The previous long-form README has been preserved as README_Outdated.md.

Current Status

• Part 1 is implemented and validated.
• Part 2 is implemented and validated.
• Part 3 is implemented and validated for the main tracked sequences in this repo.
• DAVIS sequences parkour and dance-twirl are integrated into the same CLI workflow as bmx-trees and tennis.
• DAVIS-specific metric figures are generated by src/common/figure_scripts/generate_metric_figures_davis.py into results/visualizations/figures/davis/.
• Short Part 3 smoke tests using --max-frames now write to isolated output names such as {sequence}_maxframes2 and no longer overwrite full-run results.

Environment

Commands below assume the Conda environment is named project3 and are run from the repository root.

conda activate project3
pip install -r requirements.txt

Expected model locations:

• models/yolo_v8_seg/yolov8x-seg.pt
• models/sam2/checkpoints/
• models/ProPainter/weights/
• models/sam3/checkpoints/sam3.pt
• models/controlnet/

Dataset Keys

Supported sequence keys are defined in configs/common.yaml.

• bmx-trees
• tennis
• parkour
• dance-twirl
• direct DAVIS sequence names resolved from data/raw/davis/DAVIS/JPEGImages/480p/

Supported aliases:

• bmx_trees -> bmx-trees
• dance_twirl -> dance-twirl

Part 1

Model

Part 1 combines:

• YOLOv8-Seg for candidate instance masks
• sparse Lucas-Kanade optical flow for dynamic/static filtering
• temporal propagation for background recovery
• OpenCV inpainting as fallback filling

Structure

• Entrypoint: src/part1_baseline/pipeline_part1.py
• Detection: src/part1_baseline/mask_extraction_yolo.py
• Motion filtering: src/part1_baseline/dynamic_judgment.py
• Inpainting: src/part1_baseline/inpaint_traditional.py
• Shared utilities: src/common/

Configuration

• Shared dataset, naming, and output settings: configs/common.yaml
• Part 1 phase config: configs/part1_baseline.yaml
• Default target classes: person, bicycle
• Key tunable blocks: dynamic_filter, postprocess, inpainting, visualization

Outputs

• Masks: results/masks/part1/{sequence}/
• Video: results/videos/part1/{sequence}_part1.mp4
• Visualizations: results/visualizations/part1/{sequence}/
• Metrics: results/metrics/part1/{sequence}/

Run Commands

Run the default Part 1 sequence:

python src/part1_baseline/pipeline_part1.py \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Run a specific sequence:

python src/part1_baseline/pipeline_part1.py \
  --sequence parkour \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Quick smoke test:

python src/part1_baseline/pipeline_part1.py \
  --sequence bmx-trees \
  --max-frames 3 \
  --common-config configs/common.yaml \
  --phase-config configs/part1_baseline.yaml

Wrapper script:

bash scripts/run_part1.sh

Evaluation

Standard evaluation:

python scripts/evaluate_metrics.py \
  --phase-config configs/part1_baseline.yaml \
  --sequence parkour

Optional short evaluation:

python scripts/evaluate_metrics.py \
  --phase-config configs/part1_baseline.yaml \
  --sequence parkour \
  --max-frames 3

If --reference-frames-dir is not provided, PSNR and SSIM are computed in background_preservation mode.

Visualization

Each sequence visualization folder contains:

• motion_scores/
• mask_overlays/
• comparisons/

These assets are used for debugging and report figures.

Part 2

Model

Part 2 combines:

• SAM2 for prompt-based sequence mask propagation
• lightweight mask postprocessing before export and inpainting
• ProPainter for temporally coherent video restoration

Structure

• Entrypoint: src/part2_sota/pipeline_part2.py
• Prompt handling: src/part2_sota/prompt_utils.py
• SAM2 mask generation: src/part2_sota/mask_sam2.py
• Inpainting: src/part2_sota/inpaint_pro_painter.py

Configuration

• Shared config: configs/common.yaml
• Part 2 phase config: configs/part2_sota.yaml
• Sequence-specific prompts live under the prompts: block
• DAVIS prompts for parkour and dance-twirl are already configured

Outputs

• Object masks: results/masks/part2/{sequence}/objects/
• Combined masks: results/masks/part2/{sequence}/combined/
• Video: results/videos/part2/{sequence}_inpainted.mp4
• Visualizations: results/visualizations/part2/{sequence}/
• Metrics: results/metrics/part2/{sequence}/

Run Commands

Run the default Part 2 sequence:

python src/part2_sota/pipeline_part2.py \
  --common-config configs/common.yaml \
  --phase-config configs/part2_sota.yaml

Run a specific sequence:

python src/part2_sota/pipeline_part2.py \
  --sequence dance-twirl \
  --common-config configs/common.yaml \
  --phase-config configs/part2_sota.yaml

Wrapper script:

bash scripts/run_part2.sh

Evaluation

python scripts/evaluate_metrics.py \
  --phase-config configs/part2_sota.yaml \
  --sequence dance-twirl

The evaluator automatically resolves the standard Part 2 video name and uses combined/ masks for overall foreground evaluation.

Visualization

Each sequence visualization folder contains:

• motion_scores/ for object overlays
• mask_overlays/
• comparisons/

Part 3

Models

Part 3 combines:

• SAM3 for mask refinement from Part 2 coarse masks
• ProPainter for final temporal inpainting
• ControlNet / Stable Diffusion fallback for persistent occlusion cases

Structure

• Entrypoint: src/part3_exploration/pipeline_part3.py
• Refinement: src/part3_exploration/sam3_upgrade.py
• Generative fallback: src/part3_exploration/diffusion_controlnet.py
• Experimental loop: src/part3_exploration/dynamic_masking_loop.py

Configuration

• Shared config: configs/common.yaml
• Part 3 phase config: configs/part3_exploration.yaml
• Part 3 reuses Part 2 outputs as baseline masks
• Key tunable blocks: refinement, postprocess, visualization, export

Outputs

• Object masks: results/masks/part3/{sequence}/objects/
• Combined masks: results/masks/part3/{sequence}/combined/
• Optional keyframes: results/masks/part3/{sequence}/keyframes/
• Blended debug intermediates: results/masks/part3/{sequence}/sd_blend_frames/ and sd_blend_masks/
• Video: results/videos/part3/{sequence}_part3.mp4
• Visualizations: results/visualizations/part3/{sequence}/
• Metrics: results/metrics/part3/{sequence}/

If --max-frames N is used, Part 3 writes to isolated names such as results/masks/part3/{sequence}_maxframesN/ and results/videos/part3/{sequence}_maxframesN_part3.mp4.

Run Commands

Run the default Part 3 sequence:

python src/part3_exploration/pipeline_part3.py \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Run a specific sequence:

python src/part3_exploration/pipeline_part3.py \
  --sequence parkour \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Short smoke test without overwriting the full run:

python src/part3_exploration/pipeline_part3.py \
  --sequence parkour \
  --max-frames 2 \
  --common-config configs/common.yaml \
  --phase-config configs/part3_exploration.yaml

Wrapper script:

bash scripts/run_part3.sh

Evaluation

python scripts/evaluate_metrics.py \
  --phase-config configs/part3_exploration.yaml \
  --sequence parkour

Part 3 uses the same evaluation interface as Part 1 and Part 2.

Visualization

Each sequence visualization folder contains:

• motion_scores/ for refined candidate overlays
• mask_overlays/
• comparisons/

DAVIS-only metric figures for parkour and dance-twirl can be generated with:

python src/common/figure_scripts/generate_metric_figures_davis.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures/davis
```# Video Object Removal and Inpainting

This repository implements a staged video object removal project. The currently implemented pipelines cover Part 1 as a classical baseline built around YOLOv8-Seg, sparse optical flow, temporal background propagation, and traditional image inpainting, plus Part 2 as a higher-quality SOTA pipeline built around SAM2 mask propagation and ProPainter video inpainting.

Part 1 is no longer just a mask extraction demo. It now runs a full baseline pipeline:

1. Detect candidate dynamic classes with YOLOv8-Seg.
2. Use Lucas-Kanade sparse optical flow to decide whether each detected instance is actually moving.
3. Postprocess masks with dilation, hole filling, connected-component cleanup, and short temporal smoothing.
4. Restore the video by borrowing clean background pixels from nearby frames first.
5. Fill any remaining holes with `cv2.inpaint` using Telea or Navier-Stokes.
6. Export masks, restored video, visualization artifacts, and evaluation metrics.

## Current Status

- Part 1 baseline: implemented and validated.
- Part 2 SOTA pipeline: implemented; requires SAM2 and ProPainter assets under `models/sam2` and `models/ProPainter`.
- Part 3 exploration pipeline: interface scaffold only.

## Environment

The commands below assume the Conda environment is named `project3`.

```bash
conda activate project3
pip install -r requirements.txt

Expected Part 1 model weight path:

models/yolo_v8_seg/yolov8x-seg.pt

If the weight file is missing, place the YOLOv8 segmentation checkpoint at that location or update configs/part1_baseline.yaml.

Expected Part 2 model roots:

models/sam2
models/ProPainter

The SAM2 repository should include checkpoints under models/sam2/checkpoints/. ProPainter weights are downloaded into models/ProPainter/weights/ on first use.

Evaluation

Wrapper Script

Run all evaluation commands in sequence with:

bash run_all_metrics.sh

Figures

src/common/figure_scripts/ contains the figure-generation entrypoints and shared plotting modules used to convert metric CSVs into publication-style PNG figures.

Overview

• generate_metric_figures.py generates the main figure suite under results/visualizations/figures/
• generate_metric_figures_davis.py generates the DAVIS-only figure suite for parkour and dance-twirl under results/visualizations/figures/davis/
• generate_wild_comparison.py generates the Wild Video qualitative comparison figure
• metric_figure_data.py, metric_figure_plots.py, metric_figure_output.py, and metric_figure_constants.py provide the shared data loading, plotting, output, and naming logic

Wrapper Script

Run all three figure-generation commands in sequence with:

bash run_all_figure.sh

Prerequisites

Before generating figures, make sure metric CSVs already exist under results/metrics/{part1,part2,part3}/{sequence}/.

Typical workflow:

python scripts/evaluate_metrics.py --phase-config configs/part1_baseline.yaml --sequence parkour
python scripts/evaluate_metrics.py --phase-config configs/part2_sota.yaml --sequence parkour
python scripts/evaluate_metrics.py --phase-config configs/part3_exploration.yaml --sequence parkour

Repeat that for every sequence you want included in the plots.

Main Figure Suite

Generate the overall figure set:

python src/common/figure_scripts/generate_metric_figures.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures

This writes summary CSVs, a generated README manifest, distribution plots, frame-wise trend plots, paired-delta figures, heatmaps, and summary tables into results/visualizations/figures/.

DAVIS Figure Suite

Generate the DAVIS-only figure set:

python src/common/figure_scripts/generate_metric_figures_davis.py \
  --metrics-root results/metrics \
  --output-dir results/visualizations/figures/davis

This writes the same style of metric visualizations, but restricted to parkour and dance-twirl.

Wild Video Comparison

Generate the Wild Video qualitative comparison figure:

python src/common/figure_scripts/generate_wild_comparison.py

Notes

• All figure scripts write PNG outputs.
• The plotting style is centralized in metric_figure_plots.py.
• The scripts resolve output paths from the repository root, so running them from the project root is the safest option.