Hybrid Fused Depth Pipeline

Underwater image enhancement with metric-scale fused depth maps and colored point cloud export.

Fuses two depth estimators (metric model fine-tuned on underwater stereo data (FLSea-Stereo) + relative model (Depth Anything v2 Small)), into a single depth map w/ metric scale.

Setup

Download the entire checkpoints folder from google drive and insert it in the root directory of this repo.

python3 -m venv .venv
.venv/bin/python -m pip install -r pipeline_2d_metric/requirements.txt opencv-python-headless

---

Quick start (bash script)

./run_pipeline.sh runs the full hybrid pipeline on a single image and exports a point cloud.

./run_pipeline.sh <image_filename> [--fx <val>] [--fy <val>] [--device <val>]

The image must be in custom_images/. Example:

./run_pipeline.sh Fish_laser.JPG
./run_pipeline.sh BlueParrot.JPG --fx 2917.34 --fy 2917.34 --device cpu

Default flag values are 2917.34, 2917.34, and mps, respectively.

Outputs

(written to outputs/<stem>/):

pipeline_2d_metric.infer

.venv/bin/python -m pipeline_2d_metric.infer \
    --input  <path>      \   # image file or directory
    --output <path>      \   # output file (single) or directory (batch)
    --checkpoint <path>      # enhancement U-Net checkpoint
    [options]

All Flags:

• --input <path> — image file or directory (processes all images inside).
• --output <path> — output file (single) or directory (batch).
• --checkpoint <path> — enhancement U-Net .pth (e.g. checkpoints/pipeline_2d/best.pth).
• --depth-model <path> — metric DAv2 .pth (default: checkpoints/depth_anything_v2_metric_hypersim_vitb.pth; use checkpoints/dav2_metric_flsea_full/epoch_007.pth for the FLSea-finetuned backbone).
• --depth-encoder vits|vitb|vitl|vitg — encoder size; must match the .pth (default vitb).
• --depth-max — max depth in metres; must match checkpoint training (default 20.0).
• --fuse-relative — enable hybrid fusion.
• --save-metric — also write <stem>_depth_meters.npy.
• --no-save-depth — skip writing colormap visualization.
• --depth-colormap turbo|gray — colormap for depth visualization.
• --resize-w / --resize-h — input resolution before inference.
• --num-stages — iterative pipeline stages (default 3).
• --base-channels — U-Net channel count; must match checkpoint (default 32).
• --device mps|cuda|cpu — PyTorch device (default mps).

Examples:

# Hybrid fused inference with metric .npy output
.venv/bin/python -m pipeline_2d_metric.infer \
    --input custom_images/Fish_laser.JPG \
    --output outputs/Fish_laser_fused.png \
    --checkpoint checkpoints/pipeline_2d/best.pth \
    --depth-model checkpoints/dav2_metric_flsea_full/epoch_007.pth \
    --fuse-relative \
    --save-metric \
    --device mps

# Metric-only inference (no relative fusion)
.venv/bin/python -m pipeline_2d_metric.infer \
    --input custom_images/Fish_laser.JPG \
    --output outputs/Fish_laser_metric.png \
    --checkpoint checkpoints/pipeline_2d/best.pth \
    --save-metric

# Batch mode (every image in the folder)
.venv/bin/python -m pipeline_2d_metric.infer \
    --input custom_images/ \
    --output outputs/batch/ \
    --checkpoint checkpoints/pipeline_2d/best.pth \
    --fuse-relative \
    --save-metric

---

pipeline_2d_metric.depth_to_pointcloud

Converts a metric .npy depth map and its paired RGB into a binary PLY point cloud. Does not require open3d.

.venv/bin/python -m pipeline_2d_metric.depth_to_pointcloud \
    --rgb <path> --depth-npy <path> --out <path> [options]

Flags:

• --rgb <path> — RGB image to colorize cloud; should match resolution of the .npy.
• --depth-npy <path> — float32 .npy from --save-metric.
• --out <path> — output .ply file.
• --fx / --fy — pinhole focal lengths in pixels (default 2917.34).
• --min-depth / --max-depth — clip points outside range (meters) (default 0.05–20.0).

Example (full pipeline, manual steps):

# Step 1: inference → enhanced RGB + metric .npy
.venv/bin/python -m pipeline_2d_metric.infer \
    --input custom_images/Fish_laser.JPG \
    --output outputs/Fish_laser_fused.png \
    --checkpoint checkpoints/pipeline_2d/best.pth \
    --depth-model checkpoints/dav2_metric_flsea_full/epoch_007.pth \
    --fuse-relative --save-metric --device mps

# Step 2: project to PLY
.venv/bin/python -m pipeline_2d_metric.depth_to_pointcloud \
    --rgb outputs/Fish_laser_fused.png \
    --depth-npy outputs/Fish_laser_fused_depth_meters.npy \
    --out outputs/Fish_laser_fused_points.ply