SphereForge

Transform 360° camera footage into high-quality 3D Gaussian Splat scenes.

SphereForge is an 8-stage pipeline that takes equirectangular (360°) video or images and produces optimized 3D Gaussian Splat files ready for viewing and streaming. It integrates the latest research in panoramic depth estimation, distortion-aware optimization, and multi-view hole filling.

Pipeline Overview

Video/ERP Images
       │
  ┌────▼────┐
  │ Stage 1 │  Frame Selection — extract sharpest frames, filter by luminance
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 2 │  Cubemap Prep — ERP→cubemap, YOLO masking, COLMAP dataset
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 3 │  SfM — COLMAP feature extraction, matching, bundle adjustment
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 4 │  Depth Estimation — DAP (ERP-native) or RPG360 depth alignment
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 5 │  Gaussian Seeding — project to 3D, fuse, prune, write PLY
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 6 │  Optimization — ErpGS loss, 360-GeoGS reg, ImprovedGS+ densify
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 7 │  Occlusion Recovery — ShareGS fill + SD/EscherNet inpainting
  └────┬────┘
       │
  ┌────▼────┐
  │ Stage 8 │  Export — final pruning, PLY/SOG/SPZ/HTML output
  └────┬────┘
       │
  Final 3DGS Scene

Key Features

• ERP-native depth: Uses DAP (Depth Any Panoramas) for distortion-aware equirectangular depth estimation — no cubemap conversion needed for depth
• Distortion-aware optimization: ErpGS cos(latitude) weighted loss eliminates polar artifacts in 360° scenes
• 360-GeoGS regularization: Ray-ellipsoid intersection depth + D-Normal loss for geometric consistency
• ImprovedGS+ densification: Edge-Aware Score triggers + Long-Axis Split + Recovery-Aware Pruning
• Two-phase occlusion recovery: ShareGS (Gaussian homogenization) fills small gaps, then Stable Diffusion inpainting fills remaining holes with LPIPS hallucination filtering
• Multiple export formats: Standard 3DGS PLY, SOG (structured streaming), SPZ (Morton-ordered + delta-encoded), and self-contained HTML viewer

Installation

# Core pipeline
pip install -e .

# With depth estimation models
pip install -e ".[depth]"

# With gsplat rasterizer (recommended for training)
pip install -e ".[rasterizer]"

# With inpainting support
pip install -e ".[occlusion]"

# With YOLO dynamic object masking
pip install -e ".[yolo]"

# Everything
pip install -e ".[all]"

# Development
pip install -e ".[dev]"

GPU Dependencies

For full GPU-accelerated training and depth estimation:

# CUDA 12.8+ with PyTorch 2.6+
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# gsplat rasterizer (required for Stage 6 training)
pip install gsplat>=1.0

# One-line install for GPU stack
pip install 'sphereforge[all]'

External Dependencies

• COLMAP: Required for Stage 3 (SfM). Install from colmap.github.io
• FFmpeg: Required for Stage 1 (video ingestion). Usually pre-installed or apt install ffmpeg
• CUDA GPU: Required for training (Stage 6) and depth estimation (Stage 4)

Quick Start

# Process a 360° video end-to-end
sphereforge process input_video.mp4 --output ./output

# Process a directory of ERP images
sphereforge process ./erp_frames/ --output ./output

# Export to specific formats
sphereforge export ./output/optimized/ --format ply sog html

# Check pipeline progress
sphereforge status

Configuration

Each stage has sensible defaults, but all parameters can be overridden via YAML config:

sphereforge process input.mp4 --config my_config.yaml

Example config:

stage01:
  input_fps: 5
  chunk_size: 30

stage04:
  depth_model: dap       # DAP for ERP-native depth
  max_depth: 10.0

stage06:
  iterations: 30000
  erp_distortion_weights: true
  densification: igs_plus

stage07:
  gsdiff_backend: sd      # Stable Diffusion inpainting
  gsdiff_prompt: "clean indoor scene, high quality"

stage08:
  export_format:
    - ply
    - html

Project Structure

src/sphereforge/
├── cli.py                          # Click CLI entry point
├── config.py                       # Pydantic config models for all stages
├── logging_utils.py                # Progress tracking
├── common/
│   ├── io.py                       # Image/depth/PLY/COLMAP I/O
│   ├── colmap_helpers.py           # COLMAP text/binary parsers
│   ├── depth_utils.py              # Depth alignment, NCC, fusion
│   ├── metrics.py                  # PSNR, SSIM, LPIPS
│   └── model_cache.py             # Model weight download/cache
├── models/
│   ├── dap_model.py               # DAP (Depth Any Panoramas)
│   ├── depth_anything_v2.py       # Depth Anything V2
│   └── metric3d.py                # Metric3D v2
└── stages/
    ├── stage01_frames/            # Video ingestion & frame selection
    ├── stage02_cubemap/           # ERP→cubemap + COLMAP prep
    ├── stage03_sfm/               # COLMAP SfM wrappers
    ├── stage04_depth/             # Dense depth estimation
    ├── stage05_seeding/           # Initial Gaussian seeding
    ├── stage06_optimization/      # Multi-view Gaussian optimization
    ├── stage07_occlusion/         # Occlusion recovery & refinement
    └── stage08_export/            # Post-processing & export

Research Credits

SphereForge integrates techniques from the following papers:

License

GNU General Public License v3.0 or later (GPLv3+). See LICENSE for details.

This project incorporates code and model weights under various licenses:

• MIT/Apache-2.0/BSD-2: Core utilities and some model code — no restrictions
• NC-3DGS: ImprovedGS+, FastGS, CDC-GS inherit the original 3DGS non-commercial license
• CreativeML RAIL-M: EscherNet and Stable Diffusion weights — non-commercial use
• GPLv3: LichtFeld-Studio SOG/SPZ export — called as separate process

For commercial use, contact the respective rights holders or replace NC components with alternatives (e.g., gsplat for the INRIA 3DGS rasterizer).

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run only fast tests (skip GPU/slow)
pytest tests/ -v -m "not slow and not gpu"

# Lint
ruff check src/
black --check src/