Active Vision: Semantic View Selection for Novel View Synthesis

Goal: Select a small, informative subset of views (under a fixed budget) for training NeRF-style models. Key idea: Extend Farthest View Sampling (FVS) into semantic space using image embeddings from VLMs (e.g., CLIP, DINOv2). Baselines: Geometric FVS (from NeRF Director) and Random Sampling (RS).

---

Features

• Image embeddings (CLIP & DINOv2 families)
• VLM-based frame selection (semantic FVS with cosine distance)
• Camera-trajectory visualization (quick plots of camera centers/paths)
• Analysis tools: PSNR/SSIM/LPIPS curves, Sampling Efficiency Score (SES), statistical tests (Wilcoxon, t-test, Sign test) with IID/normality checks
• Nerfstudio integrations for budgeted training/evaluation

---

Repository layout

active-vision/
├─ embedding/                 # embedding extraction & utilities
├─ tools/                     # plotting, SES, stats, camera visualization
├─ nerfstudio_mods/           # small patches/wrappers for Nerfstudio usage
├─ active_vision_select.py    # main view-selection entry point (FVS, RS, VLM-FVS)
├─ run.sh                     
├─ requirements.txt           # Python requirements
└─ README.md

---

Methods

• FVS (Farthest View Sampling): farthest-first in pose space (baseline)
• RS (Random Sampling): uniform random baseline
• Semantic FVS (ours): farthest-first in VLM embedding space (cosine distance)

Embedding models used:

• CLIP ViT-B/32, CLIP ViT-L/14, CLIP ViT-L/14@336px
• DINOv2 ViT-B/14, DINOv2 ViT-L/14 Full-Resolution

---

What we observed (very short)

• Object-centric scenes: geometric FVS in general is strongest.
• Irregular scenes: VLM-FVS is competitive but inconsistent; RS can win in some cases.
• SES and statistical tests support these trends.

For full plots and tables, see the paper and supplementary materials.