FaceGS: Canonical-UV Editing for 3D Gaussian Heads

FaceGS edits a trained Nerfstudio Splatfacto / 3D Gaussian head model by rendering a small set of anchor views, editing those views with a selected 2D backend, back-projecting the masked edit signal into a canonical FLAME/VHAP UV atlas, and writing the resulting appearance delta back into each Gaussian's spherical-harmonic color parameters.

The core design is deliberately simple:

• an EditSpec JSON defines the region, prompt, backend, seeds, and anchor policy;
• canonical UV masks define where edits are allowed;
• each UV texel has one authoritative writer in the production path;
• the final output is an edited_appearance.pt sidecar that can be rendered, measured, or exported to standard INRIA/3DGS PLY.

There is no natural-language planner, hidden prompt compiler, or uncontrolled RGB averaging across views.

Entrypoints

The main entrypoints are:

• scripts/run_edit_spec.py: run one hand-written EditSpec locally or inside an allocated GPU job.
• scripts/slurm_run_edit_spec_once.sh: the only public Slurm wrapper for edit jobs. Configure it with FACEGS_SPEC_PATH, FACEGS_RUN_LABEL, and optional environment overrides.
• scripts/slurm_train_nersemble_splatfacto.sh: Slurm wrapper for preparing one static NeRSemble face scene and training a Nerfstudio splatfacto model.
• scripts/run_paper_analytics.py: refresh missing eval_renders/ folders and then call the metric script.
• scripts/submit_paper_analytics_full.sh: the only public Slurm wrapper for full paper analytics.
• scripts/compute_paper_metrics.py: compute paper-facing metrics from existing eval_renders/ folders.
• scripts/render_eval_cameras.py: render before/after/mask views for a saved edit sidecar.
• scripts/export_splat_to_inria_ply.py: export original or edited splats to a standard binary little-endian 3DGS PLY.

All old experiment-specific Slurm launchers were removed. Use the generic edit wrapper for CN-IP2P, FLUX Kontext, UV-canvas paint, and ablations by changing the spec and environment variables.

Quick Start

Set up the main environment and paths:

cd /scratch/gpfs/JORDANAT/caleb/faceGS
source scripts/_facegs_env.sh

export TRAIN_TAG=p017_exp2eyes_t000000_cam08_uvmesh_vhap30k_fix8_facegs_only
export CANONICAL_DIR="artifacts/nersemble_${TRAIN_TAG}_canonical"
export CONFIG_PATH="outputs/nersemble/splatfacto/${TRAIN_TAG}/config.yml"

Train a NeRSemble face Splatfacto model:

export NERSEMBLE_ROOT=/path/to/local/nersemble_download
export FACEGS_NERSEMBLE_PARTICIPANT=17
export FACEGS_NERSEMBLE_SEQUENCE=EXP-2-eyes
export FACEGS_NERSEMBLE_TIMESTEP=0
export FACEGS_NERSEMBLE_CAMERAS=222200037,222200038,222200039
export FACEGS_TRAIN_TAG=p017_exp2eyes_t000000_cam08_uvmesh_vhap30k_fix8_facegs_only
sbatch scripts/slurm_train_nersemble_splatfacto.sh

Create a local edit spec. The specs/ folder is intentionally ignored so paper-specific prompts and dataset paths stay out of git.

mkdir -p specs/local
cat > specs/local/clown_cn_ip2p.json <<'JSON'
{
  "editor_mode": "anchor_view_controlnet_ip2p",
  "region_name": "face_inclusive",
  "prompt": "Turn him into a clown: white face paint covering forehead, cheeks, nose, and chin; red clown nose; red lips; dark eye makeup. Keep hair, hoodie, background, identity, and face shape unchanged.",
  "seed": 42,
  "n_anchors": 3,
  "render_only_target": false,
  "use_face_parser": true,
  "reprojection_filter": "bilinear"
}
JSON

Run the edit directly after training:

"${FACEGS_PY}" scripts/run_edit_spec.py \
  --spec specs/local/clown_cn_ip2p.json \
  --config "${CONFIG_PATH}" \
  --canonical-dir "${CANONICAL_DIR}" \
  --outdir runs/manual_cn_ip2p

Submit the same edit through Slurm:

export FACEGS_SPEC_PATH=specs/local/clown_cn_ip2p.json
export FACEGS_RUN_LABEL=cn_ip2p_mean
export FACEGS_RUN_TAG=final_submission_check
sbatch scripts/slurm_run_edit_spec_once.sh

Run full analytics on a batch:

export RUNS_ROOT=/scratch/gpfs/JORDANAT/caleb/faceGS/runs/final2/final2_facegs_20260505_073908
export OUT_METRICS=/scratch/gpfs/JORDANAT/caleb/faceGS/paper_metrics/full_submission
export FACEGS_METRICS_PY=/scratch/gpfs/JORDANAT/caleb/.conda/envs/facegs-flux/bin/python
sbatch scripts/submit_paper_analytics_full.sh

Generated run folders, logs, paper metric outputs, local caches, and paper build artifacts are intentionally ignored by git.

Environments and Assets

scripts/_facegs_env.sh expects the main environment at:

/scratch/gpfs/JORDANAT/caleb/.conda/envs/facegs

Use this environment for Nerfstudio, Splatfacto, CN-IP2P, UV-canvas paint, rendering, and PLY export.

FLUX Kontext and paper metrics can use the newer Python stack:

export FACEGS_FLUX_PY=/scratch/gpfs/JORDANAT/caleb/.conda/envs/facegs-flux/bin/python
export FACEGS_METRICS_PY=/scratch/gpfs/JORDANAT/caleb/.conda/envs/facegs-flux/bin/python

The downloader checks local model assets:

"${FACEGS_PY}" scripts/download_facegs_edit_models.py --dry-run
"${FACEGS_PY}" scripts/download_facegs_edit_models.py --models all --with-clip --with-arcface

Default asset locations:

• diffusion checkpoints: /scratch/gpfs/JORDANAT/caleb/models/hf;
• InsightFace / ArcFace: /scratch/gpfs/JORDANAT/caleb/models/insightface;
• OpenAI CLIP ViT-L/14: /scratch/gpfs/JORDANAT/caleb/models/clip/ViT-L-14.pt;
• canonical bundles and trained checkpoints: artifacts/ and outputs/.

NeRSemble access is user-specific. Request access from the official project page at tobias-kirschstein.github.io/nersemble and use the official nersemble-data tooling with your personal NERSEMBLE_DATA_URL. Keep that URL in your local ~/.config/nersemble_data/.env or shell environment; never commit it to this repository.

EditSpec Basics

A minimal clown-makeup spec has this shape:

{
  "editor_mode": "anchor_view_controlnet_ip2p",
  "region_name": "face_inclusive",
  "prompt": "Turn him into a clown: white face paint covering forehead, cheeks, nose, and chin; red clown nose; red lips; dark eye makeup. Keep hair, hoodie, background, identity, and face shape unchanged.",
  "seed": 42,
  "n_anchors": 3,
  "render_only_target": false,
  "use_face_parser": true,
  "reprojection_filter": "bilinear"
}

Important conventions:

• region_name is an authorization boundary. The backend can only write where the canonical mask allows it.
• face_inclusive is the final full-face makeup region: skin plus nose, lips, and brows, excluding hair, background, neck, teeth, and inner mouth.
• render_only_target=false is the expected final setting because target-only renders tend to create pasted-on edits.
• best_owner and primary_plus_fill are production deposition modes; mean and median are ablations.
• FLUX Kontext specs should run with FACEGS_DTYPE=bfloat16 or fp32, not fp16.

Typical run outputs:

• edit_spec.json: exact spec consumed by the run;
• edited_appearance.pt: sidecar with updated Gaussian appearance;
• stats.json: backend, mask, refiner, and persistence diagnostics;
• edited_canvas.png, uv_delta.png, owner_mask.png, canonical_target_mask.png: UV-space diagnostics;
• renders/anchor_*_before.png and renders/anchor_*_after.png: anchor-view sanity checks;
• eval_renders/: paper-facing before/after/mask camera renders after analytics has been run.

Analytics and Metrics

The analytics workflow has two stages:

1. render_eval_cameras.py renders held-out before/after/mask views for each sidecar.
2. compute_paper_metrics.py computes JSON/CSV metrics, plots, and optional before-after result grids.

The renderer uses the Nerfstudio-compatible facegs environment. The metric stage can use facegs-flux via --metrics-py because CLIP, LPIPS, and MEt3R are easier to maintain on the newer PyTorch stack.

Metric groups:

• identity stability: outside-mask RGB delta, ArcFace, CLIP-I;
• localization: leakage, boundary/ring drift, writeback coverage;
• edit fidelity: target-region change, LPIPS/SSIM, CLIP-T;
• multi-view behavior: held-out deltas, sharpness retention, optional MEt3R;
• cost: render and edit wall time when available.

MEt3R defaults to RAFT geometry plus --met3r-distance lpips. This avoids the FeatUp CUDA extension, which is fragile when the visible CUDA toolkit does not match the PyTorch build. If FeatUp is installed cleanly, pass --met3r-distance cosine directly to compute_paper_metrics.py, or pass --compute-extra-args -- --met3r-distance cosine through run_paper_analytics.py.

File Map

Core pipeline:

• src/editing/edit_spec.py: EditSpec schema and editor-mode validation.
• src/editing/runtime/pipeline.py: end-to-end edit orchestration.
• src/editing/runtime/view_to_uv.py: view-to-canonical-UV reprojection.
• src/editing/runtime/region_masks.py: canonical and parser mask handling.
• src/editing/backends/_anchor_view_runner.py: shared multi-anchor deposition runner.
• src/editing/backends/anchor_view_controlnet_ip2p.py: CN-IP2P backend.
• src/editing/backends/anchor_view_flux_kontext.py: FLUX Kontext backend.
• src/editing/backends/uv_canvas_paint.py: deterministic UV-canvas deposit.
• src/ns/appearance_edit.py: features_dc writeback and optional higher-order SH damping.

Utilities and docs:

• tests/: focused pytest coverage for live runtime, geometry, and writeback code.
• scripts/run_paper_analytics.py: regenerates evaluation renders and paper metrics from ignored run folders when needed.
• THIRD_PARTY_NOTICES.md: dependency and model-credit notes.

Troubleshooting

• Missing facegs Python: set FACEGS_ENV_PREFIX before sourcing scripts/_facegs_env.sh.
• Missing FLUX Python: set FACEGS_FLUX_PY to the facegs-flux Python.
• Hugging Face asset unavailable: unset TRANSFORMERS_OFFLINE and HF_HUB_OFFLINE for the download step, set HF_TOKEN if needed, then rerun scripts/download_facegs_edit_models.py.
• Sticker-like edits: confirm render_only_target=false, inspect owner_mask.png, and keep use_face_parser=true when parser weights are available.
• Edit does not appear after writeback: inspect uv_delta.png, canonical_target_mask.png, stats.json, and edited_appearance.pt.
• Viewer splat looks sparse: use --opacity-min, --scale-multiplier, and --scale-only-edited in scripts/export_splat_to_inria_ply.py.