PBR Estimation Benchmark

A community benchmark for recovering relightable PBR materials from 3D assets with baked lighting.

Motivation

An increasing number of 3D assets are reconstructed from real objects or generated by learned models. Their geometry and RGB textures may look convincing under the original illumination, but shadows, highlights, and reflections are often baked into their appearance.

This limits their use in VFX, games, augmented reality, robotics, and other applications where an asset must respond naturally to a new scene. Recent methods such as MaterialAnything, MaterialFusion, SuperMat, and MatMart address parts of this problem, but evaluate with different data, inputs, renderers, conventions, and metrics. The lack of a shared benchmark makes progress difficult to verify and reproduce.

This repository provides a common evaluation protocol, prepared datasets, method adapters, and reproducible metrics for this task.

Problem formulation

Given a textured 3D object with baked or entangled illumination—whether generated, scanned, or authored by an artist—the goal is to recover complete UV-space PBR materials:

textured 3D object  ->  base color + roughness + metallic UV maps

The recovered asset should preserve its identity and appearance while responding correctly to changing illumination in standard rendering engines.

The initial benchmark uses the glTF metallic-roughness representation: linear-RGB base color, perceptual roughness, and metallic values in [0, 1]. It focuses on opaque, rigid objects; more expressive material models are left to future extensions.

Evaluation settings

Single view

The method receives one RGB rendering and registered object geometry buffers—such as normals, depth, position, camera parameters, and a foreground mask—and predicts visible screen-space PBR channels:

RGB + geometry buffers  ->  visible base color + roughness + metallic

This restricted task supports rapid iteration and lets image-space inverse-rendering methods contribute to object material recovery. Their predictions may later be projected and combined across a mesh, as done by projection-based object texturing pipelines.

Multi view

The method receives a textured mesh and one or more registered RGB observations. It must recover complete and consistent UV-space materials:

textured mesh + RGB observations  ->  complete PBR material asset

This is the primary practical task. It measures illumination removal, cross-view consistency, evidence aggregation, and completion of weakly observed or unseen surfaces. Results are grouped by input-view budget.

Benchmark tracks

Both settings are evaluated in two complementary ways.

1. Direct PBR evaluation

Predicted base color, roughness, and metallic are compared with reference PBR annotations. Single-view predictions are evaluated in visible screen space; multi-view predictions are evaluated in UV space and from standardized rendered views.

Direct evaluation provides interpretable per-channel errors, but it is necessarily constrained to assets with PBR annotations. These are usually synthetic artist-authored assets or digital twins whose materials remain estimates of real physical appearance. The task is also ill-posed: multiple material and illumination decompositions may explain the same input image.

We therefore report direct material accuracy as a diagnostic rather than a complete measure of quality.

2. Indirect evaluation through relighting

Predicted materials are rendered under held-out illumination and compared with reference images. This evaluates whether the recovered asset models appearance under changing light rather than whether it reproduces one particular material decomposition.

The same formulation works with both synthetic and real data. Synthetic assets provide exact reference renders and controlled PBR annotations. Real captures only require the same object under different measured lighting; they do not require exact real-world PBR labels. Relighting therefore greatly broadens the data that can be used for evaluation.

All submissions are relit using a fixed, versioned renderer and common material convention. We report linear-HDR errors, exposure-aligned metrics where necessary, and fixed-tone-map PSNR, SSIM, and LPIPS. Digital Twin Catalog, Objects With Lighting, and Stanford-ORB are candidate real-data sources.

Direct PBR scores and relighting scores remain separate so that the benchmark exposes rather than hides different failure modes.

Methods

Object-centric methods such as MaterialAnything, MaterialFusion/StableMaterial, Neural LightRig, SuperMat, DualMat, and MatMart form the primary comparison set.

General image-space inverse-rendering methods such as RGB↔X, Intrinsic Image Diffusion, IntrinsicAnything, IDArb, and image-mode DiffusionRenderer are secondary single-view baselines. They help measure how well general 2D priors transfer to object material recovery, but do not define the main task.

Third-party implementations are included as Git submodules under third_party/; this repository does not vendor or modify their source. Each method keeps its own environment and installation procedure. Small adapters under src/methods_2d/ and src/methods_3d/ translate between the benchmark sample format and the method's native interface.

git clone --recurse-submodules <repository-url>
uv sync

The original license and usage terms of every third-party project still apply. Setup instructions and known limitations are documented separately under docs/methods/.

Usage

Everything is configured through Hydra. The benchmark and method-specific environments use uv; each method environment launches the common inference entrypoint directly.

# Download the selected source lights
uv run python scripts/download_polyhaven_envmaps.py \
  --config configs/data/preprocessing/lighting/polyhaven_source_v0.yaml

# Download only the curated 64-object Objaverse PBR subset
uv run python src/data/preprocessing/objaverse/download.py

# Download the curated TexVerse PBR subset
uv run python src/data/preprocessing/texverse/download.py

# Download the DTC pilot subset. Fetch a fresh, temporary URL manifest from the
# DTC site after accepting its licence; do not commit that manifest.
uv run python src/data/preprocessing/dtc/download.py \
  --download-urls-json data/DTC_objects_all_download_urls.json

# Download the curated Poly Haven models. Every downloader writes one
# self-contained GLB per object under data/assets/<source>/<object_id>.glb.
uv run python src/data/preprocessing/polyhaven/download.py

# All source downloaders use this layout:
# data/assets/<source>/<object_id>.glb

# Render both TexVerse representations (run `module load blender` on the cluster)
uv run python src/data/preprocessing/render_views_2d.py
uv run python src/data/preprocessing/render_views_3d.py

# Use the same renderers for the downloaded DTC assets
uv run python src/data/preprocessing/render_views_2d.py \
  --config-name data/preprocessing/dtc_2d
uv run python src/data/preprocessing/render_views_3d.py \
  --config-name data/preprocessing/dtc_3d

# Render the downloaded Poly Haven model subset
uv run python src/data/preprocessing/render_views_2d.py \
  --config-name data/preprocessing/polyhaven_2d
uv run python src/data/preprocessing/render_views_3d.py \
  --config-name data/preprocessing/polyhaven_3d

# Create a cached, uv-managed environment for the selected method (once)
uv run python scripts/setup/neural_lightrig_deps.py
uv run python scripts/setup/supermat_deps.py

# Screen-space PBR inference runs in the selected method's uv environment
third_party/.venvs/supermat/bin/python src/infer_pbr_2d.py method_2d=supermat
third_party/.venvs/neural_lightrig/bin/python src/infer_pbr_2d.py \
  method_2d=neural_lightrig

# Physics-only diffuse albedo from expanded Blender relightings and proxies
# (run `module load blender` first)
uv run python src/infer_pbr_2d.py method_2d=pbr_oracle

# Restrict a development run without changing the prepared dataset
third_party/.venvs/supermat/bin/python src/infer_pbr_2d.py \
  method_2d=supermat data.max_samples=2

# Evaluate saved predictions directly against the PBR reference maps
uv run python src/eval_pbr_2d_direct.py \
  predictions_dir=outputs/pbr_2d/supermat/predictions

# Relight predicted and reference maps in Blender, then compare the renders
# (run `module load blender/3.4.1` on the cluster)
uv run python src/eval_pbr_2d_indirect.py \
  predictions_dir=outputs/pbr_2d/supermat/predictions

Each inference run stores canonical predictions and its resolved configuration in a self-contained output directory. Inference can be resumed without recomputing completed samples. Direct and relighting evaluation will be separate entry points so saved predictions can be evaluated in the benchmark's root environment.

Repository layout

configs/                 Hydra configuration groups
docs/
  methods/               per-method setup and reproduction notes
src/
  infer_pbr_2d.py        screen-space PBR inference entry point
  infer_pbr_3d.py        UV-space PBR inference entry point (planned)
  eval_pbr_2d_direct.py  direct map evaluation
  eval_pbr_2d_indirect.py
                          Blender relighting evaluation
  data/
    pbr_estimation_dataset_2d.py
                          manifest-backed screen-space dataset
    preprocessing/
      render_views_2d.py    2D rendering launcher
      _render_views_2d.py   Blender-side 2D renderer
      render_views_3d.py    3D baking launcher
      _render_views_3d.py   Blender-side mesh/texture exporter
    preprocessing/texverse/download.py
                          TexVerse source-data downloader
  methods_2d/            screen-space third-party adapters
  methods_3d/            UV-space third-party adapters (planned)
  evaluation/            direct and relighting evaluators
third_party/             upstream methods as Git submodules
scripts/                 small download and repository utilities

The detailed implementation contract is defined in SPEC.md. The proposed first data milestone is described in docs/data/sv_direct_preprocessing.md, and the literature review is in docs/single_view_literature_review.md.

Status

This README describes the intended finished project. The repository is being implemented backwards from this public contract.