Conference: IEEE/CVF International Conference on Computer Vision (ICCV), 2025

👀 Overview

DIR dataset is a dataset for Display Inverse Rendering (DIR). It contains assets captured from LCD & polarization-camera system.

• OLAT Images: are capured under display superpixels, and can be used to simulate arbitrary display patterns.
• GT Geometry: is scanned with a high-precision 3D scanner.
• Lighting information: We carefully calibrated light direction, non-linearity, and backlight.
• Stereo Imaging: is an optional feature to initialize rough geometry.

Why Display Inverse Rendering? Display inverse rendering uses a monitor as a per-pixel, programmable light source to reconstruct object geometry and reflectance from captured images. Key features include:

• Illumination Multiplexing: encodes multiple lights and reduces demanded a number of inputs.
• Leveraging Polarization: enables diffuse-specular separation based on optics.

📝 TODO List

• Release a baseline code.
• Release Display Inverse Rendering (DIR) dataset.
• Release EXPANDED version of DIR datset (HDR).
• Release EXPANDED version of DIR datset (multi-distance).
• Release additional visualization tools.
• Release raw image processing code.

🚀 Get Started

⚙️ Installation

git clone https://github.com/MichaelCSJ/DIR.git
cd DIR
conda env create -f environment.yml
conda activate DIR

🗂️ Dataset Preparation

Download the DIR dataset for perform our display inverse rendering baseline. It consists of 16 real-world objects with diverse shapes and materials under precisely calibrated directional lighting. There are some versions of dataset as 'DIR-basic', 'DIR-pms', 'DIR-hdr', and 'DIR-multi-distance'.

• DIR-basic: The basic version of the dataset released with the paper. It includes stereo polarized RAW images, RGB images from a reference view, and ground-truth surface normals and point clouds. All images are captured under a multi-light configuration projected through 16×9 superpixels on the display.

  ├── A
  │  ├──GT_geometry (for reference(main) view)
  │  │  ├──'normal.npy',
  │  │  ├──'normal.png',
  │  │  ├──'point_cloud_gt.npy'
  │  ├──main
  │  │  ├──diffuseNspecular
  │  │  │  ├──'000 - 143.png',
  │  │  │  ├──'black.png',
  │  │  │  ├──'white.png',
  │  │  ├──RAW_polar
  │  │  │  ├──'000 - 143_[SHUTTER_TIME].png',
  │  │  │  ├──'black_[SHUTTER_TIME].png',
  │  │  │  ├──'white_[SHUTTER_TIME].png',
  │  ├──side
  │  │  ├──diffuseNspecular
  │  │  │  ├──'000 - 143.png',
  │  │  │  ├──'black.png',
  │  │  │  ├──'white.png',
  │  │  ├──RAW_polar
  │  │  │  ├──'000 - 143_[SHUTTER_TIME(us)].png',
  │  │  │  ├──'black_[SHUTTER_TIME(us)].png',
  │  │  │  ├──'white_[SHUTTER_TIME(us)].png',
  │  ├──'mask.png'
  │  ├──'point_cloud.npy' (unprojected pixel w.r.t. depth & focal length)

• DIR-pms: This dataset follows the DiLiGenT format and has the same composition as DIR-basic. It provides multi-light RGB images from the reference view along with related information and the ground-truth normal maps.

  ├── A [Suffix (default "PNG")]
  │  ├──'000 - 143.png',
  │  ├──'filenames.txt',
  │  ├──'light_directions.txt'
  │  ├──'light_intensities.txt',
  │  ├──'mask.png'
  │  ├──'Normal_gt.mat'

• DIR-hdr: TBD.
• DIR-multi-distance: TBD.

After downloading, place them under data/ as the following directory tree.

🔥 Normal and basis BRDF Recovery

To run the baseline, execute train.py with the following command:

python train.py --name YOUR_SESSION_NAME --dataset_root YOUR_DATASET_PATH

By default, this code performs inverse rendering using multi-light images captured with an OLAT pattern. If you want to use a small number of multi-light images with a multiplexed display pattern, run the code as follows:

python train.py --name YOUR_SESSION_NAME --dataset_root YOUR_DATASET_PATH --use_multiplexing True --initial_light_pattern YOUR_DISPLAY_PATTERNS

You can use display patterns provided by DDPS for YOUR_DISPLAY_PATTERNS. Place display patterns under patterns/ as the following directory tree.

Lighting Patterns (Initial):

Lighting Patterns (Learned):

Once training is completed, a folder named YYYYMMDD_HHMMSS will be created inside the /results/SESSION directory, containing the TensorBoard logs, OLAT rendering results, and the fitted parameters for each object.

🖼️ Novel Relighting (Optional, TBD)

Run relighting.py to render images under novel directional lightings based on recovered normal map and BRDF parameter maps.

You can also use the provided code (relighting.py) for relighting the object under novel directional lights based on the recovered attributes. Follow the instructions displayed at the end of the prompt to use it. It should look like this.

To output .avi video:

 python relighting.py --datadir ./results/YOUR_SESSION_NAME/OBJECT_NAME --format avi

🔖 Citation

If you find this repository useful, please consider citing this paper:

(TBD)