Harmonizer | Model Card

Paper | Project Page | Code | Model | Data

Description

Harmonizer is a single-step image diffusion model trained as an online generative enhancer for neural-reconstruction image and video renderings. It transforms imperfect novel-view renderings produced by Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) reconstructions into temporally consistent outputs that are closer to real captures, while correcting illumination, shadow, and reconstruction-artifact issues that arise when dynamic objects are composited into reconstructed scenes.

Harmonizer supports two operation modes:

• Offline mode: Used during the reconstruction phase to clean up pseudo-training views rendered from the reconstruction, then distill them back into 3D. This enhances underconstrained regions and improves overall 3D representation quality.
• Online mode: Acts as a single-step neural enhancer during simulation or inference. It harmonizes color and lighting, reconstructs missing or inconsistent shadows for inserted dynamic objects, and removes residual reconstruction artifacts from imperfect 3D supervision and current reconstruction-model capacity limits.

Harmonizer is designed as a single model compatible with both NeRF and 3DGS representations. The model was trained on data curated with 3DGUT-based reconstructions and is adaptable to Gaussian Splatting scenes.

License/Terms of Use

Governing Terms

Use of this model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography: Global

Release Management

The model artifacts are released in this repository. Training and inference code is available from the Harmonizer GitHub repository. The associated dataset is available from nvidia/Harmonizer-Dataset.

Use Case

Harmonizer is intended for Physical AI developers looking to enhance and harmonize neural-reconstruction pipelines for autonomous-vehicle simulation. The model takes an image or image sequence as input and outputs a harmonized image with corrected color, lighting, shadows, and reduced reconstruction artifacts.

Benchmark Results

Benchmarks were evaluated on 864 images from NDAS MLMCF and ParkNet training sessions. PSNR is higher-is-better; LPIPS and FID are lower-is-better.

Release Date

V1: June 2026

Reference(s)

• DiffusionHarmonizer paper
• DiffusionHarmonizer project page
• Harmonizer training and inference code
• Harmonizer dataset

Model Architecture

Architecture Type: Diffusion Transformer

Network Architecture: Diffusion Transformer, based on Cosmos Predict2 0.6B, post-trained as a single-step, temporally conditioned image-to-image enhancer for neural-reconstruction renderings.

The project page describes the backbone as the CosmosPredict2 0.6B text-to-image model fine-tuned on real-world and simulation training pairs from scalable data-curation pipelines for color and lighting harmonization, shadow correction, and artifact correction.

Model Input

Input Type(s): Image / Image sequence

Input Format: Red, Green, Blue (RGB)

Input Parameters: Two-Dimensional (2D)

Other Properties Related to Input: Specific resolution: 576 px x 1024 px

Model Output

Output Type(s): Image

Output Format: Red, Green, Blue (RGB)

Output Parameters: Two-Dimensional (2D)

Other Properties Related to Output: Specific resolution: 576 px x 1024 px

Software Integration

Runtime Engine(s): PyTorch

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Hopper
• NVIDIA Blackwell

Preferred/Supported Operating System(s): Linux

NVIDIA AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA hardware and software frameworks such as CUDA libraries, the model can achieve faster training and inference times compared to CPU-only systems.

Model Version

Harmonizer-cosmos-0.6B

We release two checkpoints specified below.

1. diffusion_harmonizer.pkl — The temporally-conditioned Harmonizer checkpoint reported in the DiffusionHarmonizer paper. Recommended when temporal coherence across consecutive rendered frames is required (e.g., video-style novel-view simulation). The model supports non-temporally faster conditioned inference mode via --nontemporal flag.

   Inference speed on H100:

   • full model (default): 212 ms / 576 x 1024 px image
   • --nontemporal mode: 28 ms / 576 × 1024 px image

2. harmonizer_nontemporal.pt — Exported JIT model for non-temporal, per-image inference. The checkpoint does not support conditioning on previous frames and corresponds to diffusion_harmonizer.pkl with --nontemporal flag. Recommended for per-image enhancement use cases where neighboring-frame context is unavailable or unnecessary, or where speed is critical.

   Inference speed on H100: 28 ms / 576 × 1024 px image.

Pretrained checkpoints are hosted on Hugging Face under nvidia/Harmonizer. To download all released checkpoints into a local models/ directory:

hf download nvidia/Harmonizer --local-dir models

Refer to the code release for the exact inference entry points and configuration files associated with each checkpoint. By default, the model runs in temporal mode. To run in non-temporal mode, add the --nontemporal flag. Refer to the code release for the exact inference entry points.

Repository Contents

This repository contains the following model artifact files:

• diffusion_harmonizer.pkl — DiffusionHarmonizer paper temporal checkpoint
• harmonizer_nontemporal.pt — non-temporal single-frame checkpoint (PyTorch .pt format)

For more details please see the Model Version section.

Training, Testing, and Evaluation Datasets

Harmonizer was trained, tested, and evaluated using an internal dataset of curated synthetic–real image pairs constructed from five complementary curation pipelines (ISP modification, relighting, asset re-insertion, PBR shadow simulation, and novel-view artifact correction), where 80% of the data was used for training, 10% for evaluation, and 10% for testing. The total volume of training data amounted to ~1 million pairs. Training data is available at nvidia/Harmonizer-Dataset.

NVIDIA Internal AV Dataset

Data Collection Method: Sensors

Labeling Method by Dataset: Human

Properties: The dataset contains autonomous-driving images and videos captured by NVIDIA vehicles. It is collected by autonomous-driving vehicles and used as the source data from which the synthetic-real training pairs are derived.

Inference

Engine: PyTorch>=2.0.0

Test Hardware: We tested on H100:

diffusion_harmonizer.pkl

• full model (default): 212 ms / 576 x 1024 px image
• --nontemporal mode: 28 ms / 576 × 1024 px image

harmonizer_nontemporal.pt

• 28 ms / 576 × 1024 px image

Known Technical Limitations

The reconstruction relies on the quality and consistency of input images and camera calibrations; deficiencies in these areas can negatively impact the final output.

Known Risk(s)

The model is not guaranteed to fix 100% of image artifacts. Please verify generated scenarios are context and use appropriate.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and has established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with the terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the ModelCard++ Explainability, Bias, Safety & Security, and Privacy subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.

ModelCard++

Bias

Explainability

Privacy

Safety & Security

Support

Usage questions and discussion: please post on the NVIDIA Developer Forum (Omniverse / NuRec).

Code-level bugs, documentation issues, and feature requests: file a GitHub issue using the appropriate template. The relevant NVIDIA responder is assigned automatically.

Security vulnerabilities: use NVIDIA's Vulnerability Disclosure Program. Do not file security issues publicly in this repository.

The Hugging Face Community tab for this model card will be disabled on 08/10/2026. Please use the channels above.