Genwarp Model Card

Genwarp models are the official checkpoints for ther paper "GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping". Genwarp can generate novel view images from a single input conditioned on camera poses. In this repository, we offer the codes for inference of the model. For detailed information, please refer to the paper.

Model Details

Model Description

• Finetuned from model: Stable Diffusion v1-5
• License: Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC 4.0) and CreativeML Open RAIL-M for Use Restrictions. See LICENSE for more details.

Model Sources

• Repositosy: GitHub
• Paper: arXiv

Variations

• multi1: datasets used are RealEstate10K, ACID, and ScanNet
• multi2: datasets used are RealEstate10K, ACID, ScanNet, and MegaScene v1.0
• See Training section for more details.

Uses

Direct Use

The model is intended for research purposes only. Possible research areas and tasks include

• Safe deployment of models which have the potential to generate harmful content.
• Probing and understanding the limitations and biases of generative models.
• Generation of artworks and use in design and other artistic processes.
• Research on generative models.

Excluded uses are described below.

Misuse, Malicious Use, and Out-of-Scope Use

Note: This section is taken from the CreativeML Open RAIL-M Genwarp models.

Use Restrictions

You agree not to use the Model or Derivatives of the Model:

• In any way that violates any applicable national, federal, state, local or international law or regulation;
• For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
• To generate or disseminate verifiably false information and/or content with the purpose of harming others;
• To generate or disseminate personal identifiable information that can be used to harm an individual;
• To defame, disparage or otherwise harass others;
• For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
• For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
• To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
• For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories;
• To provide medical advice and medical results interpretation;
• To generate or disseminate information for the purpose to be used for administration of justice, law enforcement, immigration or asylum processes, such as predicting an individual will commit fraud/crime commitment (e.g. by text profiling, drawing causal relationships between assertions made in documents, indiscriminate and arbitrarily-targeted use).

How to Get Started with the Model

For use-cases and examples, follow the instructions here

Training Detalis

Training Data The model developers used the following dataset for training the model:

• RealEstate10K (CC-BY-4.0)
• Aerial Coastline Imagery Dataset (ACID)
• ScanNet (ScanNet Terms of Use)
• MegaScene v1.0 (CC-BY-4.0)

Training Procedure

Genwarp models are finetuned based on Stable Diffusion v1-5.

For more details, please refer to the paper.

Evaluation Results

Please refer to the paper.

Citation

@article{seo2024genwarp,
  title={GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping},
  author={Junyoung Seo and Kazumi Fukuda and Takashi Shibuya and Takuya Narihira and Naoki Murata and Shoukang Hu and Chieh-Hsin Lai and Seungryong Kim and Yuki Mitsufuji},
  year={2024},
  journal={arXiv preprint arXiv:2405.17251}
}