yangjiheng/nerf_and_beyond_docs

1,163

204 commits

updated Dec 26, 2024

README

NeRF and Beyond Docs

This is a collection of documents and topics NeRF/3DGS & Beyond channel accumulated, as well as papers in literaure. Since there are lots of papers out there, so we split them into two seperate repositories: NeRF and Beyond Docs and 3DGS and Beyond Docs. Please choose accordingly recarding to your preference.

Some papers we discussed in the group, will be added to the back of the paper with a Notes link. You can follow the link to check whether there is topic you are interested in. If not, welcome to join us and ask the question to the crowd. The mighty community might have your answers.

We are actively maintaining this page trying to stay up-to-date and gather important works in a daily basis. We would also like to put as many notes as possible to some works, trying to make it easier to catch up.

Please feel free to join us on WeChat group or start a discussion topic here.

NeRF/3DGS Book

I have recently published a book with PHEI(Publishing House of Electronics Industry) on NeRF/3DGS. This would not have been possible without the help of the whole 3D vision community. It is now available on jd.com (Checkout here) and it should be suitable as a reference handbook for NeRF/3DGS beginners or engineers in related areas. I sincerely hope the book can be helpful in any perspective.

For those of you who have already purchased the book, all references can be downloaded HERE. If you experience any issue reading the book or have any suggestions to improve it, please contact me through my email address: jiheng.yang@gmail.com, or directly concact me on WeChat: jiheng_yang. I'm looking forward to talk to anyone reaching out to me, thanks in advance.

For readers who are insterested in commended version (actually enhanced compared to the book, much more detailed), please click THIS. It could serve as a companion source code implementation for the book.

How to join us

For now, you can join us in the following ways

Bilibili Channel where we post near daily updates (primarily) on NeRF.
WeChat group, due to the limitation of WeChat group, you can add my personal account: jiheng_yang, and I will add you to the chat groups.
If you want to view this from a timeline perspective, please refer to this ProcessOn Diagram
If you think something is not correct or you think we could do better in some way, please write to us through all possible channels or drop an issue. All suggestions are appreciated!
For other discussed techniques that's related to 3D reconstruction and NeRF, please refer to link, we are constantly trying to add more resource to this document.
We are trying to gradually setup Discord channels, join the Discord Channel if you want to, we will certainly look forward to talk with you guys!

3DGS Progresses

For 3DGS related progress, you can refer to 3DGS and Beyond Docs

NeRF progresses

Table of Content

NeRF and Beyond Docs

New to NeRF

Begin of NeRF, Always Start Here

:fire:NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
ECCV 2020, 19 Mar 2020

Abstract

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x,y,z) and viewing direction (θ,ϕ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

[arXiv] [Project] [Code] [PyTorch Impl] [Notes]

:fire:State of the Art on Neural Rendering
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer
ECCV 2020,8 Apr 2020

Abstract

Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state-of-the-art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. This state-of-the-art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

[arXiv]

:fire:Advances in Neural Rendering
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, Tomas Simon, Christian Theobalt, Matthias Niessner, Jonathan T. Barron, Gordon Wetzstein, Michael Zollhoefer, Vladislav Golyanik
ECCV 2022, 10 Nov 2021

Abstract

Synthesizing photo-realistic images and videos is at the heart of computer graphics and has been the focus of decades of research. Traditionally, synthetic images of a scene are generated using rendering algorithms such as rasterization or ray tracing, which take specifically defined representations of geometry and material properties as input. Collectively, these inputs define the actual scene and what is rendered, and are referred to as the scene representation (where a scene consists of one or more objects). Example scene representations are triangle meshes with accompanied textures (e.g., created by an artist), point clouds (e.g., from a depth sensor), volumetric grids (e.g., from a CT scan), or implicit surface functions (e.g., truncated signed distance fields). The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering. Neural rendering is closely related, and combines ideas from classical computer graphics and machine learning to create algorithms for synthesizing images from real-world observations. Neural rendering is a leap forward towards the goal of synthesizing photo-realistic image and video content. In recent years, we have seen immense progress in this field through hundreds of publications that show different ways to inject learnable components into the rendering pipeline. This state-of-the-art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non-rigidly deforming objects...

[arXiv]

:fire:NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review
Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, Jonathan Li
TPAMI 2022, 1 Oct 2022

Abstract

Neural Radiance Field (NeRF) has recently become a significant development in the field of Computer Vision, allowing for implicit, neural network-based scene representation and novel view synthesis. NeRF models have found diverse applications in robotics, urban mapping, autonomous navigation, virtual reality/augmented reality, and more. Due to the growing popularity of NeRF and its expanding research area, we present a comprehensive survey of NeRF papers from the past two years. Our survey is organized into architecture and application-based taxonomies and provides an introduction to the theory of NeRF and its training via differentiable volume rendering. We also present a benchmark comparison of the performance and speed of key NeRF models. By creating this survey, we hope to introduce new researchers to NeRF, provide a helpful reference for influential works in this field, as well as motivate future research directions with our discussion section.

[arXiv]

Neural Radiance Fields: Past, Present, and Future
Ansh Mittal
In Progress,20 Apr 2023
[arXiv]

Survey on Fundamental Deep Learning 3D Reconstruction Techniques
Yonge Bai, LikHang Wong, TszYin Twan
arXiv preprint, 11 Jul 2024
[arXiv]

3D Representation Methods: A Survey
Zhengren Wang
arXiv preprint, 9 Oct 2024
[arXiv]

NeRF Tutorials

:fire:Neural Rendering Course
SIGGRAPH 2021 [BiliBili]

:fire:Neural Volumetric Rendering for Computer Vision
ECCV 2022 Tutorial [Website]

:fire:Scaling NeRF Up and Down: Big Scenes and Real-Time View Synthesis
I3D 2023 Keynote [Video]

NeRF Profiling

NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
Jonas Kulhanek, Torsten Sattler
arXiv preprint, 25 Jun 2024
[arXiv] [Project]

NeRF OpenSource Tools

:fire:Nerfstudio: A Modular Framework for Neural Radiance Field Development
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, Angjoo Kanazawa
arXiv preprint, 8 Feb 2023

Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and visualizing NeRFs. The library supports an interpretable implementation of NeRFs by modularizing each component.

[arXiv] [Website] [Github]

:fire:NerfAcc: Efficient Sampling Accelerates NeRFs
Ruilong Li, Hang Gao, Matthew Tancik, Angjoo Kanazawa
arXiv preprint, 8 May 2023

NerfAcc is a PyTorch Nerf acceleration toolbox for both training and inference. It focus on efficient sampling in the volumetric rendering pipeline of radiance fields, which is universal and plug-and-play for most of the NeRFs. With minimal modifications to the existing codebases, Nerfacc provides significant speedups in training various recent NeRF papers. And it is pure Python interface with flexible APIs!

[arXiv] [Website] [Github]

:fire:threestudio: A unified framework for 3D content generation
Yuan-Chen Guo and Ying-Tian Liu and Chen Wang and Zi-Xin Zou and Guan Luo and Chia-Hao Chen and Yan-Pei Cao and Song-Hai Zhang
Github repo,2023

threestudio is a unified framework for 3D content creation from text prompts, single images, and few-shot images, by lifting 2D text-to-image generation models.

[Github]

NeRF Fundamental Enhancements

:fire:NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth
CVPR 2021, 5 Aug 2020

Abstract

We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues, thereby enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.

[arXiv] [Project]

:fire:Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan
ICCV 2021, 24 Mar 2021

Abstract

The rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. The straightforward solution of supersampling by rendering with multiple rays per pixel is impractical for NeRF, because rendering each ray requires querying a multilayer perceptron hundreds of times. Our solution, which we call "mip-NeRF" (a la "mipmap"), extends NeRF to represent the scene at a continuously-valued scale. By efficiently rendering anti-aliased conical frustums instead of rays, mip-NeRF reduces objectionable aliasing artifacts and significantly improves NeRF's ability to represent fine details, while also being 7% faster than NeRF and half the size. Compared to NeRF, mip-NeRF reduces average error rates by 17% on the dataset presented with NeRF and by 60% on a challenging multiscale variant of that dataset that we present. Mip-NeRF is also able to match the accuracy of a brute-force supersampled NeRF on our multiscale dataset while being 22x faster.

[arXiv] [Project] [Github] [Notes]

:fire:Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman
CVPR 2022, 23 Nov 2021

Abstract

Though neural radiance fields (NeRF) have demonstrated impressive view synthesis results on objects and small bounded regions of space, they struggle on "unbounded" scenes, where the camera may point in any direction and content may exist at any distance. In this setting, existing NeRF-like models often produce blurry or low-resolution renderings (due to the unbalanced detail and scale of nearby and distant objects), are slow to train, and may exhibit artifacts due to the inherent ambiguity of the task of reconstructing a large scene from a small set of images. We present an extension of mip-NeRF (a NeRF variant that addresses sampling and aliasing) that uses a non-linear scene parameterization, online distillation, and a novel distortion-based regularizer to overcome the challenges presented by unbounded scenes. Our model, which we dub "mip-NeRF 360" as we target scenes in which the camera rotates 360 degrees around a point, reduces mean-squared error by 57% compared to mip-NeRF, and is able to produce realistic synthesized views and detailed depth maps for highly intricate, unbounded real-world scenes.

[arXiv] [Project] [Github] [Notes]

:fire:Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, Pratul P. Srinivasan
CVPR 2022, 7 Dec 2021

Abstract

Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons that provide the volume density and view-dependent emitted radiance at each location. While NeRF-based techniques excel at representing fine geometric structures with smoothly varying view-dependent appearance, they often fail to accurately capture and reproduce the appearance of glossy surfaces. We address this limitation by introducing Ref-NeRF, which replaces NeRF's parameterization of view-dependent outgoing radiance with a representation of reflected radiance and structures this function using a collection of spatially-varying scene properties. We show that together with a regularizer on normal vectors, our model significantly improves the realism and accuracy of specular reflections. Furthermore, we show that our model's internal representation of outgoing radiance is interpretable and useful for scene editing.

[arXiv] [Project] [Github]

:fire:PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo
Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong
ECCV 2022, 23 Jul 2022

Abstract

Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer. After optimization, the reconstructed object can be used for novel-view rendering, relighting, and material editing. Experiments on both synthetic and real datasets demonstrate that our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods. Our code and model can be found at this https URL.

[arXiv] [Project] [Github]

4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions
Zhongshu Wang, Lingzhi Li, Zhen Shen, Li Shen, Liefeng Bo
arXiv preprint, 9 Dec 2022
[arXiv] [Project] [Github]

:fire:Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman
arXiv preprint, 13 Apr 2023

Abstract

Neural Radiance Field training can be accelerated through the use of grid-based representations in NeRF's learned mapping from spatial coordinates to colors and volumetric density. However, these grid-based approaches lack an explicit understanding of scale and therefore often introduce aliasing, usually in the form of jaggies or missing scene content. Anti-aliasing has previously been addressed by mip-NeRF 360, which reasons about sub-volumes along a cone rather than points along a ray, but this approach is not natively compatible with current grid-based techniques. We show how ideas from rendering and signal processing can be used to construct a technique that combines mip-NeRF 360 and grid-based models such as Instant NGP to yield error rates that are 8% - 77% lower than either prior technique, and that trains 24x faster than mip-NeRF 360.

[arXiv] [Project] [Unofficial Impl] [Notes]

Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs
Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, Angjoo Kanazawa
arXiv preprint, 20 Apr 2023
[arXiv] [Project] [Github] [Video]

Multi-Space Neural Radiance Fields
Ze-Xin Yin, Jiaxiong Qiu, Ming-Ming Cheng, Bo Ren
CVPR 2023, 7 May 2023
[arXiv] [Project] [Github] [Video] [Notes]

NeuManifold: Neural Watertight Manifold Reconstruction with Efficient and High-Quality Rendering Support
Xinyue Wei, Fanbo Xiang, Sai Bi, Anpei Chen, Kalyan Sunkavalli, Zexiang Xu, Hao Su
arXiv preprint, 26 May 2023
[arXiv] [Project] [Video]

Analyzing the Internals of Neural Radiance Fields
Lukas Radl, Andreas Kurz, Markus Steinberger
arXiv preprint, 1 Jun 2023
[arXiv] [Project]

GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner
arXiv preprint, 9 Jun 2023
[arXiv] [Video]

HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork
Bipasha Sen, Gaurav Singh, Aditya Agarwal, Rohith Agaram, K Madhava Krishna, Srinath Sridhar
arXiv preprint, 9 Jun 2023
[arXiv]

:fire:Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields
Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, Yuewen Ma
ICCV 2023, 21 Jul 2023

Abstract

Despite the tremendous progress in neural radiance fields (NeRF), we still face a dilemma of the trade-off between quality and efficiency, e.g., MipNeRF presents fine-detailed and anti-aliased renderings but takes days for training, while Instant-ngp can accomplish the reconstruction in a few minutes but suffers from blurring or aliasing when rendering at various distances or resolutions due to ignoring the sampling area. To this end, we propose a novel Tri-Mip encoding that enables both instant reconstruction and anti-aliased high-fidelity rendering for neural radiance fields. The key is to factorize the pre-filtered 3D feature spaces in three orthogonal mipmaps. In this way, we can efficiently perform 3D area sampling by taking advantage of 2D pre-filtered feature maps, which significantly elevates the rendering quality without sacrificing efficiency. To cope with the novel Tri-Mip representation, we propose a cone-casting rendering technique to efficiently sample anti-aliased 3D features with the Tri-Mip encoding considering both pixel imaging and observing distance. Extensive experiments on both synthetic and real-world datasets demonstrate our method achieves state-of-the-art rendering quality and reconstruction speed while maintaining a compact representation that reduces 25% model size compared against Instant-ngp.

[arXiv] [Project] [Video]

Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields
Lily Goli, Cody Reading, Silvia Selllán, Alec Jacobson, Andrea Tagliasacchi
arXiv preprint, 6 Sep 2023
[arXiv] [Project]

ResFields: Residual Neural Fields for Spatiotemporal Signals
Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, Siyu Tang
arXiv preprint, 1 Oct 2023 [arXiv] [Project] [Github]

:fire:NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions
Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, Yi Xu
ICCV 2023, 27 Sep 2023

Abstract

We present a novel type of neural fields that uses general radial bases for signal representation. State-of-the-art neural fields typically rely on grid-based representations for storing local neural features and N-dimensional linear kernels for interpolating features at continuous query points. The spatial positions of their neural features are fixed on grid nodes and cannot well adapt to target signals. Our method instead builds upon general radial bases with flexible kernel position and shape, which have higher spatial adaptivity and can more closely fit target signals. To further improve the channel-wise capacity of radial basis functions, we propose to compose them with multi-frequency sinusoid functions. This technique extends a radial basis to multiple Fourier radial bases of different frequency bands without requiring extra parameters, facilitating the representation of details. Moreover, by marrying adaptive radial bases with grid-based ones, our hybrid combination inherits both adaptivity and interpolation smoothness. We carefully designed weighting schemes to let radial bases adapt to different types of signals effectively. Our experiments on 2D image and 3D signed distance field representation demonstrate the higher accuracy and compactness of our method than prior arts. When applied to neural radiance field reconstruction, our method achieves state-of-the-art rendering quality, with small model size and comparable training speed.

[arXiv] [Project] [Github]

Multi-task View Synthesis with Neural Radiance Fields
Shuhong Zheng, Zhipeng Bao, Martial Hebert, Yu-Xiong Wang
ICCV 2023, 29 Sep 2023
[arXiv] [Project] [Github]

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields
Yifan Wang, Yi Gong, Yuan Zeng
arXiv preprint, 21 Nov 2023
[arXiv]

VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
Yiying Yang, Wen Liu, Fukun Yin, Xin Chen, Gang Yu, Jiayuan Fan, Tao Chen
AAAI 2024, 23 Oct 2023
[arXiv]

Rethinking Directional Integration in Neural Radiance Fields
Congyue Deng, Jiawei Yang, Leonidas Guibas, Yue Wang
arXiv preprint, 28 Nov, 2023
[arXiv]

RING-NeRF: A Versatile Architecture based on Residual Implicit Neural Grids
Doriand Petit, Steve Bourgeois, Dumitru Pavel, Vincent Gay-Bellile, Florian Chabot, Loic Barthe
arXiv preprint, 6 Dec, 2023
[arXiv]

Methods and strategies for improving the novel view synthesis quality of neural radiation field
Shun Fang, Ming Cui, Xing Feng, Yanna Lv
arXiv preprint, 23 Jan 2024
[arXiv]

Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance Fields
Rongkai Ma, Leo Lebrat, Rodrigo Santa Cruz, Gil Avraham, Yan Zuo, Clinton Fookes, Olivier Salvado
arXiv preprint, 29 Jan 2024
[arXiv]

TaylorGrid: Towards Fast and High-Quality Implicit Field Learning via Direct Taylor-based Grid Optimization
Renyi Mao, Qingshan Xu, Peng Zheng, Ye Wang, Tieru Wu, Rui Ma
arXiv preprint, 22 Feb 2024
[arXiv]

Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields
Seungtae Nam, Daniel Rho, Jong Hwan Ko, Eunbyung Park
NeurIPS 2023, 22 Feb 2024
[arXiv] [Project] [Code]

NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning
Linsheng Chen, Guangrun Wang, Liuchun Yuan, Keze Wang, Ken Deng, Philip H.S. Torr
AAAI 2024, 2 Mar 2024
[arXiv] [Code]

RoGUENeRF: A Robust Geometry-Consistent Universal Enhancer for NeRF
Sibi Catley-Chandar, Richard Shaw, Gregory Slabaugh, Eduardo Perez-Pellitero
arXiv preprint, 18 Mar 2024
[arXiv]

Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation
Yujin Chen, Yinyu Nie, Benjamin Ummenhofer, Reiner Birkl, Michael Paulitsch, Matthias Müller, Matthias Nießner
arXiv preprint, 28 Mar 2024
[arXiv] [Project] [Video]

Sine Activated Low-Rank Matrices for Parameter Efficient Learning
Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, Simon Lucey
arXiv preprint, 28 Mar 2024
[arXiv]

Neural Radiance Fields with Torch Units
Bingnan Ni, Huanyu Wang, Dongfeng Bai, Minghe Weng, Dexin Qi, Weichao Qiu, Bingbing Liu
arXiv preprint, 3 Apr 2024
[arXiv]

Alpha Invariance: On Inverse Scaling Between Distance and Volume Density in Neural Radiance Fields
Joshua Ahn, Haochen Wang, Raymond A. Yeh, Greg Shakhnarovich
CVPR 2024, 2 Apr 2024
[arXiv] [Project]

RaFE: Generative Radiance Fields Restoration
Zhongkai Wu, Ziyu Wan, Jing Zhang, Jing Liao, Dong Xu
arXiv preprint, 4 Apr 2024
[arXiv] [Project] [Code] [Video]

Bayesian NeRF: Quantifying Uncertainty with Volume Density in Neural Radiance Fields
Sibeak Lee, Kyeongsu Kang, Hyeonwoo Yu
arXiv preprint, 10 Apr 2024
[arXiv]

:fire:Rip-NeRF: Anti-aliasing Radiance Fields with Ripmap-Encoded Platonic Solids
Junchen Liu, Wenbo Hu, Zhuo Yang, Jianteng Chen, Guoliang Wang, Xiaoxue Chen, Yantong Cai, Huan-ang Gao, Hao Zhao
SIGGRAPH 2024, 3 May 2024

Abstract

Despite significant advancements in Neural Radiance Fields (NeRFs), the renderings may still suffer from aliasing and blurring artifacts, since it remains a fundamental challenge to effectively and efficiently characterize anisotropic areas induced by the cone-casting procedure. This paper introduces a Ripmap-Encoded Platonic Solid representation to precisely and efficiently featurize 3D anisotropic areas, achieving high-fidelity anti-aliasing renderings. Central to our approach are two key components: Platonic Solid Projection and Ripmap encoding. The Platonic Solid Projection factorizes the 3D space onto the unparalleled faces of a certain Platonic solid, such that the anisotropic 3D areas can be projected onto planes with distinguishable characterization. Meanwhile, each face of the Platonic solid is encoded by the Ripmap encoding, which is constructed by anisotropically pre-filtering a learnable feature grid, to enable featurzing the projected anisotropic areas both precisely and efficiently by the anisotropic area-sampling. Extensive experiments on both well-established synthetic datasets and a newly captured real-world dataset demonstrate that our Rip-NeRF attains state-of-the-art rendering quality, particularly excelling in the fine details of repetitive structures and textures, while maintaining relatively swift training times.

[arXiv] [Project] [Code] [Video]

DistGrid: Scalable Scene Reconstruction with Distributed Multi-resolution Hash Grid
Sidun Liu, Peng Qiao, Zongxin Ye, Wenyu Li, Yong Dou
Siggraph Asia 2023, 7 May 2024
[arXiv]

NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo
Fotios Logothetis, Ignas Budvytis, Roberto Cipolla
arXiv preprint, 20 May 2024
[arXiv]

PruNeRF: Segment-Centric Dataset Pruning via 3D Spatial Consistency
Yeonsung Jung, Heecheol Yun, Joonhyung Park, Jin-Hwa Kim, Eunho Yang
arXiv preprint, 2 Jun 2024
[arXiv]

NeRF Director: Revisiting View Selection in Neural Volume Rendering
Wenhui Xiao, Rodrigo Santa Cruz, David Ahmedt-Aristizabal, Olivier Salvado, Clinton Fookes, Leo Lebrat
CVPR 2024, 13 Jun 2024
[arXiv]

InterNeRF: Scaling Radiance Fields via Parameter Interpolation
Clinton Wang, Peter Hedman, Polina Golland, Jonathan T. Barron, Daniel Duckworth
CVPR 2024 Neural Rendering Intelligence Workshop, 17 Jun 2024
[arXiv]

Uncertainty modeling for fine-tuned implicit functions
Anna Susmelj, Mael Macuglia, Nataša Tagasovska, Reto Sutter, Sebastiano Caprara, Jean-Philippe Thiran, Ender Konukoglu
arXiv preprint, 17 Jun 2024
[arXiv]

Matching Query Image Against Selected NeRF Feature for Efficient and Scalable Localization
Huaiji Zhou, Bing Wang, Changhao Chen
arXiv preprint, 17 Jun 2024
[arXiv]

Federated Neural Radiance Field for Distributed Intelligence
Yintian Zhang, Ziyu Shao
arXiv preprint, 15 Jun 2024
[arXiv]

Drantal-NeRF: Diffusion-Based Restoration for Anti-aliasing Neural Radiance Field
Ganlin Yang, Kaidong Zhang, Jingjing Fu, Dong Liu
arXiv preprint, 10 Jul 2024
[arXiv]

RS-NeRF: Neural Radiance Fields from Rolling Shutter Images
Muyao Niu, Tong Chen, Yifan Zhan, Zhuoxiao Li, Xiang Ji, Yinqiang Zheng
ECCV 2024, 14 Jul 2024
[arXiv] [Code]

Efficient NeRF Optimization -- Not All Samples Remain Equally Hard
Juuso Korhonen, Goutham Rangu, Hamed R. Tavakoli, Juho Kannala
arXiv preprint, 6 Aug 2024
[arXiv]

Magnituder Layers for Implicit Neural Representations in 3D
Sang Min Kim, Byeongchan Kim, Arijit Sehanobish Krzysztof Choromanski, Dongseok Shim, Avinava Dubey, Min-hwan Oh
arXiv preprint, 13 Oct 2024
[arXiv]

Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder
Antoine Schnepf, Karim Kassab, Jean-Yves Franceschi, Laurent Caraffa, Flavian Vasile, Jeremie Mary, Andrew Comport, Valerie Gouet-Brunet
arXiv preprint, 30 Oct 2024
[arXiv] [Project]

Depth Supervised Reconstruction

:fire:Depth-supervised NeRF: Fewer Views and Faster Training for Free
Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan
CVPR 2022, 6 Jul 2021

Abstract

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.

[arXiv] [Project] [Github]

:fire:NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou
ICCV 2021, 2 Sep 2021

Abstract

In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM+MVS reconstruction from COLMAP. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at this https URL.

[arXiv] [Project] [Github]

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views
Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, Matthias Nießner
CVPR 2022, 6 Dec 2021
[arXiv] [Project] [Github]

RobustNeRF: Ignoring Distractors with Robust Losses
Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J. Fleet, Andrea Tagliasacchi
arXiv preprint, 2 Feb 2023
[arXiv] [Project]

Digging into Depth Priors for Outdoor Neural Radiance Fields
Chen Wang, Jiadai Sun, Lina Liu, Chenming Wu, Zhelun Shen, Dayan Wu, Yuchao Dai, Liangjun Zhang
ACMMM 2023, 8 Aug 2023
[arXiv] [Project]

AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization
Kun Wang, Zhiqiang Yan, Huang Tian, Zhenyu Zhang, Xiang Li, Jun Li, Jian Yang
AAAI 2024, 19 Aug 2023
[arXiv]

Depth Supervised Neural Surface Reconstruction from Airborne Imagery
Vincent Hackstein, Paul Fauth-Mayer, Matthias Rothermel, Norbert Haala
arXiv preprint, 25 Apr 2024
[arXiv]

Activation Function Optimization

:fire:Implicit Neural Representations with Periodic Activation Functions
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein
NeurIPS 2020, 17 Jun 2020

Abstract

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

[arXiv]

Multiplicative Filter Networks
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, J Zico Kolter
ICLR 2021, 13 Jan 2021
[OpenReview]

PREF: Phasorial Embedding Fields for Compact Neural Representations
Binbin Huang, Xinhao Yan, Anpei Chen, Shenghua Gao, Jingyi Yu
arXiv preprint, 26 May 2022
[avXiv]

Positional Encoding

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng
arXiv preprint, 18 Jun 2020
[arXiv]

BACON: Band-limited Coordinate Networks for Multiscale Scene Representation
David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein
CVPR 2022, 9 Dec 2021
[arXiv] [Project]

Improved Implicity Neural Representation with Fourier Bases Reparameterized Training
Kexuan Shi, Xingyu Zhou, Shuhang Gu
arXiv preprint, 15 Jan 2024
[arXiv]

Deformable & Dynamic NeRF

:fire:Nerfies: Deformable Neural Radiance Fields
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla
ICCV 2021, 25 Nov 2020

Abstract

We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.

[arXiv] [Project] [Github]

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Zhengqi Li, Simon Niklaus, Noah Snavely, Oliver Wang
CVPR 2021, 26 Nov 2020
[arXiv] [Project] [Github]

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, Steven M. Seitz
CVPR 2021, 24 Jun 2021
[arXiv] [Project] [Github]

CodeNeRF: Disentangled Neural Radiance Fields for Object Categories
Wonbong Jang, Lourdes Agapito
ICCV 2021, 3 Sep 2021
[arXiv] [Project] [Github]

Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, Thomas Funkhouser
CVPR 2022, 9 May 2022
[arXiv] [Project]

D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, Cengiz Oztireli
CVPR 2022, 31 May 2022
[arXiv] [Project] [Github]

:fire:Deforming Radiance Fields with Cages
Tianhan Xu, Tatsuya Harada
ECCV 2022, 25 Jul 2022

Abstract

Recent advances in radiance fields enable photorealistic rendering of static or dynamic 3D scenes, but still do not support explicit deformation that is used for scene manipulation or animation. In this paper, we propose a method that enables a new type of deformation of the radiance field: free-form radiance field deformation. We use a triangular mesh that encloses the foreground object called cage as an interface, and by manipulating the cage vertices, our approach enables the free-form deformation of the radiance field. The core of our approach is cage-based deformation which is commonly used in mesh deformation. We propose a novel formulation to extend it to the radiance field, which maps the position and the view direction of the sampling points from the deformed space to the canonical space, thus enabling the rendering of the deformed scene. The deformation results of the synthetic datasets and the real-world datasets demonstrate the effectiveness of our approach.

[arXiv] [Project] [Github]

:fire:NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, Andreas Geiger
IEEE TVCG, 28 Oct 2022

Abstract

Visually exploring in a real-world 4D spatiotemporal space freely in VR has been a long-term quest. The task is especially appealing when only a few or even single RGB cameras are used for capturing the dynamic scene. To this end, we present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. First, we propose to decompose the 4D spatiotemporal space according to temporal characteristics. Points in the 4D space are associated with probabilities of belonging to three categories: static, deforming, and new areas. Each area is represented and regularized by a separate neural field. Second, we propose a hybrid representations based feature streaming scheme for efficiently modeling the neural fields. Our approach, coined NeRFPlayer, is evaluated on dynamic scenes captured by single hand-held cameras and multi-camera arrays, achieving comparable or superior rendering performance in terms of quality and speed comparable to recent state-of-the-art methods, achieving reconstruction in 10 seconds per frame and interactive rendering.

[arXiv] [Project] [Github(in nerfstudio)]

Robust Dynamic Radiance Fields
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, Jia-Bin Huang
CVPR 2023, 5 Jan 2023
[arXiv] [Project]

:fire:HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, Changil Kim
arXiv preprint, 5 Jan 2023

Abstract

Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.

[arXiv] [Project] [Github]

CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Yicong Peng, Yichao Yan, Shengqi Liu, Yuhao Cheng, Shanyan Guan, Bowen Pan, Guangtao Zhai, Xiaokang Yang
NeurIPS 2022, 01 Nov 2022
[OpenReview] [Zhihu] [Github]

DynIBaR: Neural Dynamic Image-Based Rendering
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, Noah Snavely
CVPR 2023, 20 Nov 2022
[arXiv] [Project] [Github] [Video]

MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videos
Fengrui Tian, Shaoyi Du, Yueqi Duan
ICCV 2023, 26 Dec 2022
[arXiv] [Github]

:fire:HexPlane: A Fast Representation for Dynamic Scenes
Ang Cao, Justin Johnson
CVPR 2023, 23 Jan 2023

Abstract

Modeling and re-rendering dynamic 3D scenes is a challenging task in 3D vision. Prior approaches build on NeRF and rely on implicit representations. This is slow since it requires many MLP evaluations, constraining real-world applications. We show that dynamic 3D scenes can be explicitly represented by six planes of learned features, leading to an elegant solution we call HexPlane. A HexPlane computes features for points in spacetime by fusing vectors extracted from each plane, which is highly efficient. Pairing a HexPlane with a tiny MLP to regress output colors and training via volume rendering gives impressive results for novel view synthesis on dynamic scenes, matching the image quality of prior work but reducing training time by more than 100×. Extensive ablations confirm our HexPlane design and show that it is robust to different feature fusion mechanisms, coordinate systems, and decoding mechanisms. HexPlane is a simple and effective solution for representing 4D volumes, and we hope they can broadly contribute to modeling spacetime for dynamic 3D scenes.

[arXiv] [Project] [Github]

:fire:K-Planes: Explicit Radiance Fields in Space, Time, and Appearance
Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, Angjoo Kanazawa
CVPR 2023, 24 Jan 2023

Abstract

We introduce k-planes, a white-box model for radiance fields in arbitrary dimensions. Our model uses d choose 2 planes to represent a d-dimensional scene, providing a seamless way to go from static (d=3) to dynamic (d=4) scenes. This planar factorization makes adding dimension-specific priors easy, e.g. temporal smoothness and multi-resolution spatial structure, and induces a natural decomposition of static and dynamic components of a scene. We use a linear feature decoder with a learned color basis that yields similar performance as a nonlinear black-box MLP decoder. Across a range of synthetic and real, static and dynamic, fixed and varying appearance scenes, k-planes yields competitive and often state-of-the-art reconstruction fidelity with low memory usage, achieving 1000x compression over a full 4D grid, and fast optimization with a pure PyTorch implementation. For video results and code, please see this https URL.

[arXiv] [Project] [Github] [Notes]

PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, Chuang Gan
ICLR 2023, 02 Feb 2023
[OpenReview] [Project] [Github]

Temporal Interpolation Is All You Need for Dynamic Neural Radiance Fields
Sungheon Park, Minjung Son, Seokhwan Jang, Young Chun Ahn, Ji-Yeon Kim, Nahyup Kang
CVPR 2023, 18 Feb 2023
[arXiv] [Project] [Video]

OD-NeRF: Efficient Training of On-the-Fly Dynamic Neural Radiance Fields
Zhiwen Yan, Chen Li, Gim Hee Lee
arXiv preprint, 24 May 2023
[arXiv]

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
Authors: Edith Tretschk, Vladislav Golyanik, Michael Zollhoefer, Aljaz Bozic, Christoph Lassner, Christian Theobalt
arXiv preprint, 16 August, 2023
[arXiv] [Project] [Video]

ResFields: Residual Neural Fields for Spatiotemporal Signals
Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, Siyu Tang
arXiv preprint, 6 Sep 2023
[arXiv] [Project] [Github]

Dynamic Mesh-Aware Radiance Fields
Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, Ming C. Lin
ICCV 2023, 8 Sep 2023
[arXiv] [Project] [Github]

DynaMoN: Motion-Aware Fast And Robust Camera Localization for Dynamic NeRF
Mert Asim Karaoglu, Hannah Schieber, Nicolas Schischka, Melih Görgülü, Florian Grötzner, Alexander Ladikos, Daniel Roth, Nassir Navab, Benjamin Busam
arXiv preprint, 16 Sep 2023
[arXiv]

Point-DynRF: Point-based Dynamic Radiance Fields from a Monocular Video
Byeongjun Park, Changick Kim
WACV 2024, 14 Oct 2023
[arXiv]

Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos
Seoha Kim, Jeongmin Bae, Youngsik Yun, Hahyun Lee, Gun Bang, Youngjung Uh
arXiv preprint, 20 Oct 2023
[arXiv] [Project]

DreaMo: Articulated 3D Reconstruction From A Single Casual Video
Tao Tu, Ming-Feng Li, Chieh Hubert Lin, Yen-Chi Cheng, Min Sun, Ming-Hsuan Yang
arXiv preprint, 5 Dec, 2023
[arXiv] [Project]

NeVRF: Neural Video-based Radiance Fields for Long-duration Sequences
Minye Wu, Tinne Tuytelaars
arXiv preprint, 10 Dec 2023
[arXiv]

Fast High Dynamic Range Radiance Fields for Dynamic Scenes
Guanjun Wu, Taoran Yi, Jiemin Fang, Wenyu Liu, Xinggang Wang
3DV 2024, 11 Jan 2024
[arXiv] [Project] [Code]

DaReNeRF: Direction-aware Representation for Dynamic Scenes
Ange Lou, Benjamin Planche, Zhongpai Gao, Yamin Li, Tianyu Luan, Hao Ding, Terrence Chen, Jack Noble, Ziyan Wu
CVPR 2024, 4 Mar 2024
[arXiv]

S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes
Xingyi Li, Zhiguo Cao, Yizheng Wu, Kewei Wang, Ke Xian, Zhe Wang, Guosheng Lin
CVPR 2024, 10 Mar 2024
[arXiv] [Project] [Code]

NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation
Jiahao Chen, Yipeng Qin, Lingjie Liu, Jiangbo Lu, Guanbin Li
CVPR 2024, 26 Mar 2024
[arXiv] [Project] [Code]

LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis
Zehan Zheng, Fan Lu, Weiyi Xue, Guang Chen, Changjun Jiang
CVPR 2024, 3 Apr 2024
[arXiv] [Project] [Code]

TK-Planes: Tiered K-Planes with High Dimensional Feature Vectors for Dynamic UAV-based Scenes
Christopher Maxey, Jaehoon Choi, Yonghan Lee, Hyungtae Lee, Dinesh Manocha, Heesung Kwon
IROS2024, 4 May 2024
[arXiv]

JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression
Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang
arXiv preprint, 23 May 2024
[arXiv]

Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimization
Takuhiro Kaneko
CVPR 2024, 6 Jun 2024
[arXiv] [Project] [Video]

TutteNet: Injective 3D Deformations by Composition of 2D Mesh Deformations
Bo Sun, Thibault Groueix, Chen Song, Qixing Huang, Noam Aigerman
arXiv preprint, 17 Jun 2024
[arXiv]

Dynamic Neural Radiance Field From Defocused Monocular Video
Xianrui Luo, Huiqiang Sun, Juewen Peng, Zhiguo Cao
ECCV 2024, 8 Jul 2024
[arXiv]

KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter
Yifan Zhan, Zhuoxiao Li, Muyao Niu, Zhihang Zhong, Shohei Nobuhara, Ko Nishino, Yinqiang Zheng
ECCV 2024, 18 Jul 2024
[arXiv]

TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene
Sandika Biswas, Qianyi Wu, Biplab Banerjee, Hamid Rezatofighi
NeurIPS 2024, 26 Sep 2024
[arXiv]

Deformable NeRF using Recursively Subdivided Tetrahedra
Zherui Qiu, Chenqu Ren, Kaiwen Song, Xiaoyi Zeng, Leyuan Yang, Juyong Zhang
ACM MM 2024, 6 Oct 2024
[arXiv] [Project]

NeRF Training and Rendering Speed Enhancements

:fire:Neural Sparse Voxel Fields
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt
NeurIPS 2020, 22 Jul 2020

Abstract

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a differentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering. Code and data are available at our website: this https URL.

[arXiv]

:fire:NeRF++: Analyzing and Improving Neural Radiance Fields
Kai Zhang, Gernot Riegler, Noah Snavely, Vladlen Koltun
arXiv Preprint 2020, 15 Oct 2020

Abstract

Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360 capture of bounded scenes and forward-facing capture of bounded and unbounded scenes. NeRF fits multi-layer perceptrons (MLPs) representing view-invariant opacity and view-dependent color volumes to a set of training images, and samples novel views based on volume rendering techniques. In this technical report, we first remark on radiance fields and their potential ambiguities, namely the shape-radiance ambiguity, and analyze NeRF's success in avoiding such ambiguities. Second, we address a parametrization issue involved in applying NeRF to 360 captures of objects within large-scale, unbounded 3D scenes. Our method improves view synthesis fidelity in this challenging scenario. Code is available at this https URL.

[arXiv] [Github]

DeRF: Decomposed Radiance Fields
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, Andrea Tagliasacchi
CVPR 2021, 25 Nov 2020
[arXiv] [Project] [Github]

:fire:AutoInt: Automatic Integration for Fast Neural Volume Rendering
David B. Lindell, Julien N. P. Martel, Gordon Wetzstein
CVPR 2021, 15 Oct 2020

Abstract

Numerical integration is a foundational technique in scientific computing and is at the core of many computer vision applications. Among these applications, neural volume rendering has recently been proposed as a new paradigm for view synthesis, achieving photorealistic image quality. However, a fundamental obstacle to making these methods practical is the extreme computational and memory requirements caused by the required volume integrations along the rendered rays during training and inference. Millions of rays, each requiring hundreds of forward passes through a neural network are needed to approximate those integrations with Monte Carlo sampling. Here, we propose automatic integration, a new framework for learning efficient, closed-form solutions to integrals using coordinate-based neural networks. For training, we instantiate the computational graph corresponding to the derivative of the network. The graph is fitted to the signal to integrate. After optimization, we reassemble the graph to obtain a network that represents the antiderivative. By the fundamental theorem of calculus, this enables the calculation of any definite integral in two evaluations of the network. Applying this approach to neural rendering, we improve a tradeoff between rendering speed and image quality: improving render times by greater than 10 times with a tradeoff of slightly reduced image quality.

[arXiv] [Github]

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton Kaplanyan, Markus Steinberger
EGSR 2021, 4 Mar 2021
[arXiv] [Project] [Github]

FastNeRF: High-Fidelity Neural Rendering at 200FPS
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, Julien Valentin
ICCV 2021, 18 Mar 2021
[arXiv] [Project]

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger
ICCV 2021, 25 Mar 2021
[arXiv] [Github]

:fire:PlenOctrees for Real-time Rendering of Neural Radiance Fields
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa
ICCV 2021, 25 Mar 2021

Abstract

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: this https URL

[arXiv] [Project] [Github] [Viewer Github]

:fire:Baking Neural Radiance Fields for Real-Time View Synthesis
Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul Debevec
ICCV 2021, 26 Mar 2021

Abstract

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF's computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables real-time rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF's architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF's ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.

[arXiv] [Project] [Github]

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Cheng Sun, Min Sun, Hwann-Tzong Chen
CVPR 2022, 22 Nov 2021
[arXiv] [Project] [Github]

VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field
Naruya Kondo, Yuya Ikeda, Andrea Tagliasacchi, Yutaka Matsuo, Yoichi Ochiai, Shixiang Shane Gu
arXiv preprint, 25 Nov 2021
[arXiv] [Github]

NeuSample: Neural Sample Field for Efficient View Synthesis
Naruya Kondo, Yuya Ikeda, Andrea Tagliasacchi, Yutaka Matsuo, Yoichi Ochiai, Shixiang Shane Gu
arXiv preprint, 30 Nov 2021
[arXiv] [Project] [Github]

:fire:Plenoxels: Radiance Fields without Neural Networks
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa
arXiv preprint, 30 Nov 2021

Abstract

We introduce Plenoxels (plenoptic voxels), a system for photorealistic view synthesis. Plenoxels represent a scene as a sparse 3D grid with spherical harmonics. This representation can be optimized from calibrated images via gradient methods and regularization without any neural components. On standard, benchmark tasks, Plenoxels are optimized two orders of magnitude faster than Neural Radiance Fields with no loss in visual quality.

[arXiv] [Project] [Github]

:fire:Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller
SIGGRAPH 2022, 16 Jan 2022

Abstract

Neural graphics primitives, parameterized by fully connected neural networks, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA kernels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of 1920×1080.

[arXiv] [Project] [Github]

:fire:TensoRF: Tensorial Radiance Fields
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, Hao Su
ECCV 2022, 17 Mar 2022

Abstract

We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).

[arXiv] [Project] [Github] [Notes]

SqueezeNeRF: Further factorized FastNeRF for memory-efficient inference
Krishna Wadhwani, Tamaki Kojima
arXiv preprint, 6 Apr 2022
[arXiv]

AdaNeRF: Adaptive Sampling for Real-Time Rendering of Neural Radiance Fields
Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, Markus Steinberger
ECCV 2022, 21 Jul 2022
[arXiv] [Project] [Github]

:fire:MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, Andrea Tagliasacchi
CVPR 2023, 30 Jul 2022

Abstract

Neural Radiance Fields (NeRFs) have demonstrated amazing ability to synthesize images of 3D scenes from novel views. However, they rely upon specialized volumetric rendering algorithms based on ray marching that are mismatched to the capabilities of widely deployed graphics hardware. This paper introduces a new NeRF representation based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors. Traditional rendering of the polygons with a z-buffer yields an image with features at every pixel, which are interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, achieving interactive frame rates on a wide range of compute platforms, including mobile phones.

[arXiv] [Project] [Github]

Real-Time Neural Light Field on Mobile Devices
Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, Jian Ren
CVPR 2023, 15 Dec 2022
[arXiv] [Project] [Github]

Factor Fields: A Unified Framework for Neural Fields and Beyond Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, Andreas Geiger
arXiv preprint, 2 Feb 2023
[arXiv] [Project] [Github]

:fire:MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes
Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, Peter Hedman
arXiv preprint, 23 Feb 2023

Abstract

Neural radiance fields enable state-of-the-art photorealistic view synthesis. However, existing radiance field representations are either too compute-intensive for real-time rendering or require too much memory to scale to large scenes. We present a Memory-Efficient Radiance Field (MERF) representation that achieves real-time rendering of large-scale scenes in a browser. MERF reduces the memory consumption of prior sparse volumetric radiance fields using a combination of a sparse feature grid and high-resolution 2D feature planes. To support large-scale unbounded scenes, we introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection. We design a lossless procedure for baking the parameterization used during training into a model that achieves real-time rendering while still preserving the photorealistic view synthesis quality of a volumetric radiance field.

[arXiv] [Project]

:fire:BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis
Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, Ben Mildenhall
arXiv preprint, 28 Feb 2023

Abstract

We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.

[arXiv] [Project]

Volume Feature Rendering for Fast Neural Radiance Field Reconstruction
Kang Han, Wei Xiang, Lu Yu
arXiv preprint, 29 May 2023
[arXiv]

Compact Real-time Radiance Fields with Neural Codebook
Lingzhi Li, Zhongshu Wang, Zhen Shen, Li Shen, Ping Tan
ICME 2023, 29 May 2023
[arXiv]

Dictionary Fields: Learning a Neural Basis Decomposition
Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, Andreas Geiger
SIGGRAPH 2023
[Paper] [Github]

NAS-NeRF: Generative Neural Architecture Search for Neural Radiance Fields
NAS-NeRF: Generative Neural Architecture Search for Neural Radiance Fields
arXiv preprint, 25 Sep 2023
[arXiv] [Project]

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering
Tong Wang, Shuichi Kurabayashi
arXiv preprint, 3 Oct 2023
[arXiv]

MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields
Takuhiro Kaneko
ICCV 2023, 3 Oct 2023
[arXiv] [Project]

Neural Processing of Tri-Plane Hybrid Neural Fields
Adriano Cardace, Pierluigi Zama Ramirez, Francesco Ballerini, Allan Zhou, Samuele Salti, Luigi Di Stefano
arXiv preprint, 2 Oct 2023
[arXiv]

CAwa-NeRF: Instant Learning of Compression-Aware NeRF Features
Omnia Mahmoud, Théo Ladune, Matthieu Gendrin
arXiv preprint, 23 Oct 2023
[arXiv]

Efficient Encoding of Graphics Primitives with Simplex-based Structures
Yibo Wen, Yunfan Yang
arXiv preprint, 26 Nov 2023
[arXiv]

ProNeRF: Learning Efficient Projection-Aware Ray Sampling for Fine-Grained Implicit Neural Radiance Fields
Juan Luis Gonzalez Bello, Minh-Quan Viet Bui, Munchurl Kim
arXiv preprint, 13 Dec 2023
[arXiv] [Project] [Code]

HyperPlanes: Hypernetwork Approach to Rapid NeRF Adaptation
Paweł Batorski, Dawid Malarz, Marcin Przewięźlikowski, Marcin Mazur, Sławomir Tadeja, Przemysław Spurek
arXiv preprint, 2 Feb 2024
[arXiv]

Preconditioners for the Stochastic Training of Implicit Neural Representations
Shin-Fang Chng, Hemanth Saratchandran, Simon Lucey
arXiv preprint, 13 Feb 2024
[arXiv]

Improved Generalization of Weight Space Networks via Augmentations
Aviv Shamsian, Aviv Navon, David W. Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, Haggai Maron
arXiv preprint, 6 Feb 2024
[arXiv]

Vosh: Voxel-Mesh Hybrid Representation for Real-Time View Synthesis
Chenhao Zhang, Yongyang Zhou, Lei Zhang
arXiv preprint, 11 Mar 2024
[arXiv]

Plug-and-Play Acceleration of Occupancy Grid-based NeRF Rendering using VDB Grid and Hierarchical Ray Traversal
Yoshio Kato, Shuhei Tarashima
CVPR Neural Rendering Intelligence Workshop 2024, 16 Apr 2024
[arXiv] [Code]

Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations
Yu Feng, Zihan Liu, Jingwen Leng, Minyi Guo, Yuhao Zhu
arXiv preprint, 18 Apr 2024
[arXiv]

:fire:NeRF-XL: Scaling NeRFs with Multiple GPUs
Ruilong Li, Sanja Fidler, Angjoo Kanazawa, Francis Williams
arXiv preprint, 24 Apr 2024

Abstract

We present NeRF-XL, a principled method for distributing Neural Radiance Fields (NeRFs) across multiple GPUs, thus enabling the training and rendering of NeRFs with an arbitrarily large capacity. We begin by revisiting existing multi-GPU approaches, which decompose large scenes into multiple independently trained NeRFs, and identify several fundamental issues with these methods that hinder improvements in reconstruction quality as additional computational resources (GPUs) are used in training. NeRF-XL remedies these issues and enables the training and rendering of NeRFs with an arbitrary number of parameters by simply using more hardware. At the core of our method lies a novel distributed training and rendering formulation, which is mathematically equivalent to the classic single-GPU case and minimizes communication between GPUs. By unlocking NeRFs with arbitrarily large parameter counts, our approach is the first to reveal multi-GPU scaling laws for NeRFs, showing improvements in reconstruction quality with larger parameter counts and speed improvements with more GPUs. We demonstrate the effectiveness of NeRF-XL on a wide variety of datasets, including the largest open-source dataset to date, MatrixCity, containing 258K images covering a 25km^2 city area.

[arXiv] [Project]

Towards Real-Time Neural Volumetric Rendering on Mobile Devices: A Measurement Study
Zhe Wang, Yifei Zhu
ACM SIGCOMM Workshop on Emerging Multimedia Systems 2024, 23 Jun 2024
[arXiv]

NGP-RT: Fusing Multi-Level Hash Features with Lightweight Attention for Real-Time Novel View Synthesis
Yubin Hu, Xiaoyang Guo, Yang Xiao, Jingwei Huang, Yong-Jin Liu
ECCV 2024, 15 Jul 2024
[arXiv]

Boost Your NeRF: A Model-Agnostic Mixture of Experts Framework for High Quality and Efficient Rendering
Francesco Di Sario, Riccardo Renzulli, Enzo Tartaglione, Marco Grangetto
arXiv preprint, 15 Jul 2024
[arXiv]

Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture
Yu Feng, Weikai Lin, Zihan Liu, Jingwen Leng, Minyi Guo, Han Zhao, Xiaofeng Hou, Jieru Zhao, Yuhao Zhu
arXiv preprint, 13 Aug 2024
[arXiv]

Expansive Supervision for Neural Radiance Field
Weixiang Zhang, Shuzhao Xie, Shijia Ge, Wei Yao, Chen Tang, Zhi Wang
arXiv preprint, 12 Sep 2024
[arXiv]

One-Shot/Few-Shot Sparse View Reconstruction

:fire:pixelNeRF: Neural Radiance Fields from One or Few Images
Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa
CVPR 2021, 3 Dec 2020

Abstract

We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects as well as entire unseen categories. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes and real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction. For the video and code, please visit the project website: this https URL

[arXiv] [Project] [Github]

IBRNet: Learning Multi-View Image-Based Rendering
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser
CVPR 2021, 25 Feb 2021
[arXiv] [Project] [Github]

:fire:Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
Ajay Jain, Matthew Tancik, Pieter Abbeel
arXiv preprint, 1 Apr 2021

Abstract

We present DietNeRF, a 3D neural scene representation estimated from a few images. Neural Radiance Fields (NeRF) learn a continuous volumetric representation of a scene through multi-view consistency, and can be rendered from novel viewpoints by ray casting. While NeRF has an impressive ability to reconstruct geometry and fine details given many images, up to 100 for challenging 360° scenes, it often finds a degenerate solution to its image reconstruction objective when only a few input views are available. To improve few-shot quality, we propose DietNeRF. We introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses. DietNeRF is trained on individual scenes to (1) correctly render given input views from the same pose, and (2) match high-level semantic attributes across different, random poses. Our semantic loss allows us to supervise DietNeRF from arbitrary poses. We extract these semantics using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse single-view, 2D photographs mined from the web with natural language supervision. In experiments, DietNeRF improves the perceptual quality of few-shot view synthesis when learned from scratch, can render novel views with as few as one observed image when pre-trained on a multi-view dataset, and produces plausible completions of completely unobserved regions.

[arXiv] [Project] [Github]

MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis
Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, Gim Hee Lee
ICCV 2021, 27 Mar 2021
[arXiv] [Project] [Github]

:fire:MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su
ICCV 2021, 29 Mar 2021

Abstract

We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. We train our network on real objects in the DTU dataset, and test it on three different datasets to evaluate its effectiveness and generalizability. Our approach can generalize across scenes (even indoor scenes, completely different from our training scenes of objects) and generate realistic view synthesis results using only three input images, significantly outperforming concurrent works on generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

[arXiv] [Project] [Github]

Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, David Novotny
ICCV 2021, 29 Mar 2021
[arXiv] [Co3D Dataset]

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs
Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, Noha Radwan
CVPR 2022, 1 Dec 2021
[arXiv] [Project] [Code] [Notes]

:fire:ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers
Jonáš Kulhánek, Erik Derner, Torsten Sattler, Robert Babuška
ECCV 2022, 18 Mar 2022

Abstract

Novel view synthesis is a long-standing problem. In this work, we consider a variant of the problem where we are given only a few context views sparsely covering a scene or an object. The goal is to predict novel viewpoints in the scene, which requires learning priors. The current state of the art is based on Neural Radiance Field (NeRF), and while achieving impressive results, the methods suffer from long training times as they require evaluating millions of 3D point samples via a neural network for each image. We propose a 2D-only method that maps multiple context views and a query pose to a new image in a single pass of a neural network. Our model uses a two-stage architecture consisting of a codebook and a transformer model. The codebook is used to embed individual images into a smaller latent space, and the transformer solves the view synthesis task in this more compact space. To train our model efficiently, we introduce a novel branching attention mechanism that allows us to use the same model not only for neural rendering but also for camera pose estimation. Experimental results on real-world scenes show that our approach is competitive compared to NeRF-based methods while not reasoning explicitly in 3D, and it is faster to train.

[arXiv] [Project] [Github]

S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong
NeurIPS 2022, 17 Oct 2022
[arXiv] [Project] [Github]

NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong
arXiv preprint, 29 Nov 2022
[arXiv] [Project] [Github]

SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
Abdullah Hamdi, Bernard Ghanem, Matthias Nießner
arXiv preprint, 18 Dec 2022
[arXiv] [Project] [Github]

Geometry-biased Transformers for Novel View Synthesis
Naveen Venkat, Mayank Agarwal, Maneesh Singh, Shubham Tulsiani
arXiv preprint, 11 Jan 2023
[arXiv] [Project] [Github]

Behind the Scenes: Density Fields for Single View Reconstruction
Felix Wimbauer, Nan Yang, Christian Rupprecht, Daniel Cremers
CVPR 2023, 18 Jan 2023
[arXiv] [Project] [Github] [Video]

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, Ravi Ramamoorthi
arXiv preprint, 20 Feb 2023
[arXiv] [Project]

DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models
Jamie Wynn, Daniyar Turmukhambetov
arXiv preprint, 23 Feb 2023
[arXiv] [Github]

:fire:FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization
Jiawei Yang, Marco Pavone, Yue Wang
CVPR 2023, 13 Mar 2023

Abstract

Novel view synthesis with sparse inputs is a challenging problem for neural radiance fields (NeRF). Recent efforts alleviate this challenge by introducing external supervision, such as pre-trained models and extra depth signals, and by non-trivial patch-based rendering. In this paper, we present Frequency regularized NeRF (FreeNeRF), a surprisingly simple baseline that outperforms previous methods with minimal modifications to the plain NeRF. We analyze the key challenges in few-shot neural rendering and find that frequency plays an important role in NeRF's training. Based on the analysis, we propose two regularization terms. One is to regularize the frequency range of NeRF's inputs, while the other is to penalize the near-camera density fields. Both techniques are ``free lunches'' at no additional computational cost. We demonstrate that even with one line of code change, the original NeRF can achieve similar performance as other complicated methods in the few-shot setting. FreeNeRF achieves state-of-the-art performance across diverse datasets, including Blender, DTU, and LLFF. We hope this simple baseline will motivate a rethinking of the fundamental role of frequency in NeRF's training under the low-data regime and beyond.

[arXiv] [Project] [Github] [Notes]

Zero-1-to-3: Zero-shot One Image to 3D Object
Jiawei Yang, Marco Pavone, Yue Wang
CVPR 2023, 20 Mar 2023
[arXiv] [Project] [Github]

SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, Ziwei Liu
ICCV 2023, 28 Mar 2023
[arXiv] [Project] [Github]

VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs
Jiakai Sun, Zhanjie Zhang, Jiafu Chen, Guangyuan Li, Boyan Ji, Lei Zhao, Wei Xing
IJCAI 2023, 26 Apr 2023
[arXiv] [Github]

ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields
Nagabhushan Somraj, Rajiv Soundararajan
SIGGRAPH 2023, 28 Apr 2023
[arXiv] [Project] [Github] [Video]

DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation
Jiuhn Song, Seonghoon Park, Honggyu An, Seokju Cho, Min-Seop Kwak, Sungjin Cho, Seungryong Kim
arXiv preprint, 30 May 2023
[arXiv] [Project]

ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields
Kanghyeok Ko, Minhyeok Lee
arXiv preprint, 5 Jun 2023
[arXiv]

Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images
Tianyu Liu, Hao Zhao, Yang Yu, Guyue Zhou, Ming Liu
IEEE RA-L, 26 Jul, 2023
[arXiv] [Project]

Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs
Yanqi Bao, Yuxin Li, Jing Huo, Tianyu Ding, Xinyue Liang, Wenbin Li, Yang Gao
ACMMM 2023, 5 Aug 2023
[arXiv] [Github]

Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views
Wentian Qu, Zhaopeng Cui, Yinda Zhang, Chenyu Meng, Cuixia Ma, Xiaoming Deng, Hongan Wang
arXiv preprint, 22 Aug 2023
[arXiv] [Project]

PERF: Panoramic Neural Radiance Field from a Single Panorama
Guangcong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang, Chen Change Loy, Ziwei Liu
arXir preprint, 25 Oct 2023
[arXiv] [Project] [Github]

ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields
Daiju Kanaoka, Motoharu Sonogashira, Hakaru Tamukoh, Yasutomo Kawanishi
BMVC2023, 20 Oct 2023
[arXiv] [Github]

How Many Views Are Needed to Reconstruct an Unknown Object Using NeRF?
Sicong Pan, Liren Jin, Hao Hu, Marija Popović, Maren Bennewitz
ICRA 2024, 1 Oct 2023
[arXiv]

CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering
Haidong Zhu, Tianyu Ding, Tianyi Chen, Ilya Zharkov, Ram Nevatia, Luming Liang
arXiv preprint, 27 Nov 2023
[arXiv] [Project]

CorresNeRF: Image Correspondence Priors for Neural Radiance Fields
Yixing Lao, Xiaogang Xu, Zhipeng Cai, Xihui Liu, Hengshuang Zhao
NeurIPS 2023, 11 Dec, 2023
[arXiv] [Project] [Code]

Novel View Synthesis with View-Dependent Effects from a Single Image
Juan Luis Gonzalez Bello, Munchurl Kim
arXiv preprint, 13 Dec 2023
[arXiv] [Project]

ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Process
Kiyohiro Nakayama, Mikaela Angelina Uy, Yang You, Ke Li, Leonidas Guibas
arXiv preprint, 16 Jan 2024
[arXiv] [Project]

HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs
Zelin Gao, Weichen Dai, Yu Zhang
arXiv preprint, 22 Jan 2024
[arXiv]

FrameNeRF: A Simple and Efficient Framework for Few-shot Novel View Synthesis
Yan Xing, Pan Wang, Ligang Liu, Daolun Li, Li Zhang
arXiv preprint, 22 Feb, 2024
[arXiv]

CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency
Hanxin Zhu, Tianyu He, Zhibo Chen
IEEE VR 2024, 26 Feb 2024
[arXiv]

DreamUp3D: Object-Centric Generative Models for Single-View 3D Scene Understanding and Real-to-Sim Transfer
Yizhe Wu, Haitz Sáez de Ocáriz Borde, Jack Collins, Oiwi Parker Jones, Ingmar Posner
arXiv preprint, 26 Feb 2024
[arXiv]

Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views
Shuai Guo, Qiuwen Wang, Yijie Gao, Rong Xie, Li Song
arXiv preprint, 4 Mar 2024
[arXiv]

Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis?
Hanxin Zhu, Tianyu He, Xin Li, Bingchen Li, Zhibo Chen
CVPR 2024, 10 Mar 2024
[arXiv]

FSViewFusion: Few-Shots View Generation of Novel Objects
Rukhshanda Hussain, Hui Xian Grace Lim, Borchun Chen, Mubarak Shah, Ser Nam Lim
arXiv preprint, 11 Mar 2024
[arXiv]

CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance Fields from Sparse Inputs
Yingji Zhong, Lanqing Hong, Zhenguo Li, Dan Xu
CVPR 2024, 25 Mar 2024
[arXiv] [Project]

UPNeRF: A Unified Framework for Monocular 3D Object Reconstruction and Pose Estimation
Yuliang Guo, Abhinav Kumar, Cheng Zhao, Ruoyu Wang, Xinyu Huang, Liu Ren
arXiv preprint, 23 Mar 2024
[arXiv]

Stable Surface Regularization for Fast Few-Shot NeRF
Byeongin Joung, Byeong-Uk Lee, Jaesung Choe, Ukcheol Shin, Minjun Kang, Taeyeop Lee, In So Kweon, Kuk-Jin Yoon
3DV 2024, 29 Mar 2024
[arXiv]

SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance
Yuru Xiao, Xianming Liu, Deming Zhai, Kui Jiang, Junjun Jiang, Xiangyang Ji
arXiv preprint, 1 Apr 2024
[arXiv]

Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning
Rui Li, Tobias Fischer, Mattia Segu, Marc Pollefeys, Luc Van Gool, Federico Tombari
CVPR 2024, 4 Apr 2024
[arXiv] [Project] [Code]

Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation
Keonhee Han, Dominik Muhle, Felix Wimbauer, Daniel Cremers
arXiv preprint, 11 Apr 2024
[arXiv]

NeRFG-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images
Zixiong Huang, Qi Chen, Libo Sun, Yifan Yang, Naizhou Wang, Mingkui Tan, Qi Wu
CVPR 2024, 11 Apr 2024
[arXiv]

Simple-RF: Regularizing Sparse Input Radiance Fields with Simpler Solutions
Nagabhushan Somraj, Adithyan Karanayil, Sai Harsha Mupparaju, Rajiv Soundararajan
arXiv preprint, 29 Apr 2024
[arXiv] [Project]

GDGS: Gradient Domain Gaussian Splatting for Sparse Representation of Radiance Fields
Yuanhao Gong
arXiv preprint, 8 May 2024
[arXiv]

Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs
Mingyu Kim, Jun-Seong Kim, Se-Young Yun, Jin-Hwa Kim
ICML 2024, 13 May 2024
[arXiv] [Project] [Code]

Spatial Annealing Smoothing for Efficient Few-shot Neural Rendering
Yuru Xiao, Xianming Liu, Deming Zhai, Kui Jiang, Junjun Jiang, Xiangyang Ji
arXiv preprint, 12 Jun 2024
[arXiv]

M-LRM: Multi-view Large Reconstruction Model
Mengfei Li, Xiaoxiao Long, Yixun Liang, Weiyu Li, Yuan Liu, Peng Li, Xiaowei Chi, Xingqun Qi, Wei Xue, Wenhan Luo, Qifeng Liu, Yike Guo
arXiv preprint, 11 Jun 2024
[arXiv] [Project]

GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement
Peiye Zhuang, Songfang Han, Chaoyang Wang, Aliaksandr Siarohin, Jiaxu Zou, Michael Vasilkovsky, Vladislav Shakhrai, Sergey Korolev, Sergey Tulyakov, Hsin-Ying Lee
arXiv preprint, 9 Jun 2024
[arXiv] [Project]

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views
Jiawei Guo, HungChyun Chou, Ning Ding
arXiv preprint, 8 Jul 2024
[arXiv]

MomentsNeRF: Leveraging Orthogonal Moments for Few-Shot Neural Rendering
Ahmad AlMughrabi, Ricardo Marques, Petia Radeva
arXiv preprint, 2 Jul 2024
[arXiv] [Code]

InfoNorm: Mutual Information Shaping of Normals for Sparse-View Reconstruction
Xulong Wang, Siyan Dong, Youyi Zheng, Yanchao Yang
ECCV 2024, 17 Jul 2024
[arXiv] [Code]

FewShotNeRF: Meta-Learning-based Novel View Synthesis for Rapid Scene-Specific Adaptation
Piraveen Sivakumar, Paul Janson, Jathushan Rajasegaran, Thanuja Ambegoda
arXiv preprint, 9 Aug 2024
[arXiv]

SSNeRF: Sparse View Semi-supervised Neural Radiance Fields with Augmentation
Xiao Cao, Beibei Lin, Bo Wang, Zhiyong Huang, Robby T. Tan
arXiv preprint, 17 Aug 2024
[arXiv]

GeoTransfer : Generalizable Few-Shot Multi-View Reconstruction via Transfer Learning
Shubhendu Jena, Franck Multon, Adnane Boukhayma
arXiv preprint, 27 Aug 2024
[arXiv]

Generic Objects as Pose Probes for Few-Shot View Synthesis
Zhirui Gao, Renjiao Yi, Chenyang Zhu, Ke Zhuang, Wei Chen, Kai Xu
arXiv preprint, 29 Aug 2024
[arXiv] [Project]

Toward General Object-level Mapping from Sparse Views with 3D Diffusion Priors
Ziwei Liao, Binbin Xu, Steven L. Waslander
CoRL 2024, 7 Oct 2024
[arXiv] [Code]

Few-shot NeRF by Adaptive Rendering Loss Regularization
Qingshan Xu, Xuanyu Yi, Jianyao Xu, Wenbing Tao, Yew-Soon Ong, Hanwang Zhang
ECCV 2024, 23 Oct 2024
[arXiv]

FrugalNeRF: Fast Convergence for Few-shot Novel View Synthesis without Learned Priors
Chin-Yang Lin, Chung-Ho Wu, Chang-Han Yeh, Shih-Han Yen, Cheng Sun, Yu-Lun Liu
arXiv preprint, 21 Oct 2024
[arXiv] [Project]

NeRF-3DGS Transfer

NeRFs to Gaussian Splats, and Back
Siming He, Zach Osman, Pratik Chaudhari
arXiv preprint, 15 May 2024
[arXiv]

NeRF Based SLAM

NeRF SLAM Survey

How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey
Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R. Oswald, Matteo Poggi
arXiv preprint, 20 Feb 2024
[arXiv]

NeRF and Gaussian Splatting SLAM in the Wild
Fabian Schmidt, Markus Enzweiler, Abhinav Valada
4 Dec 2024
[arXiv] [Code]

NeRF SLAM Progress

:fire:NICE-SLAM: Neural Implicit Scalable Encoding for SLAM
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys
CVPR 2022, 22 Dec 2021

Abstract

Neural implicit representations have recently shown encouraging results in various domains, including promising progress in simultaneous localization and mapping (SLAM). Nevertheless, existing methods produce over-smoothed scene reconstructions and have difficulty scaling up to large scenes. These limitations are mainly due to their simple fully-connected network architecture that does not incorporate local information in the observations. In this paper, we present NICE-SLAM, a dense SLAM system that incorporates multi-level local information by introducing a hierarchical scene representation. Optimizing this representation with pre-trained geometric priors enables detailed reconstruction on large indoor scenes. Compared to recent neural implicit SLAM systems, our approach is more scalable, efficient, and robust. Experiments on five challenging datasets demonstrate competitive results of NICE-SLAM in both mapping and tracking quality. Project page: this https URL

[arXiv] [Project] [Github] [Notes]

H2-Mapping: Real-time Dense Mapping Using Hierarchical Hybrid Representation
Chenxing Jiang, Hanwen Zhang, Peize Liu, Zehuan Yu, Hui Cheng, Boyu Zhou, Shaojie Shen
IEEE Robotics and Automation Letters, 5 Jun 2023
[arXiv] [Github] [Video]

PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields
Boming Zhao, Luwei Yang, Mao Mao, Hujun Bao, Zhaopeng Cui
AAAI 2024, 17 Dec 2023
[arXiv]

CaLDiff: Camera Localization in NeRF via Pose Diffusion
Rashik Shrestha, Bishad Koju, Abhigyan Bhusal, Danda Pani Paudel, François Rameau
arXiv preprint, 23 Dec 2023
[arXiv]

Hi-Map: Hierarchical Factorized Radiance Field for High-Fidelity Monocular Dense Mapping
Tongyan Hua, Haotian Bai, Zidong Cao, Ming Liu, Dacheng Tao, Lin Wang
arXiv preprint, 6 Jan 2024
[arXiv]

N^3-Mapping: Normal Guided Neural Non-Projective Signed Distance Fields for Large-scale 3D Mapping
Shuangfu Song, Junqiao Zhao, Kai Huang, Jiaye Lin, Chen Ye, Tiantian Feng
arXiv preprint, 7 Jan 2024
[arXiv]

Q-SLAM: Quadric Representations for Monocular SLAM
Chensheng Peng, Chenfeng Xu, Yue Wang, Mingyu Ding, Heng Yang, Masayoshi Tomizuka, Kurt Keutzer, Marco Pavone, Wei Zhan
arXiv preprint, 12 Mar 2024
[arXiv]

Learning Neural Volumetric Pose Features for Camera Localization
Jingyu Lin, Jiaqi Gu, Bojian Wu, Lubin Fan, Renjie Chen, Ligang Liu, Jieping Ye
arXiv preprint, 19 Mar 2024
[arXiv]

DVN-SLAM: Dynamic Visual Neural SLAM Based on Local-Global Encoding
Wenhua Wu, Guangming Wang, Ting Deng, Sebastian Aegidius, Stuart Shanks, Valerio Modugno, Dimitrios Kanoulas, Hesheng Wang
arXiv preprint, 18 Mar 2024
[arXiv]

WSCLoc: Weakly-Supervised Sparse-View Camera Relocalization
Jialu Wang, Kaichen Zhou, Andrew Markham, Niki Trigoni
arXiv preprint, 22 Mar 2024
[arXiv]

NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising
Tianchen Deng, Yanbo Wang, Hongle Xie, Hesheng Wang, Jingchuan Wang, Danwei Wang, Weidong Chen
arXiv preprint, 29 Mar 2024
[arXiv]

VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field
Fei Xue, Ignas Budvytis, Daniel Olmeda Reino, Roberto Cipolla
arXiv preprint, 14 Apr 2024
[arXiv] [Code]

SLAIM: Robust Dense Neural SLAM for Online Tracking and Mapping
Vincent Cartillier, Grant Schindler, Irfan Essa
arXiv preprint, 17 Apr 2024
[arXiv]

EC-SLAM: Real-time Dense Neural RGB-D SLAM System with Effectively Constrained Global Bundle Adjustment
Guanghao Li, Qi Chen, YuXiang Yan, Jian Pu
arXiv preprint, 20 Apr 2024
[arXiv] [Code]

S3-SLAM: Sparse Tri-plane Encoding for Neural Implicit SLAM
Zhiyao Zhang, Yunzhou Zhang, Yanmin Wu, Bin Zhao, Xingshuo Wang, Rui Tian
arXiv preprint, 28 Apr 2024
[arXiv]

Fast Global Localization on Neural Radiance Field
Mangyu Kong, Seongwon Lee, Jaewon Lee, Euntai Kim
arXiv preprint, 18 Jun 2024
[arXiv]

I^2-SLAM: Inverting Imaging Process for Robust Photorealistic Dense SLAM
Gwangtak Bae, Changwoon Choi, Hyeongjun Heo, Sang Min Kim, Young Min Kim
ECCV 2024, 16 Jul 2024
[arXiv]

Evaluating geometric accuracy of NeRF reconstructions compared to SLAM method
Adam Korycki, Colleen Josephson, Steve McGuire
arXiv preprint, 15 Jul 2024
[arXiv]

Visual Localization in 3D Maps: Comparing Point Cloud, Mesh, and NeRF Representations
Lintong Zhang, Yifu Tao, Jiarong Lin, Fu Zhang, Maurice Fallon
arXiv preprint, 21 Aug 2024
[arXiv]

Neural Implicit Representation for Highly Dynamic LiDAR Mapping and Odometry
Qi Zhang, He Wang, Ru Li, Wenbin Li
arXiv preprint, 26 Sep 2024
[arXiv]

The Radiance of Neural Fields: Democratizing Photorealistic and Dynamic Robotic Simulation
Georgina Nuthall, Richard Bowden, Oscar Mendez
25 Nov 2024
[arXiv]

Camera Pose Estimation & Weak Camera Pose Reconstruction

INeRF: Inverting Neural Radiance Fields for Pose Estimation
Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin
IROS 2021, 10 Dec 2020
[arXiv] [Github]

:fire:NeRF--: Neural Radiance Fields Without Known Camera Parameters
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, Victor Adrian Prisacariu
arXiv preprint, 14 Feb 2021

Abstract

Considering the problem of novel view synthesis (NVS) from only a set of 2D images, we simplify the training process of Neural Radiance Field (NeRF) on forward-facing scenes by removing the requirement of known or pre-computed camera parameters, including both intrinsics and 6DoF poses. To this end, we propose NeRF−−, with three contributions: First, we show that the camera parameters can be jointly optimised as learnable parameters with NeRF training, through a photometric reconstruction; Second, to benchmark the camera parameter estimation and the quality of novel view renderings, we introduce a new dataset of path-traced synthetic scenes, termed as Blender Forward-Facing Dataset (BLEFF); Third, we conduct extensive analyses to understand the training behaviours under various camera motions, and show that in most scenarios, the joint optimisation pipeline can recover accurate camera parameters and achieve comparable novel view synthesis quality as those trained with COLMAP pre-computed camera parameters. Our code and data are available at this https URL.

[arXiv] [Project] [Github]

iMAP: Implicit Mapping and Positioning in Real-Time
Edgar Sucar, Shikun Liu, Joseph Ortiz, Andrew J. Davison
ICCV 2021, 23 Mar 2021
[arXiv] [Project]

GNeRF: GAN-based Neural Radiance Field without Posed Camera
Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, Jingyi Yu
ICCV 2021, 29 Mar 2021
[arXiv] [Github]

:fire:BARF: Bundle-Adjusting Neural Radiance Fields
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey
ICCV 2021, 13 Apr 2021

Abstract

Neural Radiance Fields (NeRF) have recently gained a surge of interest within the computer vision community for its power to synthesize photorealistic novel views of real-world scenes. One limitation of NeRF, however, is its requirement of accurate camera poses to learn the scene representations. In this paper, we propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect (or even unknown) camera poses -- the joint problem of learning neural 3D representations and registering camera frames. We establish a theoretical connection to classical image alignment and show that coarse-to-fine registration is also applicable to NeRF. Furthermore, we show that naïvely applying positional encoding in NeRF has a negative impact on registration with a synthesis-based objective. Experiments on synthetic and real-world data show that BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems (e.g. SLAM) and potential applications for dense 3D mapping and reconstruction.

[arXiv] [Project] [Github]

Self-Calibrating Neural Radiance Fields
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, Jaesik Park
ICCV 2021, 13 Apr 2021
[arXiv] [Project] [Github]

Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields
Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, Fei Wang
CVPR 2023, 21 Nov 2022
[arXiv] [Project] [Github]

:fire:NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, Victor Adrian Prisacariu
CVPR 2023, 14 Dec 2022

Abstract

Training a Neural Radiance Field (NeRF) without pre-computed camera poses is challenging. Recent advances in this direction demonstrate the possibility of jointly optimising a NeRF and camera poses in forward-facing scenes. However, these methods still face difficulties during dramatic camera movement. We tackle this challenging problem by incorporating undistorted monocular depth priors. These priors are generated by correcting scale and shift parameters during training, with which we are then able to constrain the relative poses between consecutive frames. This constraint is achieved using our proposed novel loss functions. Experiments on real-world indoor and outdoor scenes show that our method can handle challenging camera trajectories and outperforms existing methods in terms of novel view rendering quality and pose estimation accuracy. Our project page is this https URL.

[arXiv] [Project] [Github]

Towards Open World NeRF-Based SLAM
Daniil Lisus, Connor Holmes, Steven Waslander
CRV 2023, 8 Jan 2023
[arXiv]

F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories
Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang
CVPR 2023, 28 Mar 2023
[arXiv] [Project] [Github]

Neural Lens Modeling
Wenqi Xian, Aljaž Božič, Noah Snavely, Christoph Lassner
CVPR 2023, 10 Apr 2023
[arXiv] [Project]

LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
Zezhou Cheng, Carlos Esteves, Varun Jampani, Abhishek Kar, Subhransu Maji, Ameesh Makadia
arXiv preprint, 8 Jun 2023
[arXiv] [Project]

CamP: Camera Preconditioning for Neural Radiance Fields
Keunhong Park, Philipp Henzler, Ben Mildenhall, Jonathan T. Barron, Ricardo Martin-Brualla
Siggraph Asia 2023, 21 Aug, 2023
[arXiv] [Project]

MC-NeRF: Muti-Camera Neural Radiance Fields for Muti-Camera Image Acquisition Systems
Yu Gao, Lutong Su, Hao Liang, Yufeng Yue, Yi Yang, Mengyin Fu
arXiv preprint, 14 Sep 2023
[arXiv] [Github]

BID-NeRF: RGB-D image pose estimation with inverted Neural Radiance Fields
Ágoston István Csehi, Csaba Máté Józsa
Nerf4ADR of ICCV 2023, 5 Oct 2023
[arXiv]

NeRFtrinsic Four: An End-To-End Trainable NeRF Jointly Optimizing Diverse Intrinsic and Extrinsic Camera Parameters
Hannah Schieber, Fabian Deuser, Bernhard Egger, Norbert Oswald, Daniel Roth
arXiv preprint, 26 Oct 2023
[arXiv]

PoRF: Pose Residual Field for Accurate Neural Surface Reconstruction
Jia-Wang Bian, Wenjing Bian, Victor Adrian Prisacariu, Philip Torr
arXiv preprint, 11 Oct 2023 [arXiv]

CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses
Hongyu Fu, Xin Yu, Lincheng Li, Li Zhang
arXiv preprint, 15 Oct 2023
[arXiv]

Continuous Pose for Monocular Cameras in Neural Implicit Representation
Qi Ma, Danda Pani Paudel, Ajad Chhatkuli, Luc Van Gool
arXiv preprint, 28 Nov 2023
[arXiv]

IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment
Letian Zhang, Ming Li, Chen Chen, Jie Xu
arXiv preprint, 10 Dec 2023
[arXiv]

Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs
Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, Chong Luo
CVPR 2024, 12 Dec 2023
[arXiv] [Project] [Code]

Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields
Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu
AAAI 2024, 20 Feb 2024
[arXiv] [Project] [Code] [Video]

IFFNeRF: Initialisation Free and Fast 6DoF pose estimation from a single image and a NeRF model
Matteo Bortolon, Theodore Tsesmelis, Stuart James, Fabio Poiesi, Alessio Del Bue
ICRA 2024, 19 Mar 2024
[arXiv] [Project] [Code] [Video]

VF-NeRF: Viewshed Fields for Rigid NeRF Registration
Leo Segre, Shai Avidan
arXiv preprint, 4 Apr 2024
[arXiv]

CT-NeRF: Incremental Optimizing Neural Radiance Field and Poses with Complex Trajectory
Yunlong Ran, Yanxu Li, Qi Ye, Yuchi Huo, Zechun Bai, Jiahao Sun, Jiming Chen
arXiv preprint, 22 Apr 2024
[arXiv]

TD-NeRF: Novel Truncated Depth Prior for Joint Camera Pose and Neural Radiance Field Optimization
Zhen Tan, Zongtan Zhou, Yangbing Ge, Zi Wang, Xieyuanli Chen, Dewen Hu
arXiv preprint, 11 May 2024
[arXiv] [Code]

Leveraging Neural Radiance Fields for Pose Estimation of an Unknown Space Object during Proximity Operations
Antoine Legrand, Renaud Detry, Christophe De Vleeschouwer
arXiv preprint, 21 May 2024
[arXiv]

Camera Relocalization in Shadow-free Neural Radiance Fields
Shiyao Xu, Caiyun Liu, Yuantao Chen, Zhenxin Zhu, Zike Yan, Yongliang Shi, Hao Zhao, Guyue Zhou
ICRA 2024, 23 May 2024
[arXiv] [Code]

SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization
Yiyang Chen, Siyan Dong, Xulong Wang, Lulu Cai, Youyi Zheng, Yanchao Yang
ECCV 2024, 17 Jul 2024
[arXiv] [Code]

Invertible Neural Warp forNeRF
Shin-Fang Chng, Ravi Garg, Hemanth Saratchandran, Simon Lucey
ECCV 2024, 17 Jul 2024
[arXiv] [Project] [Code]

:fire:TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks
Jinjie Mai, Wenxuan Zhu, Sara Rojas, Jesus Zarzar, Abdullah Hamdi, Guocheng Qian, Bing Li, Silvio Giancola, Bernard Ghanem
ECCV 2024, 20 Aug 2024

Abstract

Neural radiance fields (NeRFs) generally require many images with accurate poses for accurate novel view synthesis, which does not reflect realistic setups where views can be sparse and poses can be noisy. Previous solutions for learning NeRFs with sparse views and noisy poses only consider local geometry consistency with pairs of views. Closely following \textit{bundle adjustment} in Structure-from-Motion (SfM), we introduce TrackNeRF for more globally consistent geometry reconstruction and more accurate pose optimization. TrackNeRF introduces \textit{feature tracks}, \ie connected pixel trajectories across \textit{all} visible views that correspond to the \textit{same} 3D points. By enforcing reprojection consistency among feature tracks, TrackNeRF encourages holistic 3D consistency explicitly. Through extensive experiments, TrackNeRF sets a new benchmark in noisy and sparse view reconstruction. In particular, TrackNeRF shows significant improvements over the state-of-the-art BARF and SPARF by ∼8 and ∼1 in terms of PSNR on DTU under various sparse and noisy view setups. The code is available at \href{this https URL}.

[arXiv] [Project] [Code]

KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction
Davide Di Nucci, Alessandro Simoni, Matteo Tomei, Luca Ciuffreda, Roberto Vezzani, Rita Cucchiara
ECCVW 2024, 9 Sep 2024
[arXiv]

Robust SG-NeRF: Robust Scene Graph Aided Neural Surface Reconstruction
Yi Gu, Dongjun Ye, Zhaorui Wang, Jiaxu Wang, Jiahang Cao, Renjing Xu
arXiv preprint, 20 Nov 2024
[arXiv] [Project]

NeRF with MVS

BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis in Large-scale Scenes
Chih-Hai Su, Chih-Yao Hu, Shr-Ruei Tsai, Jie-Ying Lee, Chin-Yang Lin, Yu-Lun Liu
SIGGRAPH 2024, 22 Jul 2024
[arXiv] [Project] [Code]

NeRF AIGC

NeRF AIGC Survey

Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era
Chenghao Li, Chaoning Zhang, Atish Waghwase, Lik-Hang Lee, Francois Rameau, Yang Yang, Sung-Ho Bae, Choong Seon Hong
arXiv preprint, 10 May 2023
[arXiv]

Advances in 3D Generation: A Survey
Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, Ying Shan
arXiv preprint, 31 Jan 2024
[arXiv]

A Survey On Text-to-3D Contents Generation In The Wild
Chenhan Jiang
arXiv preprint, 15 May 2024
[arXiv]

NeRF AIGC Progresses

Zero-Shot Text-Guided Object Generation with Dream Fields
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole
CVPR 2022, 2 Dec 2021
[arXiv] [Project] [Github]

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao
CVPR 2022, 9 Dec 2021
[arXiv] [Project] [Github]

LaTeRF: Label and Text Driven Object Radiance Fields
Ashkan Mirzaei, Yash Kant, Jonathan Kelly, Igor Gilitschenski
CVPR 2022, 4 Jul 2022
[arXiv] [Project]

DreamFusion: Text-to-3D using 2D Diffusion
Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall
arXiv preprint, 29 Sep 2022
[arXiv] [Project] [Unofficial Impl] [threeStudio] [Notes]

Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, Daniel Cohen-Or
arXiv preprint, 14 Nov 2022
[arXiv] [Github] [threeStudio]

:fire:Magic3D: High-Resolution Text-to-3D Content Creation
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin
CVPR 2023, 18 Nov 2022

Abstract

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

[arXiv] [Project] [threeStudio] [Notes]

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, Greg Shakhnarovich
CVPR 2023, 1 Dec 2022
[arXiv] [Project] [Github] [threeStudio]

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction
Zhizhuo Zhou, Shubham Tulsiani
CVPR 2023, 1 Dec 2022
[arXiv] [Project] [Github]

DiffRF: Rendering-Guided 3D Radiance Field Diffusion
Ashkan Mirzaei, Yash Kant, Jonathan Kelly, Igor Gilitschenski
CVPR 2023, 2 Dec 2022
[arXiv] [Project]

:fire:Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, Shenghua Gao
CVPR 2023, 28 Dec 2022

Abstract

Recent CLIP-guided 3D optimization methods, such as DreamFields and PureCLIPNeRF, have achieved impressive results in zero-shot text-to-3D synthesis. However, due to scratch training and random initialization without prior knowledge, these methods often fail to generate accurate and faithful 3D structures that conform to the input text. In this paper, we make the first attempt to introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. Specifically, we first generate a high-quality 3D shape from the input text in the text-to-shape stage as a 3D shape prior. We then use it as the initialization of a neural radiance field and optimize it with the full prompt. To address the challenging text-to-shape generation task, we present a simple yet effective approach that directly bridges the text and image modalities with a powerful text-to-image diffusion model. To narrow the style domain gap between the images synthesized by the text-to-image diffusion model and shape renderings used to train the image-to-shape generator, we further propose to jointly optimize a learnable text prompt and fine-tune the text-to-image diffusion model for rendering-style image generation. Our method, Dream3D, is capable of generating imaginative 3D content with superior visual quality and shape accuracy compared to state-of-the-art methods.

[arXiv] [Project]

Text-To-4D Dynamic Scene Generation
Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, Shenghua Gao
arXiv preprint, 26 Jan 2023
[arXiv] [Project]

:fire:LERF: Language Embedded Radiance Fields
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, Matthew Tancik
arXiv preprint, 16 Mar 2023

Abstract

Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. The project website can be found at this https URL .

[arXiv] [Project] [Github]

:fire:Shap-E: Generating Conditional 3D Implicit Functions
Heewoo Jun, Alex Nichol
arXiv preprint, 3 May 2023

Abstract

We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at this https URL.

[arXiv] [Github]

:fire:3DGen: Triplane Latent Diffusion for Textured Mesh Generation
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, Barlas Oğuz
arXiv preprint, 9 Mar 2023

Abstract

Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which generates the triplane features. For the first time this architecture allows conditional and unconditional generation of high quality textured or untextured 3D meshes across multiple diverse categories in a few seconds on a single GPU. It outperforms previous work substantially on image-conditioned and unconditional generation on mesh quality as well as texture generation. Furthermore, we demonstrate the scalability of our model to large datasets for increased quality and diversity. We will release our code and trained models.

[arXiv]

:fire:MeshDiffusion: Score-based Generative 3D Mesh Modeling
Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, Weiyang Liu
ICLR 2023, 14 Mar 2023

Abstract

We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation. Compared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which are mostly optimized for meshes. Previous scalable methods for generating meshes typically rely on sub-optimal post-processing, and they tend to produce overly-smooth or noisy surfaces without fine-grained geometric details. To overcome these shortcomings, we take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization. We demonstrate the effectiveness of our model on multiple generative tasks.

[arXiv] [Project] [Github]

:fire:DreamBooth3D: Subject-Driven Text-to-3D Generation
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, Varun Jampani
arXiv preprint, 23 Mar 2023

Abstract

We present DreamBooth3D, an approach to personalize text-to-3D generative models from as few as 3-6 casually captured images of a subject. Our approach combines recent advances in personalizing text-to-image models (DreamBooth) with text-to-3D generation (DreamFusion). We find that naively combining these methods fails to yield satisfactory subject-specific 3D assets due to personalized text-to-image models overfitting to the input viewpoints of the subject. We overcome this through a 3-stage optimization strategy where we jointly leverage the 3D consistency of neural radiance fields together with the personalization capability of text-to-image models. Our method can produce high-quality, subject-specific 3D assets with text-driven modifications such as novel poses, colors and attributes that are not seen in any of the input images of the subject.

[arXiv] [Project]

:fire:Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, Dong Chen
arXiv preprint, 23 Mar 2023

Abstract

In this work, we investigate the problem of creating high-fidelity 3D content from only a single image. This is inherently challenging: it essentially involves estimating the underlying 3D geometry while simultaneously hallucinating unseen textures. To address this challenge, we leverage prior knowledge from a well-trained 2D diffusion model to act as 3D-aware supervision for 3D creation. Our approach, Make-It-3D, employs a two-stage optimization pipeline: the first stage optimizes a neural radiance field by incorporating constraints from the reference image at the frontal view and diffusion prior at novel views; the second stage transforms the coarse model into textured point clouds and further elevates the realism with diffusion prior while leveraging the high-quality textures from the reference image. Extensive experiments demonstrate that our method outperforms prior works by a large margin, resulting in faithful reconstructions and impressive visual quality. Our method presents the first attempt to achieve high-quality 3D creation from a single image for general objects and enables various applications such as text-to-3D creation and texture editing.

[arXiv] [Project] [Github] [Notes]

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation
Rui Chen, Yongwei Chen, Ningxin Jiao, Kui Jia
arXiv preprint, 24 Mar 2023
[arXiv] [Project] [Github] [threeStudio]

DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model
Hoigi Seo, Hayeon Kim, Gwanghyun Kim, Se Young Chun
CVPR 2023, 6 Apr 2023
[arXiv] [Project] [Github]

Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, Hao Su
arXiv preprint, 13 Apr 2023
[arXiv] [Project] [Github]

Deceptive-NeRF: Enhancing NeRF Reconstruction using Pseudo-Observations from Diffusion Models
Xinhang Liu, Shiu-hong Kao, Jiaben Chen, Yu-Wing Tai, Chi-Keung Tang
arXiv preprint, 24 May 2023
[arXiv]

:fire:ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu
arXiv preprint, 25 May 2023

Abstract

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512×512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: this https URL

[arXiv] [Project] [Unofficial Implementation]

ATT3D: Amortized Text-to-3D Object Synthesis
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa,Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, James Lucas
arXiv preprint, 6 Jun 2023
[Project]

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, Hao Su
arXiv preprint, 29 Jun 2023
[arXiv] [Project] [Github]

Points-to-3D: Bridging the Gap between Sparse Points and Shape-Controllable Text-to-3D Generation
Chaohui Yu, Qiang Zhou, Jingliang Li, Zhe Zhang, Zhibin Wang, Fan Wang
ACMMM 2023, 26 Jul, 2023
[arXiv]

HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation
Jinbo Wu, Xiaobo Gao, Xing Liu, Zhengyang Shen, Chen Zhao, Haocheng Feng, Jingtuo Liu, Errui Ding
arXiv preprint, 30 Jul 2023
[arXiv]

AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose
Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang, Li Chen, Chao Long, Feida Zhu, Kang Du, Min Zheng
arXiv preprint, 7 Aug 2023
[arXiv] [Project]

Animate124: Animating One Image to 4D Dynamic Scene
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, Gim Hee Lee
arXiv preprint, 24 Nov 2023
[arXiv] [Project]

Prompt2NeRF-PIL: Fast NeRF Generation via Pretrained Implicit Latent
Jianmeng Liu, Yuyao Zhang, Zeyuan Meng, Yu-Wing Tai, Chi-Keung Tang
arXiv preprint, 5 Dec 2023
[arXiv]

ReconFusion: 3D Reconstruction with Diffusion Priors
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, Aleksander Holynski
CVPR 2024, 5 Dec 2023
[arXiv] [Project]

DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior
Tianyu Huang, Yihan Zeng, Zhilu Zhang, Wan Xu, Hang Xu, Songcen Xu, Rynson W. H. Lau, Wangmeng Zuo
CVPR 2024, 11 Dec 2023
[arXiv] [Code]

Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation
Yuze He, Yushi Bai, Matthieu Lin, Jenny Sheng, Yubin Hu, Qi Wang, Yu-Hui Wen, Yong-Jin Liu
arXiv preprint, 19 Dec 2023
[arXiv]

Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
Desai Xie, Jiahao Li, Hao Tan, Xin Sun, Zhixin Shu, Yi Zhou, Sai Bi, Sören Pirk, Arie E. Kaufman
CVPR 2024, 21 Dec 2023
[arXiv] [Project] [Code]

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models
Han Jiang, Haosen Sun, Ruoxuan Li, Chi-Keung Tang, Yu-Wing Tai
arXiv preprint, 30 Dec 2023
[arXiv]

SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
Jan-Niklas Dihlmann, Andreas Engelhardt, Hendrik Lensch
arXiv preprint, 3 Jan 2024
[arXiv] [Project] [Code] [Video]

GO-NeRF: Generating Virtual Objects in Neural Radiance Fields
Peng Dai, Feitong Tan, Xin Yu, Yinda Zhang, Xiaojuan Qi
arXiv preprint, 11 Jan 2024
[arXiv] [Project]

Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation
Minglin Chen, Longguang Wang, Weihao Yuan, Yukun Wang, Zhe Sheng, Yisheng He, Zilong Dong, Liefeng Bo, Yulan Guo
arXiv preprint, 25 Jan 2024
[arXiv]

ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields
Edward Bartrum, Thu Nguyen-Phuoc, Chris Xie, Zhengqin Li, Numair Khan, Armen Avetisyan, Douglas Lanman, Lei Xiao
arXiv preprint, 31 Jan 2024
[arXiv] [Project]

ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis
Bernard Spiegl, Andrea Perin, Stéphane Deny, Alexander Ilin
arXiv preprint, 5 Feb 2024
[arXiv]

VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
Yang Chen, Yingwei Pan, Haibo Yang, Ting Yao, Tao Mei
CVPR 2024, 25 Mar 2024
[arXiv] [Project]

FlexiDreamer: Single Image-to-3D Generation with FlexiCubes
Ruowen Zhao, Zhengyi Wang, Yikai Wang, Zihan Zhou, Jun Zhu
arXiv preprint, 1 Apr 2024
[arXiv] [Project] [Code]

SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer
Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, Xiang Bai
arXiv preprint, 4 Apr 2024
[arXiv] [Project] [Code] [Video]

Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion
Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, Guosheng Lin
arXiv preprint, 9 Apr 2024
[arXiv]

Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences
Seungwook Kim, Kejie Li, Xueqing Deng, Yichun Shi, Minsu Cho, Peng Wang
CVPR 2024, 16 Apr 2024
[arXiv]

MeshLRM: Large Reconstruction Model for High-Quality Mesh
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, Zexiang Xu
arXiv preprint, 18 Apr 2024
[arXiv] [Project]

SketchDream: Sketch-based Text-to-3D Generation and Editing
Feng-Lin Liu, Hongbo Fu, Yu-Kun Lai, Lin Gao
arXiv preprint, 10 May 2024
[arXiv]

4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, Yu Qiao
arXiv preprint, 31 May 2024
[arXiv] [Project] [Code]

Rethinking Score Distillation as a Bridge Between Image Distributions
David McAllister, Songwei Ge, Jia-Bin Huang, David W. Jacobs, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa
arXiv preprint, 13 Jun 2024
[arXiv] [Project]

Preserving Identity with Variational Score for General-purpose 3D Editing
Duong H. Le, Tuan Pham, Aniruddha Kembhavi, Stephan Mandt, Wei-Chiu Ma, Jiasen Lu
arXiv preprint, 13 Jun 2024
[arXiv]

Generative Lifting of Multiview to 3D from Unknown Pose: Wrapping NeRF inside Diffusion
Xin Yuan, Rana Hanocka, Michael Maire
arXiv preprint, 11 Jun 2024
[arXiv]

C3DAG: Controlled 3D Animal Generation using 3D pose guidance
Sandeep Mishra, Oindrila Saha, Alan C. Bovik
arXiv preprint, 11 Jun 2024
[arXiv]

OrientDream: Streamlining Text-to-3D Generation
Yuzhong Huang, Zhong Li, Zhang Chen, Zhiyuan Ren, Guosheng Lin, Fred Morstatter, Yi Xu
arXiv preprint, 14 Jun 2024
[arXiv]

Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
Xinyang Li, Zhangyu Lai, Linning Xu, Yansong Qu, Liujuan Cao, Shengchuan Zhang, Bo Dai, Rongrong Ji
arXiv preprint, 25 jUN 2024
[arXiv] [Code]

A3D: Does Diffusion Dream about 3D Alignment?
Savva Ignatyev, Nina Konovalova, Daniil Selikhanovych, Nikolay Patakin, Oleg Voynov, Dmitry Senushkin, Alexander Filippov, Anton Konushin, Peter Wonka, Evgeny Burnaev
arXiv preprint, 21 Jun 2024
[arXiv]

DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors
Zizheng Yan, Jiapeng Zhou, Fanpeng Meng, Yushuang Wu, Lingteng Qiu, Zisheng Ye, Shuguang Cui, Guanying Chen, Xiaoguang Han
ECCV 2024, 23 Jul 2024
[arXiv] [Project]

HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation
Zezeng Li, Weimin Wang, WenHai Li, Na Lei, Xianfeng Gu
arXiv preprint, 19 Jul 2024
[arXiv]

DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity
Jiwook Kim, Seonho Lee, Jaeyo Shin, Jiho Choi, Hyunjung Shim
arXiv preprint, 16 Jul 2024
[arXiv] [Project]

Generalization

Local Implicit Ray Function for Generalizable Radiance Field Representation
Xin Huang, Qi Zhang, Ying Feng, Xiaoyu Li, Xuan Wang, Qing Wang
CVPR 2023, 25 Apr 2023
[arXiv] [Project] [Video]

MuRF: Multi-Baseline Radiance Fields
Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Marc Pollefeys, Andreas Geiger, Fisher Yu
CVPR 2024, 7 Dec, 2023
[arXiv] [Project] [Code]

GD^2-NeRF: Generative Detail Compensation via GAN and Diffusion for One-shot Generalizable Neural Radiance Fields
Xiao Pan, Zongxin Yang, Shuai Bai, Yi Yang
arXiv preprint, 1 Jan 2024
[arXiv]

Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation
Jiaxu Wang, Ziyi Zhang, Renjing Xu
arXiv preprint, 25 Jan 2024
[arXiv]

Generalizable Novel-View Synthesis using a Stereo Camera
Haechan Lee, Wonjoon Jin, Seung-Hwan Baek, Sunghyun Cho
CVPR 2024, 21 Apr 2024
[arXiv] [Project] [Code]

Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields
Tianqi Liu, Xinyi Ye, Min Shi, Zihao Huang, Zhiyu Pan, Zhan Peng, Zhiguo Cao
CVPR 2024, 26 Apr 2024
[arXiv] [Project] [Code]

:fire:LaRa: Efficient Large-Baseline Radiance Fields
Anpei Chen, Haofei Xu, Stefano Esposito, Siyu Tang, Andreas Geiger
arXiv preprint, 5 Jul 2024

Abstract

Radiance field methods have achieved photorealistic novel view synthesis and geometry reconstruction. But they are mostly applied in per-scene optimization or small-baseline settings. While several recent works investigate feed-forward reconstruction with large baselines by utilizing transformers, they all operate with a standard global attention mechanism and hence ignore the local nature of 3D reconstruction. We propose a method that unifies local and global reasoning in transformer layers, resulting in improved quality and faster convergence. Our model represents scenes as Gaussian Volumes and combines this with an image encoder and Group Attention Layers for efficient feed-forward reconstruction. Experimental results demonstrate that our model, trained for two days on four GPUs, demonstrates high fidelity in reconstructing 360 deg radiance fields, and robustness to zero-shot and out-of-domain testing. Our project Page: this https URL.

[arXiv]

GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer
Youngho Yoon, Hyun-Kurl Jang, Kuk-Jin Yoon
ECCV 2024, 1 Oct 2024
[arXiv] [Code]

OPONeRF: One-Point-One NeRF for Robust Neural Rendering
Yu Zheng, Yueqi Duan, Kangfu Zheng, Hongru Yan, Jiwen Lu, Jie Zhou
arXiv preprint, 30 Sep 2024
[arXiv] [Project] [Code]

Towards Degradation-Robust Reconstruction in Generalizable NeRF
Chan Ho Park, Ka Leong Cheng, Zhicheng Wang, Qifeng Chen
arXiv preprint, 18 Nov 2024
[arXiv]

Model Compression

HollowNeRF: Pruning Hashgrid-Based NeRFs with Trainable Collision Mitigation
Xiufeng Xie, Riccardo Gherardi, Zhihong Pan, Stephen Huang
ICCV 2023, 19 Aug 2023
[arXiv]

SPC-NeRF: Spatial Predictive Compression for Voxel Based Radiance Field
Zetian Song, Wenhong Duan, Yuhuai Zhang, Shiqi Wang, Siwei Ma, Wen Gao
arXiv preprint, 26 Feb 2024
[arXiv]

NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation
Sicheng Li, Hao Li, Yiyi Liao, Lu Yu
CVPR 2024, 2 Apr 2024
[arXiv]

How Far Can We Compress Instant-NGP-Based NeRF?
Yihang Chen, Qianyi Wu, Mehrtash Harandi, Jianfei Cai
arXiv preprint, 6 Jun 2024
[arXiv] [Project] [Code]

Explicit_NeRF_QA: A Quality Assessment Database for Explicit NeRF Model Compression
Yuke Xing, Qi Yang, Kaifa Yang, Yilin Xu, Zhu Li
arXiv preprint, 11 Jul 2024
[arXiv]

HPC: Hierarchical Progressive Coding Framework for Volumetric Video
Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang
arXiv preprint, 12 Jul 2024
[arXiv]

Lagrangian Hashing for Compressed Neural Field Representations
Shrisudhan Govindarajan, Zeno Sambugaro, Akhmedkhan (Ahan)Shabanov, Towaki Takikawa, Daniel Rebain, Weiwei Sun, Nicola Conci, Kwang Moo Yi, Andrea Tagliasacchi
arXiv preprint, 9 Sep 2024
[arXiv] [Project]

Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB
Jae Yong Lee, Yuqun Wu, Chuhang Zou, Derek Hoiem, Shenlong Wang
arXiv preprint, 24 Sep 2024
[arXiv]

Disentangled Generation and Aggregation for Robust Radiance Fields
Shihe Shen, Huachen Gao, Wangze Xu, Rui Peng, Luyang Tang, Kaiqiang Xiong, Jianbo Jiao, Ronggang Wang
ECCV 2024, 24 Sep 2024
[arXiv] [Project]

NeRF Based 2D High Quality Image Synthesis

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
Michael Niemeyer, Andreas Geiger
CVPR 2012, 24 Nov 2020
[arXiv] [Project] [Github]

SDF Based Reconstruction / Other Geometry Based Reconstruction

Neural Point-Based Graphics
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, Victor Lempitsky
arXiv preprint, 19 Jun 2019
[arXiv] [Github] [Video]

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, Yaron Lipman
NeurIPS 2020, 22 Mar 2020
[arXiv] [Project] [Github]

Neural RGB-D Surface Reconstruction
Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies
CVPR 2022, 9 Apr 2021
[arXiv] [Project] [Github]

:fire:NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang
NeurIPS 2021, 20 Jun 2021

Abstract

We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.

[arXiv] [Project] [Github]

Volume Rendering of Neural Implicit Surfaces
Lior Yariv, Jiatao Gu, Yoni Kasten, Yaron Lipman
NeurIPS 2021, 22 Jun 2021
[arXiv] [Project] [Github]

HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
Yiqun Wang, Ivan Skorokhodov, Peter Wonka
NeurIPS 2022, 15 Jun 2022
[arXiv] [Github] [Notes]

NPBG++: Accelerating Neural Point-Based Graphics
Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, Evgeny Burnaev
CVPR 2022, 24 Mar 2022
[arXiv] [Project] [Github]

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu
ICCV 2023, 10 Dec 2022
[arXiv] [Project] [Github]

PET-NeuS: Positional Encoding Tri-Planes for Neural Surfaces
Yiqun Wang, Ivan Skorokhodov, Peter Wonka
CVPR 2023, 9 May 2023
[arXiv] [Github]

NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, Wenping Wang
arXiv preprint, 27 Jun 2022
[arXiv] [Project] [Github]

:fire:TensoSDF: Roughness-aware Tensorial Representation for Robust Geometry and Material Reconstruction
Jia Li, Lu Wang, Lei Zhang, Beibei Wang
SIGGRAPH 2024, 5 Feb 2024

Abstract

Reconstructing objects with realistic materials from multi-view images is problematic, since it is highly ill-posed. Although the neural reconstruction approaches have exhibited impressive reconstruction ability, they are designed for objects with specific materials (e.g., diffuse or specular materials). To this end, we propose a novel framework for robust geometry and material reconstruction, where the geometry is expressed with the implicit signed distance field (SDF) encoded by a tensorial representation, namely TensoSDF. At the core of our method is the roughness-aware incorporation of the radiance and reflectance fields, which enables a robust reconstruction of objects with arbitrary reflective materials. Furthermore, the tensorial representation enhances geometry details in the reconstructed surface and reduces the training time. Finally, we estimate the materials using an explicit mesh for efficient intersection computation and an implicit SDF for accurate representation. Consequently, our method can achieve more robust geometry reconstruction, outperform the previous works in terms of relighting quality, and reduce 50% training times and 70% inference time.

[arXiv]

H2O-SDF: Two-phase Learning for 3D Indoor Reconstruction using Object Surface Fields
Minyoung Park, Mirae Do, YeonJae Shin, Jaeseok Yoo, Jongkwang Hong, Joongrock Kim, Chul Lee
arXiv preprint, 13 Feb 2024
[arXiv]

MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance
Yuqun Wu, Jae Yong Lee, Chuhang Zou, Shenlong Wang, Derek Hoiem
arXiv preprint, 12 Apr 2024
[arXiv]

ActiveNeuS: Active 3D Reconstruction using Neural Implicit Surface Uncertainty
Hyunseo Kim, Hyeonseo Yang, Taekyung Kim, YoonSung Kim, Jin-Hwa Kim, Byoung-Tak Zhang
arXiv preprint, 4 May 2024
[arXiv]

RPBG: Towards Robust Neural Point-based Graphics in the Wild
Qingtian Zhu, Zizhuang Wei, Zhongtian Zheng, Yifan Zhan, Zhuyu Yao, Jiawang Zhang, Kejian Wu, Yinqiang Zheng
arXiv preprint, 9 May 2024
[arXiv] [Code]

LDM: Large Tensorial SDF Model for Textured Mesh Generation
Rengan Xie, Wenting Zheng, Kai Huang, Yizheng Chen, Qi Wang, Qi Ye, Wei Chen, Yuchi Huo
arXiv preprint, 23 May 2024
[arXiv]

RaNeuS: Ray-adaptive Neural Surface Reconstruction
Yida Wang, David Joseph Tan, Nassir Navab, Federico Tombari
3DV 2024, 14 Jun 2024
[arXiv]

Neural Geometry Processing via Spherical Neural Surfaces
Romy Williamson, Niloy J. Mitra
arXiv preprint, 10 Jul 2024
[arXiv]

ActiveNeRF: Learning Accurate 3D Geometry by Active Pattern Projection
Jianyu Tao, Changping Hu, Edward Yang, Jing Xu, Rui Chen
arXiv preprint, 13 Aug 2024
[arXiv]

MGFs: Masked Gaussian Fields for Meshing Building based on Multi-View Images
Tengfei Wang, Zongqian Zhan, Rui Xia, Linxia Ji, Xin Wang
arXiv preprint, 6 Aug 2024
[arXiv]

Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems
Jianheng Liu, Chunran Zheng, Yunfei Wan, Bowen Wang, Yixi Cai, Fu Zhang
arXiv preprint, 9 Sep 2024
[arXiv] [Code]

Rethinking Directional Parameterization in Neural Implicit Surface Reconstruction
Zijie Jiang, Tianhan Xu, Hiroharu Kato
ECCV 2024, 11 Sep 2024
[arXiv]

NeRF + Hardware Optimization/Acceleration

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera
Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, Vladislav Golyanik
CVPR 2023, 23 Jun 2022
[arXiv] [Project] [Github]

Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction
Sixu Li, Chaojian Li, Wenbo Zhu, Boyang (Tony)Yu, Yang (Katie)Zhao, Cheng Wan, Haoran You, Huihong Shi, Yingyan (Celine)Lin
ISCA 2023, 24 Apr 2023
[arXiv]

Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion
Weng Fei Low, Gim Hee Lee
ICCV 2023, 15 Sep 2023
[arXiv] [Project] [Github]

Deformable Neural Radiance Fields using RGB and Event Cameras
Qi Ma, Danda Pani Paudel, Ajad Chhatkuli, Luc Van Gool
arXiv preprint, 25 Sep 2023
[arXiv]

USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
Moyang Li, Peng Wang, Lingzhe Zhao, Bangyan Liao, Peidong Liu
arXiv preprint, 4 Oct 2023
[arXiv]

Thermal-NeRF: Neural Radiance Fields from an Infrared Camera
Tianxiang Ye, Qi Wu, Junyuan Deng, Guoqing Liu, Liu Liu, Songpengcheng Xia, Liang Pang, Wenxian Yu, Ling Pei
arXiv preprint, 15 Mar 2024
[arXiv]

URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields
Bo Xu, Ziao Liu, Mengqi Guo, Jiancheng Li, Gim Hee Li
arXiv preprint, 15 Mar 2024
[arXiv]

Spike-NeRF: Neural Radiance Field Based On Spike Camera
Yijia Guo, Yuanxi Bai, Liwen Hu, Mianzhi Liu, Ziyi Guo, Lei Ma, Tiejun Huang
ICME 2024, 25 Mar 2024
[arXiv]

Mitigating Motion Blur in Neural Radiance Fields with Events and Frames
Marco Cannici, Davide Scaramuzza
CVPR 2024, 28 Mar 2024
[arXiv]

SpikeNVS: Enhancing Novel View Synthesis from Blurry Images via Spike Camera
Gaole Dai, Zhenyu Wang, Qinwen Xu, Wen Cheng, Ming Lu, Boxing Shi, Shanghang Zhang, Tiejun Huang
arXiv preprint, 10 Apr 2024
[arXiv]

Radiance Fields from Photons
Sacha Jungerman, Mohit Gupta
arXiv preprint, 12 Jul 2024
[arXiv]

TeX-NeRF: Neural Radiance Fields from Pseudo-TeX Vision
Chonghao Zhong, Chao Xu
arXiv preprint, 7 Oct 2024
[arXiv]

NeRF-enabled Analysis-Through-Synthesis for ISAR Imaging of Small Everyday Objects with Sparse and Noisy UWB Radar Data
Md Farhan Tasnim Oshim, Albert Reed, Suren Jayasuriya, Tauhidur Rahman
arXiv preprint, 14 Oct 2024
[arXiv]

NeRF + Light Field Rendering

DirectL: Efficient Radiance Fields Rendering for 3D Light Field Displays
Zongyuan Yang, Baolin Liu, Yingde Song, Yongping Xiong, Lan Yi, Zhaohe Zhang, Xunbo Yu
arXiv preprint, 19 Jul 2024
[arXiv] [Project]

NeRF + Point Cloud / LiDAR

:fire:Point-NeRF: Point-based Neural Radiance Fields
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, Ulrich Neumann
CVPR 2022, 21 Jan 2022

Abstract

Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism. The experiments on the DTU, the NeRF Synthetics , the ScanNet and the Tanks and Temples datasets demonstrate Point-NeRF can surpass the existing methods and achieve the state-of-the-art results.

[arXiv] [Project] [Github]

Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, Ulrich Neumann
arXiv preprint, 19 Apr 2023
[arXiv] [Project] [Github] [Notes]

Neural LiDAR Fields for Novel View Synthesis
Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, Or Litany
arXiv preprint, 2 May 2023
[arXiv] [Project] [Notes]

Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields
Tao Hu, Xiaogang Xu, Shu Liu, Jiaya Jia
arXiv preprint, 29 Mar 2023
[arXiv] [Video]

Just Add $100 More: Augmenting NeRF-based Pseudo-LiDAR Point Cloud for Resolving Class-imbalance Problem
Mincheol Chang, Siyeong Lee, Jinkyu Kim, Namil Kim
arXiv preprint, 18 Mar 2024
[arXiv]

GPN: Generative Point-based NeRF
Haipeng Wang
arXiv preprint, 12 Apr 2024
[arXiv]

Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction
Ekansh Agrawal
arXiv preprint, 12 May 2024
[arXiv]

A Probabilistic Formulation of LiDAR Mapping with Neural Radiance Fields
Matthew McDermott, Jason Rife
arXiv preprint, 4 Nov 2024
[arXiv] [Code]

NeRF + Auto Data Collection

AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
Pierre Marza, Laetitia Matignon, Olivier Simonin, Dhruv Batra, Christian Wolf, Devendra Singh Chaplot
arXiv preprint, 21 Apr 2023
[arXiv] [Project] [Video]

NerfBridge: Bringing Real-time, Online Neural Radiance Field Training to Robotics
Javier Yu, Jun En Low, Keiko Nagami, Mac Schwager
ICRA 2023, 16 May 2023
[arXiv] [Github] [Video]

NeRF + Avatar/Talking Head

:fire:AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, Juyong Zhang
ICCV 2021, 20 Mar 2021

Abstract

Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit f

Truncated — view the full README on GitHub.

Contributors

yangjiheng

202 commits

Nplace-su

1 commits

zjwzcx

1 commits

Similar repos

EricLee0224/awesome-nerf-editing

🧙🏻‍♂️A list of papers curated for you to dive into the Awesome Radiance Field-based 3D Editing.

506

ziplab/Awesome-Feed-Forward-3D

An curated list for feed-forward 3D scene modeling, including research directions, datasets, and…

292

yyeboah/Awesome-Text-to-3D

A growing curation of Text-to-3D, Diffusion-to-3D works.

603

dragonlong/Trending-in-3D-Vision

An on-going paper list on new trends in 3D vision with deep learning

333

justimyhxu/awesome-3D-generation

A curated list of awesome 3d generation papers

1,199

chang-xinhai/Awesome-Embodied-3DV

A curated map of 3D/4D perception, reconstruction, generation, simulation-ready assets, and world…

cwchenwang/awesome-3d-diffusion

A collection of papers on diffusion models for 3D generation.

1,257

2hiTee/awesome-3D-Generation

This is a collective repository for all 3D and 4D Object Generation papers

yangjiheng/nerf_and_beyond_docs

1,163

204 commits

updated Dec 26, 2024

See the code

README

NeRF and Beyond Docs

Please feel free to join us on WeChat group or start a discussion topic here.

NeRF/3DGS Book

How to join us

For now, you can join us in the following ways

Bilibili Channel where we post near daily updates (primarily) on NeRF.
WeChat group, due to the limitation of WeChat group, you can add my personal account: jiheng_yang, and I will add you to the chat groups.
If you want to view this from a timeline perspective, please refer to this ProcessOn Diagram
If you think something is not correct or you think we could do better in some way, please write to us through all possible channels or drop an issue. All suggestions are appreciated!
For other discussed techniques that's related to 3D reconstruction and NeRF, please refer to link, we are constantly trying to add more resource to this document.
We are trying to gradually setup Discord channels, join the Discord Channel if you want to, we will certainly look forward to talk with you guys!

3DGS Progresses

For 3DGS related progress, you can refer to 3DGS and Beyond Docs

NeRF progresses

Table of Content

NeRF and Beyond Docs