`

Planar Reflection-Aware Neural Radiance Fields

Chen Gao1, Yipeng Wang1, Changil Kim1, Jia-Bin Huang2, Johannes Kopf1,
1Meta 2University of Maryland
Paper arXiv Supplemental

We introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections.

The videos above demonstrate our RA-NeRF's capabilities by comparing reflection-free renderings (left) with our method's complete renderings that combine both primary colors and reflections (right). Click on the video to pause it and hover over to see comparisons.

Abstract

Neural Radiance Fields (NeRF) have demonstrated exceptional capabilities in reconstructing complex scenes with high fidelity. However, NeRF's view dependency can only handle low-frequency reflections. It falls short when handling complex planar reflections, often interpreting them as erroneous scene geometries and leading to duplicated and inaccurate scene representations. To address this challenge, we introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections. We query a single radiance field to render the primary color and the source of the reflection. We propose a sparse edge regularization to help utilize the true sources of reflections for rendering planar reflections rather than creating a duplicate along the primary ray at the same depth.

As a result, we obtain accurate scene geometry. Rendering along the primary ray results in a clean, reflection-free view, while explicitly rendering along the reflected ray allows us to reconstruct highly detailed reflections. Our extensive quantitative and qualitative evaluations of real-world datasets demonstrate our method's enhanced performance in accurately handling reflections.

Method

Main figure.

Why NeRF generates false geometry?

(a) NeRF only traces a single ray, thus incorrectly interpreting reflections as false geometry, leading to inaccurate scene representations.

(b) Our planar reflection-aware NeRF addresses this by employing the law of reflection during rendering, which involves explicitly casting a reflected ray to capture the source of the reflections. Our method prevents the creation of false geometries along the primary ray at depths corresponding to real objects on the reflected ray.

NeRF traditionally traces a single ray, which can misinterpret reflections as false geometry, leading to inaccurate scene representations. Our approach explicitly casts a reflected ray to accurately capture the source of reflections. We model the attenuation of light and calculate the transmittance at the ray-plane intersection. The final color is composed by integrating these elements to achieve a more accurate rendering.

BibTeX

@article{gao2024planar,
  author    = {Gao, Chen and Wang, Yipeng and Kim, Changil and Huang, Jia-Bin and Kopf, Johannes},
  title     = {Planar Reflection-Aware Neural Radiance Fields},
  journal   = {SIGGRAPH ASIA},
  year      = {2024},
}