Rho-NLR: A Neural Lumigraph Renderer with Controllable Illumination

The field of computer graphics has seen the rapid development of a class of solutions to difficult inverse problems such as determining 3D structure and view-dependent properties of an object from a sparse set of images. The Neural Lumigraph Rendering (NLR) approach disentangles geometry and appearance into two separate implicit neural representations, leveraging the unique fitting capabilities of sinusoidal representation networks (SIRENs), and then exports the result into a mesh with the unstructured lumigraph rendering technique for real-time rendering. While this technique presents robust reconstruction and synthesis quality, the problem of modelling illumination and reflectance properties in Neural Lumigraph Renderers has not yet been treated. We propose a straightforward modification of the NLR neural pipeline, dubbed rho-NLR, which demonstrates the robust capabilities of the NLR structure for high-fidelity view synthesis while allowing controllable illumination. By altering the appearance model to output the coefficients of a reflectance distribution function in a finite spherical harmonic basis, we obtain a lightweight representation which requires one small matrix multiplication per pixel to evaluate, allowing for dynamic scene relighting which is real-time within a given viewpoint. Finally, we publish an open-source implementation of Neural Lumigraph Rendering in TensorFlow 2.5.0, as well as our own rho-NLR.