Meta-learning Neural Lumigraph Representations | NeurIPS 2021

Alexander W. Bergman, Petr Kellnhofer, Gordon Wetzstein

Fast training and real-time rendering of neural surface representations using meta learning.

ABSTRACT

Novel view synthesis is a long-standing problem in machine learning and computer vision. Significant progress has recently been made in developing neural scene representations and rendering techniques that synthesize photorealistic images from arbitrary views. These representations, however, are extremely slow to train and often also slow to render. Inspired by neural variants of image-based rendering, we develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. To push representation convergence times down to minutes, we leverage meta learning to learn neural shape and image feature priors which accelerate training. The optimized shape and image features can then be extracted using traditional graphics techniques and rendered in real time. We show that MetaNLR++ achieves similar or better novel view synthesis results in a fraction of the time that competing methods require.

FILES

CITATION

A.W. Bergman, P. Kellnhofer, G. Wetzstein, Fast Training of Neural Lumigraph Representations using Meta Learning, NeurIPS 2021.

@inproceedings{bergman2021metanlr,
author = {Bergman, Alexander W. and Kellnhofer, Petr and Wetzstein, Gordon},
title = {Fast Training of Neural Lumigraph Representations using Meta Learning},
booktitle = {NeurIPS},
year = {2021},
}

TIMING RESULTS
Training and rendering time of various methods. At all times in training (left plots) and in rendering (right plot), MetaNLR++ is comparable to or better than the respectively best method in novel view synthesis.
Method 25dB PSNR 30dB PSNR Maximum PSNR
NeRF 33.3 min. – 27.95 dB
IDR – – 24.73 dB
NLR 14.7 min. 191.4 min. 32.95 dB
SVS* 2.1 min. – 28.19 dB
NLR++ 3.2 min. 37.4 min. 31.02 dB
MetaNLR++ 1.9 min. 22.5 min. 30.57 dB

During training, MetaNLR++ is faster than other methods at reaching specified synthesized image quality milestones, and does not sacrifice final converged synthesized image quality. Time to reach specified PSNR metrics on the DTU dataset with 7 training images per scene.

METHOD

Overview of our framework. Given a set of multi-view images, we learn a shape representation network, convolutional image encoder and decoder, and on-surface feature aggregation function end-to-end. By working with image features and using meta-learning to learn a prior over feature processing networks and shape representations, our representation is trained significantly faster than other methods. The resulting models can be exported to enable view-dependent real-time rendering using traditional graphics pipelines.

RENDERING RESULTS
MetaNLR++ achieves high-quality synthesized views after only 10 minutes of training from the sparse set of input cameras used in this experiment. Additionally, MetaNLR++ can be rendered in a fraction of the time of volumetric methods such as NeRF or IBRNet.


After training to convergence, MetaNLR++ achieves on-par novel view synthesis results with state-of-the-art methods. Maximum synthesis quality is not sacrificed for the purpose of fast training.


MetaNLR++ can be applied to diverse datasets from which a prior can be learned. When applied to the NLR dataset, shape and synthesized image quality converge significantly faster than the NLR method.

RELATED PROJECTS

You may also be interested in related projects focusing on neural scene representations and rendering:

  • Kellnhofer et al. Neural Lumigraph Rendering. CVPR 2021 (link)
  • Martel et al. ACORN: Adaptive Coordinate Networks for Neural Representation. SIGGRAPH 2021 (link)
  • Chan et al. pi-GAN. CVPR 2021 (link)
  • Lindell et al. Automatic Integration for Fast Neural Rendering. CVPR 2021 (link)
  • Sitzmann et al. Implicit Neural Representations with Periodic Activation Functions. NeurIPS 2020 (link)
  • Sitzmann et al. MetaSDF. NeurIPS 2020 (link)
  • Sitzmann et al. Scene Representation Networks. NeurIPS 2019 (link)
  • Sitzmann et al. Deep Voxels. CVPR 2019 (link)