Automatic Integration | CVPR 2021

David B. Lindell*, Julien N.P. Martel*, Gordon Wetzstein

A new framework to integrate signals with implicit neural representations and its application to volume rendering.

ABSTRACT

Numerical integration is a foundational technique in scientific computing and is at the core of many computer vision applications. Among these applications, implicit neural volume rendering has recently been proposed as a new paradigm for view synthesis, achieving photorealistic image quality. However, a fundamental obstacle to making these methods practical is the extreme computational and memory requirements caused by the required volume integrations along the rendered rays during training and inference. Millions of rays, each requiring hundreds of forward passes through a neural network are needed to approximate those integrations with Monte Carlo sampling. Here, we propose automatic integration, a new framework for learning efficient, closed-form solutions to integrals using implicit neural representation networks. For training, we instantiate the computational graph corresponding to the derivative of the implicit neural representation. The graph is fitted to the signal to integrate. After optimization, we reassemble the graph to obtain a network that represents the antiderivative. By the fundamental theorem of calculus, this enables the calculation of any definite integral in two evaluations of the network. Using this approach, we demonstrate a greater than 10× improvement in computation requirements, enabling fast neural volume rendering.

Results

An example of AutoInt for compressive sensing computed tomography. Left: illustration of the parameterization. Center: sinograms computed with the integral networks using different nonlinear activation functions. The ground truth (GT) sinogram is subsampled in angle by 4× (top), 8× (middle), and 16× (bottom). The optimized networks are used to interpolate the missing measurements. Using the Swish activation performs best in these experiments. Right: 1D scanlines of the sinogram centers shows the interpolation behavior of each method for each subsampling level.

Results of volumetric rendering on the NerF synthetic dataset.

Results of volumetric rendering on the NerF synthetic dataset.

Results of volumetric rendering on real captured data.

FILES

CITATION

D. B. Lindell*, J. N. P. Martel*, G. Wetzstein, AutoInt: Automatic Integration for Fast Neural Volume Rendering (CVPR), 2021

@inproceedings{lindell2021autoint,
author = {Lindell, David B. and Martel, Julien N.P. and Wetzstein, Gordon},
title = {AutoInt: Automatic Integration for Fast Neural Volume Rendering},
booktitle = {Proceedings of the conference on Computer Vision and Pattern Recognition (CVPR)},
year={2021}
}

The AutoInt framework. After (1) defining an integral network
architecture, (2) AutoInt builds the corresponding grad network,
which is (3) optimized to represent a function. (4) Definite in-
tegrals can then be computed by evaluating the integral network,
which shares parameters with its grad network.
Volume rendering pipeline. During training, the grad networks representing volume density σ and color c are optimized for a
given set of multi-view images (top left). For inference, the grad networks’ parameters are reassembled to form the integral networks,
which represent antiderivatives that can be efficiently evaluated to calculate ray integrals through the volume (bottom left). A sampling
network predicts the locations of piecewise sections used for evaluating the definite integrals (right).

Related Projects

You may also be interested in related projects focusing on neural scene representations and rendering:

  • Chan et al. pi-GAN. CVPR 2021 (link)
  • Kellnhofer et al. Neural Lumigraph Rendering. CVPR 2021 (link)
  • Sitzmann et al. Implicit Neural Representations with Periodic Activation Functions. NeurIPS 2020 (link)
  • Sitzmann et al. MetaSDF. NeurIPS 2020 (link)
  • Sitzmann et al. Scene Representation Networks. NeurIPS 2019 (link)
  • Sitzmann et al. Deep Voxels. CVPR 2019 (link)