End-to-end Optimization of Optics and Image Processing | SIGGRAPH 2018

Vincent Sitzmann*, Steven Diamond*, Yifan Peng*, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, Gordon Wetzstein

Jointly optimizing high-level image processing and camera optics to design novel domain-specific cameras.

ABSTRACT

In typical cameras the optical system is designed first; once it is fixed, the parameters in the image processing algorithm are tuned to get good image reproduction. In contrast to this sequential design approach, we consider joint optimization of an optical system (for example, the physical shape of the lens) together with the parameters of the reconstruction algorithm. We build a fully-differentiable simulation model that maps the true source image to the reconstructed one. The model includes diffractive light propagation, depth and wavelength-dependent effects, noise and nonlinearities, and the image post-processing. We jointly optimize the optical parameters and the image processing algorithm parameters so as to minimize the deviation between the true and reconstructed image, over a large set of images. We implement our joint optimization method using autodifferentiation to efficiently compute parameter gradients in a stochastic optimization algorithm. We demonstrate the efficacy of this approach by applying it to achromatic extended depth of field and snapshot super-resolution imaging.

Optimizing Domain-specific optics

Differentiable PSF simulation: The proposed framework is an end-to-end differentiable pipeline architecture. In each forward pass, the PSF of the current optical element is simulated using the proposed wave-based image formation module. The simulated PSF is then convolved with a batch of images, and noise is added to account for sensor read noise. A post-processing algorithm solves the Tikhonov-regularized least-squares problem for image reconstruction with the image formation model. Finally, a differentiable loss, such as mean squared error with respect to the ground-truth image, is defined on the reconstructed images. In the backward pass, the error is backpropagated all the way back to the PSF simulation, through the diffraction, to the optical element itself.

FILES

 

CITATION

V. Sitzmann et al., “End-to-end Optimization of Optics and Image Processing for Achromatic Extended Depth of Field and Super-resolution Imaging,” in ACM SIGGRAPH, 2018.
doi: 10.1145/3197517.3201333

BibTeX

@article{Sitzmann:2018:EndToEndCam,
author = {V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, G. Wetzstein},
title = {End-to-end Optimization of Optics and Image Processing for Achromatic Extended Depth of Field and Super-resolution Imaging},
journal = {ACM Trans. Graph. (SIGGRAPH)},
year = {2018}
}

Related Projects

You may also be interested in related projects, where we apply the idea of Deep Optics, i.e. end-to-end optimization of optics and image processing, to other applications, like image classification, extended depth-of-field imaging, superresolution imaging, or optical computing.

  • Wetzstein et al. 2020. AI with Optics & Photonics. Nature (review paper, link)
  • Martel et al. 2020. Neural Sensors. ICCP & TPAMI 2020 (link)
  • Dun et al. 2020. Learned Diffractive Achromat. Optica 2020 (link)
  • Metzler et al. 2020. Deep Optics for HDR Imaging. CVPR 2020 (link)
  • Chang et al. 2019. Deep Optics for Depth Estimation and Object Detection. ICCV 2019 (link)
  • Peng et al. 2019. Large Field-of-view Imaging with Learned DOEs. SIGGRAPH Asia 2019 (link)
  • Chang et al. 2018. Hybrid Optical-Electronic Convolutional Neural Networks with Optimized Diffractive Optics for Image Classification. Scientific Reports (link)
  • Sitzmann et al. 2018. End-to-end Optimization of Optics and Imaging Processing for Achromatic Extended Depth-of-field and Super-resolution Imaging. ACM SIGGRAPH 2018 (link)