Learned Diffractive Achromat for Full-Spectrum Imaging | OSA Optica 2020

Xiong Dun, Hayato Ikoma, Gordon Wetzstein, Zhanshan Wang, Xinbin Cheng, Yifan (Evan) Peng

We realize the joint learning of a diffractive achromat with a realistic aperture size and an image recovery neural network in an end-to-end manner across the full visible spectrum.

ABSTRACT

Diffractive achromats (DAs) promise ultra-thin and light-weight form factors for full-color computational imaging systems. However, designing DAs with the optimal optical transfer function (OTF) distribution suitable for image reconstruction algorithms has been a difficult challenge. Emerging end-to-end optimization paradigms of diffractive optics and processing algorithms have achieved impressive results, but these approaches require immense computational resources and solve non-convex inverse problems with millions of parameters. Here, we propose a learned rotational symmetric DA design using a concentric ring decomposition that reduces the computational complexity and memory requirements by one order of magnitude compared with conventional end-to-end optimization procedures, which simplifies the optimization significantly. With this approach, we realize the joint learning of a DA with an aperture size of 8 mm and an image recovery neural network, i.e., Res-Unet, in an end-to-end manner across the full visible spectrum(429–699 nm). The peak signal-to-noise ratio of the recovered images of our learned DA is 1.3 dB higher than that ofDAs designed by conventional sequential approaches. This is because the learned DA exhibits higher amplitudes of theOTF at high frequencies over the full spectrum. We fabricate the learned DA using imprinting lithography. Experiments show that it resolves both fine details and color fidelity of diverse real-world scenes under natural illumination. The proposed design paradigm paves the way for incorporating DAs for thinner, lighter, and more compact full-spectrum imaging systems.

FILES

 

CITATION

Xiong Dun, Hayato Ikoma, Gordon Wetzstein, Zhanshan Wang, Xinbin Cheng, and Yifan Peng, “Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging,” Optica 7, 913-922 (2020)

BibTeX

@article{Dun:2020:LearnedDiffractiveAchromat,
author = {X. Dun, H. Ikoma, G. Wetzstein, Z. Wang, X. Cheng, Y. Peng},
title = {Learned Rotationally Symmetric Diffractive Achromat for Full-Spectrum Computational Imaging},
journal = {Optica},
number = {8},
pages = {913–922},
publisher = {OSA},
volume = {7},
year = {2020},
doi = {10.1364/OPTICA.394413},
}

* Collaboration between Stanford University and Tongji University.

OVERVIEW

Overview of proposed end-to-end learning: The parameters of the diffractive achromat (DA) and image recovery algorithm are learned jointly using the end-to-end optimization paradigm. In each forward pass, the spectrally varying scene is convolved with the spectrally varying PSFs of the rotationally symmetric parametrized DA. Then, Gaussian noise is added to the simulated sensor image after integrating over the color response of the RGB sensor for each channel. A neural network, e.g., a Res-Unet consisting of two base network units, is applied as the image recovery unit to resolve a high-fidelity color image. Finally, a differentiable loss, such as the mean squared error for the ground truth image, is defined on the recovered image. An extra energy regularization is added to force light rays to hit within the designated sensor area. In the backward pass, the error is backpropagated to the learned parameters of the image recovery network and height profile of the DA.
Principle illustration of the rotationally symmetric imaging model: (a) DOE parameterization in traditional 2D manners is used as a reference. (b) The dimension of optimization parameters can be shrunk to 1D by applying the rotationally symmetric parameterization. (c) The complex transmittance function of the rotationally symmetric DOE is superimposed with a sequence of discrete concentric rings, that can be further decomposed to a 1D sum of series of circ functions. Each PSF of circ function can be represented by the 1D 1st order Bessel function of the first kind. Using this rotationally symmetric imaging model, the calculation dimension of PSFs and that of optimization parameters can both be reduced to 1D.
Experimental results of the fabricated DA: For each pair, we show the degraded sensor measurement and the recovery result. The exposure time for these images is 2.5, 125, 76, 600, 25, and 600 ms (from left to right, top to bottom) at ISO 100. The images are center-cropped regions (3,000  2,000) of the original camera measurement. The processing time at this image size on an NVIDIA 1080Ti GPU is around 4 s.

Related Projects

You may also be interested in related projects, where we apply the idea of Deep Optics, i.e. end-to-end optimization of optics and image processing, to other applications, like image classification, extended depth-of-field imaging, superresolution imaging, or optical computing.

  • Wetzstein et al. 2020. AI with Optics & Photonics. Nature (review paper, link)
  • Martel et al. 2020. Neural Sensors. ICCP & TPAMI 2020 (link)
  • Dun et al. 2020. Learned Diffractive Achromat. Optica 2020 (link)
  • Metzler et al. 2020. Deep Optics for HDR Imaging. CVPR 2020 (link)
  • Chang et al. 2019. Deep Optics for Depth Estimation and Object Detection. ICCV 2019 (link)
  • Peng et al. 2019. Large Field-of-view Imaging with Learned DOEs. SIGGRAPH Asia 2019 (link)
  • Chang et al. 2018. Hybrid Optical-Electronic Convolutional Neural Networks with Optimized Diffractive Optics for Image Classification. Scientific Reports (link)
  • Sitzmann et al. 2018. End-to-end Optimization of Optics and Imaging Processing for Achromatic Extended Depth-of-field and Super-resolution Imaging. ACM SIGGRAPH 2018 (link)