Neural Holography | SIGGRAPH 2020

Yifan (Evan) Peng, Suyeon Choi, Nitish Padmanaban, Jonghyun Kim, Gordon Wetzstein

We develop Neural Holography – an algorithmic CGH framework that uses camera-in-the-loop training to achieve unprecedented image fidelity and real-time framerates.

SIGGRAPH 2020 ETech - 3 min overview

SIGGRAPH 2020 ETech - 15 min Tech Talk

ABSTRACT

Holographic displays promise unprecedented capabilities for direct-view displays as well as virtual and augmented reality applications. However, one of the biggest challenges for computer-generated holography (CGH) is the fundamental tradeoff between algorithm runtime and achieved image quality, which has prevented high-quality holographic image synthesis at fast speeds. Moreover, the image quality achieved by most holographic displays is low, due to the mismatch between the optical wave propagation of the display and its simulated model. Here, we develop an algorithmic CGH framework that achieves unprecedented image fidelity and real-time framerates. Our framework comprises several parts, including a novel camera-in-the-loop optimization strategy that allows us to either optimize a hologram directly or train an interpretable model of the optical wave propagation and a neural network architecture that represents the first CGH algorithm capable of generating full-color high-quality holographic images at 1080p resolution in real time.

FILES

    • SIGGRAPH 2020, Emerging Technologies, Extended Abstract (pdf)
    • SIGGRAPH Asia 2020, Technical Paper (pdf)
    • SIGGRAPH Asia 2020, Supplement (pdf)
    • Source Code (github repo)

 

CITATION

Y. Peng, S. Choi, N. Padmanaban, J. Kim, G. Wetzstein. Neural Holography. In SIGGRAPH Emerging Technologies, 2020

Y. Peng, S. Choi, N. Padmanaban, G. Wetzstein. Neural Holography with Camera-in-the-loop Training. In SIGGRAPH Asia, 2020

 

BibTeX

@article{Peng:2020:NeuralHolography,
author = {Y. Peng and S. Choi and N. Padmanaban and G. Wetzstein},
title = {{Neural Holography with Camera-in-the-loop Training}},
journal = {ACM Trans. Graph. (SIGGRAPH Asia)},
year = {2020},
}

 

Holographic Near-eye Displays
Illustration of a holographic near-eye display. A laser emits coherent light that is collimated by a lens and propagates to the spatial light modulator (SLM), where the phase of the wave field is delayed in a per-pixel manner. The field continues to propagate and interference creates a visible intensity pattern that is perceived as an image by the user, who observes it through the magnifying lens, which is called the eyepiece.

Camera-in-the-loop (CITL) holography. We develop a family of techniques that use a camera in the loop to optimize a hologram, given a target image, taking into account optical aberrations of the system, phase nonlinearities of the SLM, and other physical imperfections that typically make it difficult to achieve high image quality with experimental holographic displays.

Overview of results
Experimental results generated with our camera-in-the-loop holography algorithm and captured with a holographic near-eye display prototype.



Experimental results (not real time). Comparison of state-of-the-art iterative CGH algorithm (i.e., Wirtinger Holography) and our CITL holography approach. CITL holography significantly improves the experimentally measured image quality by optimizing speckle, contrast, and other aspects.



Experimental results (real time). Comparison of the state-of-the-art real-time CGH algorithm (i.e., double phase–amplitude coding or DPAC) and our HoloNet. Being trained with our CITL techniques, HoloNet achieves a significantly improved image quality compared to existing methods.

Experimental results. Comparison of all the computer-generated holography algorithm we evaluated, including Wirtinger Holography, double phase-amplitude coding (DPAC), Gerchberg-Saxton (GS), our CITL technique, and our HoloNet.

Experimental 3D results. In our paper, we explore several approaches to 3D holographic display, including the varifocal holographic display mode shown here and a multiplane holographic display mode.


Direct vs. iterative CGH algorithms. Direct CGH algorithms achieve real-time rates, but HoloNet is the only one to also achieve a peak signal-to-noise ratio (PSNR) of approximately 30~dB. Iterative algorithms, such as Gerchberg–Saxton (GS) or Wirtinger Holography (WH), offer a slightly improved quality at the cost of extensive computing times. Our SGD algorithm achieves the best image quality among all CGH algorithms. PSNR values are averaged over 100 test images.

Holographic display prototype. Our benchtop display uses a fiber-coupled RGB laser module, collimating optics, an liquid crystal on silicon (LCoS) spatial light modulator, and a machine vision camera.

Related Projects

You may also be interested in related projects from our group on holographic near-eye displays:

  • S. Choi et al. “Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays”, ACM SIGGRAPH Asia 2021 (link)
  • S. Choi et al. “Michelson Holography”, Optica, 2021 (link)
  • N. Padmanaban et al. “Holographic Near-Eye Displays Based on Overlap-Add Stereograms”, ACM SIGGRAPH Asia 2019 (link)

and other next-generation near-eye display and wearable technology:

  • R. Konrad et al. “Gaze-contingent Ocular Parallax Rendering for Virtual Reality”, ACM Transactions on Graphics 2020 (link)
  • B. Krajancich et al. “Optimizing Depth Perception in Virtual and Augmented Reality through Gaze-contingent Stereo Rendering”, ACM SIGGRAPH Asia 2020 (link)
  • B. Krajancich et al. “Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display”, IEEE TVCG, 2020 (link)
  • N. Padmanaban et al. “Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes”, Science Advances 2019 (link)
  • K. Rathinavel et al. “Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics”, IEEE TVCG 2019 (link)
  • N. Padmanaban et al. “Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays”, PNAS 2017 (link)
  • R. Konrad et al. “Accommodation-invariant Computational Near-eye Displays”, ACM SIGGRAPH 2017 (link)
  • R. Konrad et al. “Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays”, ACM SIGCHI 2016 (link)
  • F.C. Huang et al. “The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-Eye Light Field Display with Focus Cues”, ACM SIGGRAPH 2015 (link)

 

Acknowledgements

We would like to thank Julien Martel for help with the camera calibration. Suyeon Choi was supported by a Kwanjeong Scholarship and a Korea Government Scholarship. This project was further supported by Ford, NSF (awards 1553333 and 1839974), a Sloan Fellowship, an Okawa Research Grant, and a PECASE by the ARO.

 

Disclaimer on DPAC

The results of the double-phase amplitude coding (DPAC) approach we show are inspired by, but not representative of, Maimone et al.’s SIGGRAPH 2017 paper. We use our own implementation of their algorithm and run our SLM using phase values in the range [0,2π] whereas they used a range of [0,3π]. Also, their SLM was likely better calibrated than ours. This resulted in a worse image quality of their method reported here compared to their original work. Yet, all CGH methods, including DPAC and neural holography, use the same experimental settings for captured results and are thus directly comparable. Moreover, you can find extensive simulations that are not affected by the range of phase values or SLM calibration in our paper and supplement.