Neural 3D Holography | SIGGRAPH Asia 2021

Suyeon Choi*, Manu Gopakumar*, Yifan (Evan) Peng, Jonghyun Kim, Gordon Wetzstein

A differentiable camera-calibrated wave propagation model for holographic near-eye displays that enables unprecedented 3D image fidelity.

SIGGRAPH Asia 2021 - 5 MIN OVERVIEW

ABSTRACT

Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the physical optics. We propose a neural network–parameterized plane-to-multiplane wave propagation model that closes the gap between physics and simulation. Our model is automatically trained using camera feedback and it outperforms related techniques in 2D plane-to-plane settings by a large margin. Moreover, it is the first network-parameterized model to naturally extend to 3D settings, enabling high-quality 3D computer-generated holography using a novel phase regularization strategy of the complex-valued wave field. The efficacy of our approach is demonstrated through extensive experimental evaluation with both VR and optical see-through AR display prototypes.

METHOD

Illustration of our 3D wave propagation model and RGBD supervision strategy. The phase pattern displayed by the SLM is processed by a CNN. The resulting complex-valued wave field is propagated to all target planes using a conventional ASM wave propagation operator. The wave fields at each target plane are processed again by smaller CNNs. The loss function constrains the masked amplitudes at the target planes to match the masked target RGB image, where the binary masks at each target plane are computed from the target depth map.
AR prototype

VR prototype
Prototypes. Diagrams and photographs of the prototype holographic near-eye display systems for AR and VR.

 

FILES

  • SIGGRAPH Asia 2021, Technical Paper (pdf)
  • SIGGRAPH Asia 2021, Technical Paper Supplement (pdf)
  • Source Code and Data (github repo)

CITATION

S. Choi*, M. Gopakumar*, Y. Peng, J. Kim, G. Wetzstein, Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays, ACM Trans. Graph. (SIGGRAPH Asia), 2021

@article{choi2021neural3d,
author = {Choi, Suyeon and Gopakumar, Manu and Peng, Yifan and Kim, Jonghyun and Wetzstein, Gordon},
title = {Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays},
journal = {ACM Trans. Graph. (SIGGRAPH Asia)},
year={2021}
}

Experimental results

VR 2D results

ASM vs. NH3D (ours)


NH (Peng et al., 2020) vs. NH3D (ours)

DPAC vs. NH3D (ours)

CITL (Peng et al., 2020) vs NH3D (ours, no camera)
Ours gets even better than CITL!
2D results. Our model results in sharper images with higher contrast and less speckle than other models under the same experimental conditions for previous CGH algorithms, including the angular spectrum method (ASM), neural holography (NH), the double phase-amplitude coding (DPAC), and camera-in-the-loop holography (CITL) approach.

VR 3D results

DPAC vs. ours


ASM vs. ours


Captured video (ours)
3D results. Our method exhibits very good image quality for both in-focus and out-of-focus parts, improving upon the quality achievable by DPAC and the ASM in similar experimental conditions.

AR results

DPAC vs. ours

ASM vs. ours
Optical see-through AR results. Red arrows indicate virtual objects that are focused at a particular depth. We see that algorithms using our wave propagation model perform better than those not using it.

understanding wave propagation model

Wave propagation model gradients. We show simulated gradients for the ASM, NH, HIL, and the proposed model as well as a gradient of the physical optical system captured using the finite differences method. Our model provides the best approximation to the physical gradient.

speckle-free 3d holography

Phase smoothness regularization. A conventional gradient descent solver (SGD) using our wave propagation model that only constrains in-focus scene parts results in good image quality in those regions but significant out-of-focus speckle artifacts. The same model used with our proposed ADMM solver promoting piecewise smooth phases for the in-focus parts of the scene exhibits very good image quality for both in-focus and out-of-focus parts.

Related Projects

You may also be interested in related projects from our group on holographic near-eye displays:

  • S. Choi et al. “Michelson Holography”, Optica, 2021 (link)
  • Y. Peng et al. “Neural Holography”, ACM SIGGRAPH Asia 2020 (link)
  • N. Padmanaban et al. “Holographic Near-Eye Displays Based on Overlap-Add Stereograms”, ACM SIGGRAPH Asia 2019 (link)

and other next-generation near-eye display and wearable technology:

  • R. Konrad et al. “Gaze-contingent Ocular Parallax Rendering for Virtual Reality”, ACM Transactions on Graphics 2020 (link)
  • B. Krajancich et al. “Optimizing Depth Perception in Virtual and Augmented Reality through Gaze-contingent Stereo Rendering”, ACM SIGGRAPH Asia 2020 (link)
  • B. Krajancich et al. “Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display”, IEEE TVCG, 2020 (link)
  • N. Padmanaban et al. “Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes”, Science Advances 2019 (link)
  • K. Rathinavel et al. “Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics”, IEEE TVCG 2019 (link)
  • N. Padmanaban et al. “Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays”, PNAS 2017 (link)
  • R. Konrad et al. “Accommodation-invariant Computational Near-eye Displays”, ACM SIGGRAPH 2017 (link)
  • R. Konrad et al. “Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays”, ACM SIGCHI 2016 (link)
  • F.C. Huang et al. “The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-Eye Light Field Display with Focus Cues”, ACM SIGGRAPH 2015 (link)

 

Acknowledgements

We thank Mert Pilanci for helpful discussions and advice. Suyeon Choi was supported by a Kwanjeong Scholarship and a Korea Government Scholarship. Manu Gopakumar was supported by a Stanford Graduate Fellowship. This project was further supported by Ford (Alliance Project), Intel, NSF (award 1839974), and a PECASE by the ARO.

 

Disclaimer on DPAC

The results of the double-phase amplitude coding (DPAC) approach we show are inspired by, but not representative of, Maimone et al.’s SIGGRAPH 2017 paper. We use our own implementation of their algorithm and run our SLM using phase values in the range [0,2π] whereas they used a range of [0,3π]. Also, their SLM was likely better calibrated than ours. This resulted in a worse image quality of their method reported here compared to their original work. Yet, all CGH methods, including DPAC and neural holography, use the same experimental settings for captured results and are thus directly comparable. Moreover, you can find extensive simulations that are not affected by the range of phase values or SLM calibration in our paper and supplement.