Gaze-contingent Stereo Rendering | SIGGRAPH Asia 2020

Brooke Krajancich, Petr Kellnhofer, Gordon Wetzstein

We introduce a gaze-contingent stereo rendering technique that improves perceptual realism and depth perception of VR and physical-digital object alignment in AR displays.

SIGGRAPH Asia 2020 - 3 Min Overview

SIGGRAPH Asia 2020 - 15 min Tech Talk

ABSTRACT

Virtual and augmented reality (VR/AR) displays crucially rely on stereoscopic rendering to enable perceptually realistic user experiences. Yet, existing near-eye display systems ignore the gaze-dependent shift of the no-parallax point in the human eye. Here, we introduce a gaze-contingent stereo rendering technique that models this effect and conduct several user studies to validate its effectiveness. Our findings include experimental validation of the location of the no-parallax point, which we then use to demonstrate significant improvements of disparity and shape distortion in a VR setting, and consistent alignment of physical and digitally rendered objects across depths in optical see-through AR. Our work shows that gaze-contingent stereo rendering improves perceptual realism and depth perception of emerging wearable computing systems.

FILES

  • Technical Paper (pdf)
  • Supplement (pdf)

CITATION

B. Krajancich, P. Kellnhofer, G. Wetzstein, “Optimizing Depth Perception in Virtual and Augmented Reality through Gaze-contingent Stereo Rendering”, in ACM Trans. Graph., 39 (6), 2020.

BibTeX
@article{Krajancich:2020:gc_stereo,
author = {Krajancich, Brooke
and Kellnhofer, Petr
and Wetzstein, Gordon},
title = {Optimizing Depth Perception in Virtual and Augmented Reality through Gaze-contingent Stereo Rendering},
journal = {ACM Trans. Graph.},
volume = {39},
issue = {6},
year={2020}
}

User Experiment Results

Depth Perception in VR: Subjects viewed two identical triangle wave random dot stereogram stimuli, one rendered with fine-tuned (FT) IPD and the other with gaze-contingent (GC) rendering. Both stimuli were rendered at a target depth of either 0.3, 0.5 or 0.7m and we asked which of the two contained angles closer to 90º. Despite the seemingly small effect size, shape distortion was detectable with GC rendering chosen as significantly better for closer distances in particular.

Digital-Physical Object Alignment in AR: Subjects viewed a playing card rendered at a target distance of either 0.5, 1.0, 1.5 or 2.0 m next to a physical reference and were asked to select which of two rendering modes provided better alignment in depth. Native HoloLens (HL) and gaze-contingent (GC) rendering were each compared with fine-tunedn (FT) IPD rendering in separate blocks of trials. We find that using an initial calibration procedure to accurately measure the subject’s IPD significantly improved alignment compared to the standard HoloLens approach, however GC rendering was able to further improve alignment at closer distances indicating that it is most critical when arm’s reach viewing is required.

The No-parallax Point and Disparity Distortion

Gaze-dependent Shift of the Optical Center: The offset of the position of the no-parallax point (or center of perspective of the eye) has implications on perceived disparity. Namely, since a user’s inter-pupillary distance (IPD) changes with gaze direction, disparity and hence depth is being distorted by current stereo rendering approaches. For example, as a user verges closer the no-parallax points shift inwards and their effective IPD decreases. With conventional stereo rendering, this would result in the perception of the object being closer than that intended.

The Optics of the Eye: Illustration of the optical and visual axes and relevant points in the right eye (top view). The optical axis connects the anterior vertex of the cornea (V) and the center of rotation (C). The visual axis connects the point of fixation (P) with the front nodal point (N), which extends through the rear nodal point (N’), to intersect with the fovea (F). The angle α offsets the visual axis on average by 5º in the nasal and 3º in the inferior directions. In this work we show that the front nodal point is the no-parallax point of the human eye, located 7–8 mm in front of the center of rotation.

Gaze-contingent Disparity Distortion Example: Rendering models that do not account for ocular motion can create distortions of binocular disparity, as seen in this example (top). The gaze position is indicated by the red and white dot. The color-coded error map (bottom) illustrates the magnitude of this effect as the difference between angular disparities resulting from classical and our gaze-contingent stereoscopic rendering for a translating fixation point. Both shortening (red) and stretching (blue) of disparity gradients can be observed.

Related Projects

You may also be interested in related projects from our group on perceptual aspects of near-eye displays:

  • R. Konrad et al. “Gaze-contingent Ocular Parallax Rendering for Virtual Reality”, ACM Transactions on Graphics 2020 (link)
  • B. Krajancich et al. “Optimizing Depth Perception in Virtual and Augmented Reality through Gaze-contingent Stereo Rendering”, ACM SIGGRAPH Asia 2020 (link)
  • N. Padmanaban et al. “Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays”, PNAS 2017 (link)

and other next-generation near-eye display and wearable technology:

  • Y. Peng et al. “Neural Holography with Camera-in-the-loop Training”, ACM SIGGRAPH 2020 (link)
  • B. Krajancich et al. “Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display”, IEEE TVCG, 2020 (link)
  • N. Padmanaban et al. “Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes”, Science Advances 2019 (link)
  • K. Rathinavel et al. “Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics”, IEEE TVCG 2019 (link)
  • R. Konrad et al. “Accommodation-invariant Computational Near-eye Displays”, ACM SIGGRAPH 2017 (link)
  • R. Konrad et al. “Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays”, ACM SIGCHI 2016 (link)
  • F.C. Huang et al. “The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-Eye Light Field Display with Focus Cues”, ACM SIGGRAPH 2015 (link)

 

ACKNOWLEDGEMENTS

B.K. was supported by a Stanford Knight-Hennessy Fellowship. G.W. was supported by an Okawa Research Grant and a Sloan Fellowship. Other funding for the project was provided by NSF (award numbers 1553333 and 1839974) and a PECASE by the ARO. The authors would also like to thank Professor Anthony Norcia, for advising on an appropriate experiment to measure depth distortion, and Robert Konrad, for providing additional insights on ocular parallax.