Saliency in VR: How do people explore virtual environments? | IEEE VR 2018

Vincent Sitzmann*, Ana Serrano Pacheu*, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein

Using eyetracking data to explore visual saliency in VR

Saliency in VR: How do people explore virtual environments?

ABSTRACT

Understanding how people explore immersive virtual environments is crucial for many applications, such as designing virtual reality (VR) content, developing new compression algorithms, or learning computational models of saliency or visual attention. Whereas a body of recent work has focused on modeling saliency in desktop viewing conditions, VR is very different from these conditions in that viewing behavior is governed by stereoscopic vision and by the complex interaction of head orientation, gaze, and other kinematic constraints. To further our understanding of viewing behavior and saliency in VR, we capture and analyze gaze and head orientation data of 169 users exploring stereoscopic, static omni-directional panoramas, for a total of 1980 head and gaze trajectories for three different viewing conditions. We provide a thorough analysis of our data, which leads to several important insights, such as the existence of a particular fixation bias, which we then use to adapt existing saliency predictors to immersive VR conditions. In addition, we explore other applications of our data and analysis, including automatic alignment of VR video cuts, panorama thumbnails, panorama video synopsis, and saliency-based compression.

Selected insights

This is a summary of a few insights we present in the paper – please refer to the paper for more.

The equator bias: Similar to the “center bias” observed in fixations of humans in 2D images, humans exhibit a bias towards fixating at the equator of a given VR scene. The equator bias is present both in scenes observed with a VR headset (left) or in panoramas observed in a desktop environment (right).

Using existing 2D saliency predictors on VR panoramas: We show how to leverage existing saliency predictors for 2D images to predict visual saliency in VR scenes – here, we project single viewports of the panorama to 2D, predict their saliency, and stitch them back to a saliency panorama.

Gaze and head statistics: Our dataset allows in-depth analysis of heand and eye movement statistics in VR. The vestibulo-ocular reflex (left) can be observed clearly – it arises when head movement occurs while gaze is fixated. (center, right): Two modes of head and gaze movement can be observed in VR: When people are fixating, there head velocity as well as eye eccentricity are lower – they are in the “exploration” mode. When people are reorienting to a new salient part of the scene, they’re moving their head much faster and eye eccentricity is higher – they are in “re-orientation” mode.

Dataset of head and Gaze trajectories in VR

Using an Oculus DK2 headset equipped with a PupilLabs eyetracker, we collect gaze and head trajectories of users exploring VR scenes. Here, you can see an example of participants of our user study exploring a photorealistic rendering of an apartment. The dataset as well as code for analysis is public.

Selected Applications

These are two of five protoype applications we present in the paper – please refer to the paper for applications in automatic VR video editing and panoramic compression using saliency!

Panoramic thumbnails: We can leverage VR saliency maps to automatically generate panoramic thumbnails that summarize the most salient viewport of a scene – this may be useful for VR content providers.

VR video synopsis: The same approach can yield automatic GIF summaries of panoramic video content in VR.

FILES

CITATION

V. Sitzmann et al., “Saliency in VR: How do people explore virtual environments?,” in IEEE Transactions on Visualization and Computer Graphics, vol. PP, no. 99, pp. 1-1.
doi: 10.1109/TVCG.2018.2793599

BibTeX

@article{Sitzmann_TVCG_VR-saliency,
author = {Vincent Sitzmann and Ana Serrano and Amy Pavel and Maneesh Agrawala and Diego Gutierrez and Belen Masia and Gordon Wetzstein},
title = {How do people explore virtual environments?},
journal = {IEEE Transactions on Visualization and Computer Graphics},
year = {2017}
}