Additional Information
Vision correction with computational displays. On a conventional screen, people with optical aberrations see a blurred image (center left). Current approaches to aberration-correcting display use multilayer prefiltering (center) or light field displays (center right). While the former technology enhances perceived image sharpness, contrast is severely reduced. Existing light field-based solutions offer high contrast but require a very high angular sampling density, which significantly reduces image resolution. In this paper, we explore the convergence of light field display optics and computational prefiltering (right), which achieves high image resolution and contrast simultaneously.
Illustration of vision-correcting displays. Observing a conventional 2D display outside the focal range of the eye results in a blurred image (top). A multilayer display with prefiltered image generation (second row) allows for improved image sharpness at the cost of reduced contrast. Image contrast can be preserved using a light field approach via lenslet arrays on the screen (third row); this approach severely reduces image resolution. Combining light field display and computational prefiltering, as proposed in this paper (bottom), allows for vision-correcting image display with significantly improved image resolution and contrast.
Light field analysis for different displays. The light field emitted by a display is parameterized by its coordinates on the screen xd, on the pupil u, and on the retina x (a). This light field propagates through the pupil and is projected into a 2D image on the retina. For an in-focus display, the light field incident on the retina is a horizontal line in the frequency domain (b). For a displayed image outside the accommodation range of the observer, the corresponding light field is slanted and energy is lost at some spatial frequencies (c). Multilayer displays utilize an additional display layer to preserve all spatial frequencies (d). With light field displays, frequency loss is also avoided; the perceived image frequencies are a combination of all spatio-angular frequencies of the incident light field (e). The ray paths in (a) show two effects for a hyperopic eye observing a light field display. First, each photoreceptor on the retina averages over multiple neighboring pixels on the screen (green shaded regions). Second, each pixel on the screen (e.g., xd0) emits different intensities toward different regions on the pupil (u0,u1), allowing the same pixel to appear differently when observed from different locations (x0,x1) on the retina (red arrows).
Light field prefiltering. The proposed prefiltering approach computes a light field (here with 3×3 views) that results in a desired 2D projection on the retina of an observer. The prefiltered light field for an example scene is shown in (a), its simulated projection on the retina in (c), and an image observed on a conventional screen in (b). Spatio-angular frequencies of the light field are amplified, resulting in the desired sharpening when integrated on the retina. Two sample “flatland” light fields with different angular sampling rates are shown in display (d,f) and in eye (e,g) coordinates. Here, the yellow boxes illustrate why 4D light field prefiltering is more powerful than 2D image prefiltering: a single region on the retina receives contributions from multiple different light field views (e,g). Wherever that is the case, the inverse problem of light field prefiltering is well-posed but in other regions the problem is the same as the ill-posed problem faced with conventional 2D displays (e).
Conditioning analysis. The light field projection matrix corresponding to a defocused eye is ill-conditioned. With more angular resolution available in the emitted light field, more degrees of freedom are added to the system, resulting in lower condition numbers (lower is better). The condition number of the projection matrix is plotted for a varying defocus distance (kernel size) and angular resolution (number of light field views). We observe that even as few as 1.5 angular light field samples entering the pupil of an observer decrease the condition number.
Tradeoff between angular light field resolution and image contrast. Top: we reconstruct a test image with different combinations of angular resolution and image contrast and plot the achieved PSNR. Bottom: using prefiltering with a conventional 2D display (b), we obtain either a low-quality but high-contrast image or a high-quality but low-contrast image. For a light field display with 1.5 or more prefiltered views entering the pupil (c), a similar trend is observed but overall reconstruction quality is significantly increased. (Snellen chart courtesy of Jeff Dahl)
Compensating for a range of lateral viewpoints. Aberration-free image display is possible when the relative position of the eye with respect to the display is known. The green plot evaluates image degradation for viewpoints that deviate laterally from the sweetspot. Only slight ringing is visible and, due to periodic viewing zones of the employed parallax barrier display, image quality varies in a periodic manner (top, zoom-in). We can account for a range of perspectives in the compensation, ensuring high image quality for a wider viewing range (blue plot). The columns on the bottom right show on-axis and off-axis views with and without accounting for a range of lateral viewpoints in the optimization. (Source image by Picasso (Wikipedia))
Accounting for a range of viewing distances. Top row: when considering a fixed viewing distance, defocus errors are compensated at that exact distance (top center) but image quality degrades when the observer moves forward or back (top left and right). The proposed method can account for a range of viewing distances (bottom row), which slightly degrades quality at the sweetspot but significantly improves all other distances. (Source image courtesy of KarHan Tan)
Prototype display. We construct an aberration-correcting display using parallax barriers. The barrier mask contains a pinhole array (left) that is mounted at a slight offset in front of an Apple iPod touch 4 screen (lower right). The display emits a light field with a high-enough angular resolution so that at least two views enter the pupil of a human observer. This effect is illustrated on the top right: multiple Arabic numerals are emitted in different viewing directions; the finite pupil size then creates an average of multiple different views on the retina (here simulated with a camera).
Photographs of prototype display. The hyperopic camera simulates a human pupil with a diameter of 6 mm at a distance of 25 cm to the screen. Focused at 38 cm, images shown on a conventional screen are blurred (first column). While previous methods theoretically facilitate increased image sharpness (second column), achievable resolution is fundamentally limited by the spatio-angular resolution tradeoff of the required light field display. Light field prefiltering, as proposed in this paper, allows for significantly increased resolutions (third column). The prototype closely resembles simulations (right column). (From top, source images by dfbphotos (flickr), van Gogh (Wikipedia), Houang Stephane (flickr), JFXie (flickr), Jameziecakes (flickr), Cezanne (Wikipedia), Picasso (Wikipedia), Matisse (Wikipedia))
Evaluation and comparison to previous work. We compare simulations of conventional and vision-correcting image display qualitatively and quantitatively using contrast and quality-mean-opinion-square (QMOS) error metrics. A conventional out-of-focus display always appears blurred (second column). Multilayer displays with prefiltering improve image sharpness but at a much lower contrast (third column). Light field displays without prefiltering require high angular resolutions, hence provide a low spatial resolution (fourth column). The proposed method combines prefiltering and light field display to optimize image contrast and sharpness (right column). The QMOS error metric is a perceptually linear metric, predicting perceived quality for a human observer. We also plot maps that illustrate the probability of an observer detecting the difference of a displayed image to the target image (bottom row). Our method performs best in most cases. (Source images courtesy of flickr users Jameziecakes, KarHan Tan, Mostaque Chowdhury, Thomas Quine (from top))
Correcting for higher-order aberrations. We simulate images observers with different types of higher-order aberrations perceive (top row) and show corresponding point spread functions (top row, insets), which exhibit a range of different shapes. Most of them are difficult to compensate with a conventional 2D display (bottom row, lower left parts), although the blur kernel associated with trefoil (lower right) is frequency preserving and therefore invertible. The proposed aberration-correcting display is successful in compensating all of these aberrations (bottom row, upper right parts). (Source image courtesy of flickr user Jameziecakes)
