Computational Imaging Publications

GazeGPT | 2024

Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear

Holographic AR Glasses with Metasurface Waveguides | Nature 2024

A near-eye display design that pairs inverse-designed metasurface waveguides with AI-driven holographic displays to enable full-colour 3D augmented reality from a compact glasses-like form factor. Photo by Andrew Brodhead

Holographic Parallax Improves 3D Perceptual Realism | SIGGRAPH 2024

Inclusion of parallax cues in CGH rendering plays a crucial role in enhancing perceptual realism.

State of the Art on Diffusion Models for Visual Computing

Tutorial and survey paper on diffusion models for 2D, 3D, and 4D generation.

Locally Conditioned 3D Diffusion | 3DV 2024

Controllable text-to-3D scene generation with intuitive user inputs.

LumiGAN: Relightable 3D GANs | 3DV 2024

Unconditional Generation of Relightable 3D Human Faces

Efficient 3D GANs with Layered Surface Volumes | 3DV 2024

Efficient 3D Articulated Human Generation with Layered Surface Volumes

Diffusion in the Dark | WACV 2024

A diffusion model for low-light image reconstruction for text recognition.

Towards Attention-aware Foveated Rendering | SIGGRAPH 2023

Existing perceptual models used in foveated graphics neglect the effects of visual attention distribution. We introduce the first attention-aware model of contrast sensitivity and motivate the development of future foveation

GeNVS: 3D-Aware Diffusion Models | ICCV 2023

Generative Novel View Synthesis with 3D-Aware Diffusion Models

3D Neural Field Generation using Triplane Diffusion | CVPR 2023

State-of-the-art 3D diffusion model for unconditional object generation.

Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Models

Generation of articulable head avatars from text prompts using diffusion models.

PixelRNN | CVPR 2024

In-pixel recurrent neural networks for end-to-end-optimized perception with neural sensors.

DehazeNeRF: Haze Removal using Neural Radiance Fields | 3DV 2024

Multiple Image Haze Removal and 3D Shape Reconstruction using Neural Radiance Fields.

SinGRAF: Learning a 3D Generative Radiance Field for a Single Scene | CVPR 2023

Generating different realizations of a single 3D scene from a few images.

DiffDreamer: View Generation with Conditional Diffusion Models | ICCV 2023

Single-image-conditioned “infinite nature” approach with conditional diffusion models.

Generative Neural Articulated Radiance Fields | NeurIPS 2022

Unconditional generation of editable radiance field representations of human bodies.

Learning to Solve PDE-constrained Inverse Problems with Graph Networks | ICML 2022

Solving difficult inverse problems in physics using fast GNNs and generative priors.

Time-multiplexed Neural Holography | SIGGRAPH 2022

A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators.

Holographic Glasses | SIGGRAPH 2022

An ultra-thin VR display using computer-generated holography.

CryoAI: Fast Reconstruction of Poses and 3D Molecular Volumes for Cryo-EM | ECCV 2022

Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images.

3D GAN Inversion for Controllable Portrait Image Animation | ECCVW 2022

Portrait image animation and attribute editing using 3D GAN inversion.

EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks | CVPR 2022

3D GAN for photorealistic, multiview consistent, and shape-aware image synthesis.

BACON: Band-limited Coordinate Networks | CVPR 2022

A new type of neural network with an analytical Fourier spectrum that enables multiscale scene representation.

Learning Spatially Varying Pixel Exposures for Motion Deblurring | ICCP 2022

Per-pixel exposures implemented on a programmable sensor that find the best tradeoff between denoising short exposures and deblurring long ones.

MantissaCam for Snapshot HDR Imaging | ICCP 2022

Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding This project won the Best Demo Award at ICCP 2022!

Deep Learning 3D Localization Microscopy | Optics Letters 2021

Deep learning multi-shot 3D localization microscopy using hybrid optical–electronic computing.

Unfiltered Holography | Optics Letters 2021

An algorithmic framework for optimizing high diffraction orders without optical filtering to enable compact holographic displays.

Partially-coherent Neural Holography | Science Advances 2021

A holographic display combining artificial intelligence with partially coherent light sources to reduce speckle in AR/VR applications.

Time Multiplexed Coded Apertures | ICCV 2021

TMCA is a new type of codification for compressive imaging systems using coded apertures. It enables better reconstruction quality due to a better conditioning of the systems’ sensing matrices.

Neural 3D Holography | SIGGRAPH Asia 2021

A differentiable camera-calibrated wave propagation model for holographic near-eye displays that enables unprecedented 3D image fidelity.

Meta-learning Neural Lumigraph Representations | NeurIPS 2021

Fast training and real-time rendering of neural surface representations using meta learning.

ACORN: Adaptive Coordinate Networks for Neural Representation | SIGGRAPH 2021

ACORN is a hybrid implicit-explicit neural representation that enables large-scale fitting of signals such as shapes or images.

Eccentricity-dependent Spatio-temporal Flicker Fusion | SIGGRAPH 2021

We derive an eccentricity-dependent spatio-temporal model of the visual system to enable the development of new temporally foveated graphics techniques.

ScanGAN360

A Generative Model of Realistic Scanpaths for 360° Images

Event-Based Eye Tracking for AR/VR | IEEE VR 2021

Ultra low-latency and low-power eye tracking for AR/VR beyond 10,000 Hz using event sensors.

Neural Lumigraph Rendering | CVPR 2021

A real-time neural rendering approach.

pi-GAN | CVPR 2021

A new GAN architecture and training procedure for 3D-aware image synthesis using periodic implicit neural representations.

Automatic Integration | CVPR 2021

A new framework to integrate signals with implicit neural representations and its application to volume rendering.

Depth from Defocus with Learned Optics | ICCP 2021

End-to-end optimization of optics and image processing for passive depth estimation.

Michelson Holography | OSA Optica 2021

A holographic display technology that optimizes image quality for emerging holographic near-eye displays using 2 SLMs and camera-in-the-loop calibration.

Keyhole Imaging | IEEE TCI 2021

Computational imaging of moving 3D objects through the keyhole of a closed door.

AI with Optics & Photonics | Nature 2020

Review article on inference in artificial intelligence with deep optics and photonics.

Confocal Diffuse Tomography | Nature Communications 2020

A new computationally efficient method to recover the 3D shape of objects hidden behind a thick scattering layer.

Neural Holography | SIGGRAPH 2020

We develop Neural Holography – an algorithmic CGH framework that uses camera-in-the-loop training to achieve unprecedented image fidelity and real-time framerates.

Gaze-contingent Stereo Rendering | SIGGRAPH Asia 2020

We introduce a gaze-contingent stereo rendering technique that improves perceptual realism and depth perception of VR and physical-digital object alignment in AR displays.

Implicit Neural Representations with Periodic Activation Functions | NeurIPS 2020

An implicitly defined neural signal representation for images, audio, shapes, and wavefields.

MetaSDF | NeurIPS 2020

A meta-learning approach to generalizing over neural signed distance functions.

Semantic Implicit Neural Scene Representations with Semi-supervised Training | 3DV 2020

We demonstrate a representation that jointly encodes shape, appearance, and semantics in a 3D-structure-aware manner.

Learned Diffractive Achromat for Full-Spectrum Imaging | OSA Optica 2020

We realize the joint learning of a diffractive achromat with a realistic aperture size and an image recovery neural network in an end-to-end manner across the full visible spectrum.

3D Imaging with an RGB Camera and a single SPAD Transient | ECCV 2020

Robust 3D imaging with an RGB camera and a single SPAD transient fueled by machine learning.

COSMOS | Neuron 2020

A computational microscope enabling fast, synchronous recording of cortex-spanning neural dynamics.

Deep Optics for Single-shot HDR Imaging | CVPR 2020

End-to-end optimization of optics and image processing for HDR imaging.

SPADnet | Optics Express 2020

A neural sensor fusion framework for robust 3D imaging with single-photon detectors.

Neural Sensors: Learning Pixel Exposures with Programmable Sensors | ICCP/T-PAMI 2020

We propose the learning of the pixel exposures of a sensor, taking into account its hardware constraints, jointly with decoders to reconstruct HDR images and high-speed videos from coded images.

Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates | ICCP 2020

An end-to-end differentiable depth imaging system which jointly optimizes the LiDAR scanning pattern and sparse depth inpainting.

Non-line-of-sight Surface Reconstruction Using the Directional Light-cone Transform | CVPR 2020

We propose a joint albedo–normal approach to non-line-of-sight (NLOS) surface reconstruction using the directional light-cone transform (D-LCT).

Factored Occlusion AR Display | IEEE VR 2020

A novel, computational approach to obtaining hard-edge mutual occlusion in optical see-through augmented reality, using only a single spatial light modulator.

Gaze-Contingent Ocular Parallax Rendering for VR | TOG 2020

A new gaze-contingent rendering mode for VR/AR that renders in perceptually correct ocular parallax which benefits depth perception and perceptual realism.

Scene Representation Networks | NeurIPS 2019

A data-efficient, scalable, interpretable and flexible neural scene representation.

Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes | Science Advances 2019

A new presbyopia correction technology that uses eye tracking, depth sensing, and focus-tunable lenses to automatically refocus the real world.

Learned Large Field-of-View Imaging With Thin-Plate Optics | SIGGRAPH Asia 2019

We propose a lens design and learned reconstruction architecture that provide an order of magnitude increase in field of view for computational imaging using only a single thin-plate lens element.

Holographic Near-Eye Displays Based on Overlap-Add Stereograms | SIGGRAPH Asia 2019

The overlap-add stereogram (OLAS) algorithm uses overlapping hogels to encode the view-dependent lighting effects of a light field into a hologram achieving better quality than other holographic stereograms.

Varifocal Occlusion-capable AR Display | IEEE ISMAR 2019

A new optical see-through AR display system that renders mutual occlusion in a depth-dependent, perceptually realistic manner.

Deep Optics for Monocular Depth Estimation and 3D Object Detection | ICCV 2019

Monocular depth estimation and 3D object detection with optimized optical elements, defocus blur, and chromatic aberrations.

Wave-Based Non-Line-of-Sight Imaging using Fast f-k Migration | SIGGRAPH 2019

We introduce a wave-based image formation model for the problem of non-line-of-sight (NLOS) imaging. Inspired by inverse methods used in seismology, we adapt a frequency-domain method, f-k migration, for solving

Non-line-of-sight Imaging with Partial Occluders and Surface Normals | TOG 2019

A new approach to non-line-of-sight imaging that estimates partial occlusions in the hidden volume along with surface normals.

Acoustic Non-Line-of-Sight Imaging | CVPR 2019

We introduce a novel approach to seeing around corners using acoustic echoes. A system of speakers emits sound waves which scatter from a wall to a hidden object and back.

DeepVoxels: Learning Persistent 3D Feature Embeddings | CVPR 2019

3D understanding in generative neural networks.

LiFF: Light Field Features in Scale and Depth | CVPR 2019

An efficient 4D light field feature detector and descriptor along with a large-scale 4D light field dataset.

Hybrid optical-electronic convolutional neural networks for image classification | Scientific Reports 2018

Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.

Sub-picosecond Photon-efficient 3D Imaging using Single-photon Sensors | Scientific Reports 2018

State-of-the-art depth estimation with pileup correction for single-photon avalanche diodes.

3D Deconvolution for Low Photon Count Fluorescence Imaging | Scientific Reports 2018

A convex algorithm 3D deconvolution of low photon count fluorescence imaging.

End-to-end Optimization of Optics and Image Processing | SIGGRAPH 2018

Jointly optimizing high-level image processing and camera optics to design novel domain-specific cameras.

Single-Photon 3D Imaging with Deep Sensor Fusion | SIGGRAPH 2018

Photon efficient 3D imaging using single-photon detectors and deep neural networks.

Confocal Non-Line-of-Sight Imaging Based on the Light Cone Transform | Nature 2018

A confocal scanning technique solves the reconstruction problem of non-line-of-sight imaging to give fast and high-quality reconstructions of hidden objects.

Transient Imaging with SPADs | CVPR 2017, ICCP 2018

Capturing and reconstructing transient images with single-photon avalanche diodes (SPAD) at interactive rates. This page describes the following projects presented at CVPR 2017 and ICCP 2018. CVPR 2017 Reconstructing

Saliency in VR: How do people explore virtual environments? | IEEE VR 2018

Using eyetracking data to explore visual saliency in VR

Towards a Machine-learning Approach for Sickness Prediction in 360° Stereoscopic Videos | IEEE VR 2018

Using machine learning approaches to automate the process of determining sickness ratings for a given 360° stereoscopic video.

SpinVR: Towards Live VR Video | SIGGRAPH Asia 2017

A computational camera system that directly captures streaming 3D virtual reality content natively in the omni-directional stereo (ODS) format.

Accommodation-invariant Near-eye Displays | SIGGRAPH 2017

A new VR/AR display technology that produces visual stimuli that are invariant to the accommodation state of the eye. Accommodation is then driven by stereoscopic cues, instead of retinal blur, and

Optimizing VR with Gaze-contingent Focus Displays | PNAS 2017

A gaze-contingent (varifocal) near-eye display technology with focus cues and an evaluation for how to make VR displays with focus cues accessible for users of all ages.

Aperture interference light field fluorescence microscopy | ICCP 2017

Understanding and developing an approach to improve upon the poor spatial resolution of light field microscopy for state-of-the-art volumetric fluorescence imaging.

Wide-FOV Monocentric Light Field Camera | CVPR 2017

This work presents the first single-lens wide-field-of-view (FOV) light field (LF) camera. Shown below is a pan through a 138°, 72-MPix LF captured using the optical prototype. Shifting the virtual

ProxImaL: Efficient Image Optimization using Proximal Algorithms | SIGGRAPH 2016

ProxImaL is a Python-embedded modeling language for image optimization problems.

Multi-Camera Time-of-Flight Systems | SIGGRAPH 2016

Computational imaging with multi-camera time-of-flight imaging systems.

Focus-tunable and Monovision Near-eye Displays | SIGCHI 2016

We develop an optical system based on focus-tunable optics and evaluate three display modes (conventional, dynamic-focus, and monovision) with respect to the vergence-accommodation conflict.

Variable Aperture Light Field Photography: Overcoming the Diffraction-limited Spatio-angular Resolution Tradeoff | CVPR 2016

A method to measure light fields with high diffraction-limited resolution using Wigner-based point spread functions and computed tomographic reconstruction.

Extended field-of-view and increased-signal 3D holographic illumination with time-division multiplexing | Optics Express 2015

We develop a computational illumination method for fast, large field of view, 3D two-photon illumination in vivo neuronal calcium imaging.