Keyhole Imaging | IEEE TCI 2021

Christopher A. Metzler, David B. Lindell, Gordon Wetzstein

Computational imaging of moving 3D objects through the keyhole of a closed door.

ABSTRACT

Non-line-of-sight (NLOS) imaging and tracking is an emerging technology that allows the shape or position of objects around corners or behind diffusers to be recovered from transient, time-of-flight measurements. However, existing NLOS approaches require the imaging system to scan a large area on a visible surface, where the indirect light paths of hidden objects are sampled. In many applications, such as robotic vision or autonomous driving, optical access to a large scanning area may not be available, which severely limits the practicality of existing NLOS techniques. Here, we propose a new approach, dubbed keyhole imaging, that captures a sequence of transient measurements along a single optical path, for example, through a keyhole. Assuming that the hidden object of interest moves during the acquisition time, we effectively capture a series of time-resolved projections of the object’s shape from unknown viewpoints. We derive inverse methods based on expectation-maximization to recover the object’s shape and location using these measurements. Then, with the help of long exposure times and retroreflective tape, we demonstrate successful experimental results with a prototype keyhole imaging system.

FILES

CITATION

C. Metzler, D. Lindell, G. Wetzstein, Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path, IEEE Transactions on Computational Imaging, 2021.

 

BibTeX

@article{Metzler:2021:KeyholeImaging,
author = {C. Metzler and D. Lindell and G. Wetzstein},
title = {{Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path}},
journal = {IEEE Transactions on Computational Imaging},
year = {2021},
}

Overview of results


Keyhole Imaging Overview
Keyhole imaging. A time-resolved detector and pulsed laser illuminate and image a point visible through a keyhole (left). As a hidden person moves, the detector captures a series of time-resolved measurements of the indirectly scattered light (center). From these measurements, we reconstruct both hidden object shape (e.g., for a hidden mannequin) and the time-resolved trajectory (right).

Keyhole Imaging Prototype
Experimental setup. Our optical system sends a laser pulse through the keyhole of a closed door. On the other side of the door, the hidden object moves along a translation stage. When third-bounce photons return, they are recorded and time-stamped by a SPAD. Top-right inset: A beam splitter (BS) is used to place the laser and SPAD in a confocal configuration.

Keyhole Imaging Experiments
Experimental results. First row: Images of the hidden objects. Second row: Reconstructions of the hidden objects using GD when their trajectories are known. Third row: EM reconstructions of the hidden objects when their trajectories are unknown. Fourth row: EM estimates of the trajectories of the hidden objects, each of which follows a different trajectory, where the dot color indicates position over time.

Acknowledgements

This project is supported by a Stanford Graduate Fellowship in Science and Engineering, a National Science Foundation CAREER award (IIS 1553333), a Sloan Fellowship, the DARPA REVEAL program, and a PECASE form the ARO.

Related Projects

You may also be interested in related projects, where we have developed non-line-of-sight imaging systems:

  • Metzler et al. 2021. Keyhole Imaging. IEEE Trans. Computational Imaging (link)
  • Lindell et al. 2020. Confocal Diffuse Tomography. Nature Communications (link)
  • Young et al. 2020. Non-line-of-sight Surface Reconstruction using the Directional Light-cone Transform. CVPR (link)
  • Lindell et al. 2019. Wave-based Non-line-of-sight Imaging using Fast f-k Migration. ACM SIGGRAPH (link)
  • Heide et al. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (presented at SIGGRAPH) (link)
  • Lindell et al. 2019. Acoustic Non-line-of-sight Imaging. CVPR (link)
  • O’Toole et al. 2018. Confocal Non-line-of-sight Imaging based on the Light-cone Transform. Nature (link)

and direct line-of-sight or transient imaging systems:

  • Bergman et al. 2020. Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates. ICCP (link)
  • Nishimura et al. 2020. 3D Imaging with an RGB camera and a single SPAD. ECCV (link)
  • Heide et al. 2019. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Scientific Reports (link)
  • Lindell et al. 2018. Single-Photon 3D Imaging with Deep Sensor Fusions. ACM SIGGRAPH (link)
  • O’Toole et al. 2017. Reconstructing Transient Images from Single-Photon Sensors. CVPR (link)