Transient Imaging with SPADs | CVPR 2017, ICCP 2018

Capturing and reconstructing transient images with single-photon avalanche diodes (SPAD) at interactive rates. This page describes the following projects presented at CVPR 2017 and ICCP 2018.

 

CVPR 2017

Reconstructing Transient images from single-photon sensors

Matthew O’Toole, David B. Lindell, Felix Heide, Kai Zang, Steven Diamond, Gordon Wetzstein

 

ICCP 2018

TOWARDS TRANSIENT IMAGING AT INTERACTIVE RATES WITH SINGLE-PHOTON DETECTORS

David B. Lindell, Matthew O’Toole, Gordon Wetzstein

CVPR Spotlight Presentation

Supplemental Video

ABSTRACT

Computer vision algorithms build on 2D images or 3D videos that capture dynamic events at the millisecond time scale. However, capturing and analyzing “transient images” at the picosecond scale—i.e., at one trillion frames per second—reveals unprecedented information about a scene and light transport within. This is not only crucial for time-of-flight range imaging, but it also helps further our understanding of light transport phenomena at a more fundamental level and potentially allows to revisit many assumptions made in different computer vision algorithms.

In this work, we design and evaluate an imaging system that builds on single photon avalanche diode (SPAD) sensors to capture multi-path responses with picosecond-scale active illumination. We develop inverse methods that use modern approaches to deconvolve and denoise measurements in the presence of Poisson noise, and compute transient images at a higher quality than previously reported. The small form factor, fast acquisition rates, and relatively low cost of our system potentially makes transient imaging more practical for a range of applications.

CVPR 2017 Paper

Files

  • technical paper (pdf)
  • supplemental materials (pdf)
  • all results (raw and processed) as video clips (zip)
  • raw data (see below)

Citation
M. O’Toole, F. Heide, D. Lindell, K. Zang, S. Diamond, G. Wetzstein, “Reconstructing Transient Images from Single-Photon Sensors”, IEEE Int. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

BibTeX
@article{OToole:2017:SPAD,
author = {M. O’Toole and F. Heide and D. Lindell and K. Zang and S. Diamond and G. Wetzstein},
title = {{Reconstructing Transient Images from Single-Photon Sensors}},
journal = {Proc. IEEE CVPR},
year = {2017},
}

Errata
Supplementary Equations 22 and 23 were missing a term in the camera-ready CVPR paper, this is fixed in the manuscript above.

SPAD DATA

The following MATLAB .mat files contain the RAW LinoSPAD data captured with our system.

 

 

ICCP 2018 paper

Conference Presentation

Supplemental Video

Files

  • technical paper (pdf)
  • raw data (link)
  • processing code (link)

Citation
D. Lindell, M. O’Toole, G. Wetzstein, “Towards Transient Imaging at Interactive Rates with Single-photon Detectors”, IEEE Int. Conference on Computational Photography (ICCP), 2018.

BibTeX
@article{Lindell:2018:SPAD,
author = {D. Lindell and M. O’Toole and G. Wetzstein},
title = {{Towards Transient Imaging at Interactive Rates with Single-Photon Detectors}},
journal = {Proc. IEEE ICCP},
year = {2018},
}

Color Image Transient Image
Statue of David
Fiber
Pepsi Bottle
Fruit
Foam Box
Resolution Chart
Fiber – CVPR 2017

Related Projects

You may also be interested in related projects, where we have developed non-line-of-sight imaging systems:

  • Metzler et al. 2021. Keyhole Imaging. IEEE Trans. Computational Imaging (link)
  • Lindell et al. 2020. Confocal Diffuse Tomography. Nature Communications (link)
  • Young et al. 2020. Non-line-of-sight Surface Reconstruction using the Directional Light-cone Transform. CVPR (link)
  • Lindell et al. 2019. Wave-based Non-line-of-sight Imaging using Fast f-k Migration. ACM SIGGRAPH (link)
  • Heide et al. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (presented at SIGGRAPH) (link)
  • Lindell et al. 2019. Acoustic Non-line-of-sight Imaging. CVPR (link)
  • O’Toole et al. 2018. Confocal Non-line-of-sight Imaging based on the Light-cone Transform. Nature (link)

and direct line-of-sight or transient imaging systems:

  • Bergman et al. 2020. Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates. ICCP (link)
  • Nishimura et al. 2020. 3D Imaging with an RGB camera and a single SPAD. ECCV (link)
  • Heide et al. 2019. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Scientific Reports (link)
  • Lindell et al. 2018. Single-Photon 3D Imaging with Deep Sensor Fusions. ACM SIGGRAPH (link)
  • O’Toole et al. 2017. Reconstructing Transient Images from Single-Photon Sensors. CVPR (link)

 

Acknowledgements

The authors are grateful to Edoardo Charbon, Pierre-Yves Cattin, and Samuel Burri for providing the LinoSPAD sensor used in this work and continued support of it. O’Toole is supported by the Government of Canada through the Banting Postdoctoral Fellowships program, and Wetzstein is supported by a National Science Foundation CAREER award (IIS 1553333), a Terman Faculty Fellowship, a Sloan Fellowship, the Center for Automotive Research at Stanford (CARS), and by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant.