Time Multiplexed Coded Apertures | ICCV 2021

Edwin Vargas*, Julien N.P. Martel*, Gordon Wetzstein, Henry Arguello

TMCA is a new type of codification for compressive imaging systems using coded apertures. It enables better reconstruction quality due to a better conditioning of the systems’ sensing matrices.

ABSTRACT

Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot. The performance of compressive imaging systems based on CAs mostly depends on two factors: the properties of the mask’s attenuation pattern, that we refer to as “codification”, and the computational techniques used to recover the quantity of interest from the coded snapshot. In this work, we introduce the idea of using time-varying CAs synchronized with spatially varying pixel shutters. We divide the exposure of a sensor into sub-exposures at the beginning of which the CA mask changes and at which the sensor’s pixels are simultaneously and individually switched “on” or “off”. This is a practically appealing codification as it does not introduce additional optical components other than the already present CA but uses a change in the pixel shutter that can be easily realized electronically. We show that our proposed time-multiplexed coded aperture (TMCA) can be optimized end to end and induces better coded snapshots enabling superior reconstructions in two different applications: compressive light field imaging and hyperspectral imaging. We demonstrate both in simulation and with real captures (taken with prototypes we built) that this codification outperforms the state-of-the-art compressive imaging systems by a large margin in those applications.

FILES

CITATION

E. Vargas*, J. N. P. Martel*, G. Wetzstein, H. Arguello, Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems, Intl. Conf. on Computer Vision (ICCV), 2021

@article{vargas2021tmca,
author = {Vargas, Edwin and Martel, Julien N.P. and Wetzstein, Gordon and Arguello, Henry},
title = {Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems},
journal = {Intl. Conf. on Computer Vision (ICCV)},
year={2021}
}

Results on Compressive Hyperspectral Imaging

Examples of compressive spectral imaging reconstructions in simulation. The PSNR (dB) between the reconstructions and the ground truth images is shown in the lower-right corner.

Results on Compressive Light-field Imaging

Real captures of coded snapshots for light fields comparing CLFP [35] (see paper) and the proposed TMCA with their reconstructions.
TMCA Encoder-Decoder architecture. An illustration of the proposed Time Multiplexed Coded Aperture (TMCA) codification in the proposed end-to-end differentiable pipeline. We also show coded snapshot and reconstruction examples for our compressive light field and hyperspectral imaging applications.
The two systems we developed for the two applications of TMCA. A diagram of the ray optics for our light field imaging (left) and hyperspectral imaging (right) systems using TMCAs.
TMCA yields better conditionned sensing matrices. A plot of the distribution of eigenvalues sorted in descending order for the compressive light field (left) and compressive spectral imaging application (right) systems. Each graph shows two distributions: for K=1, which is the traditional coded aperture system, and K=8, which is TMCA using 8 shutter slots. Those plots illustrate that TMCA is better conditioned in both applications as the ratio between the highest and lowest eigenvalues is lower in the case of TMCA (K=8).