LiFF: Light Field Features in Scale and Depth | CVPR 2019

Donald G. Dansereau, Bernd Girod, Gordon Wetzstein

An efficient 4D light field feature detector and descriptor along with a large-scale 4D light field dataset.

ABSTRACT

Feature detectors and descriptors are key low-level vision tools that many higher-level tasks build on. Unfortunately these fail in the presence of challenging light transport effects including partial occlusion, low contrast, and reflective or refractive surfaces. Building on spatio-angular imaging modalities offered by emerging light field cameras, we introduce a new and computationally efficient 4D light field feature detector and descriptor: LiFF. LiFF is scale invariant and utilizes the full 4D light field to detect features that are robust to changes in perspective. This is particularly useful for structure from motion (SfM) and other tasks that match features across viewpoints of a scene. We demonstrate significantly improved 3D reconstructions via SfM when using LiFF instead of the leading 2D or 4D features, and show that LiFF runs an order of magnitude faster than the leading 4D approach. Finally, LiFF inherently estimates depth for each feature, opening a path for future research in light field-based SfM.

FILES

CITATION

D. Dansereau, B. Girod, G. Wetzstein, “LiFF: Light Field Features in Scale and Depth”, CVPR 2019.

BibTeX

@article{Dansereau:2019:LiFF,
author = {D. Dansereau and B. Girod and G. Wetzstein},
title = {LiFF: Light Field Features in Scale and Depth},
journal = {In Proc. CVPR},
year = {2019}
}

4D LIGHT FIELD DATASET

We’re happy to introduce two new multi-view light field datasets. The datsets were captured with hand-held Lytro Illum cameras, and unlike previous datasets each scene is photographed from a diversity of camera poses.

This data emulates the case of a mobile platform employing a lenslet-based light field camera, or multiple light field cameras operating simultaneously. The individual light fields have small baselines, on the order of a centimeter, while the camera poses vary over a broader baseline, on the order of a meter or more.

These are the first such datasets to our knowledge, and it is our hope that they will enable research into multi-view light field processing including registration, self-calibration, structure-from-motion, interpolation, and feature extraction.

Scenes include examples of Lambertian and non-Lambertian surfaces, occlusion, specularity, subsurface scattering, fine detail, and transparency.

You can find more information and download the data from: http://lightfields.stanford.edu/mvlf/

(left) One of five views of a scene that COLMAP’s structure-from-motion (SfM) solution fails to reconstruct using SIFT, but successfully reconstructs using LiFF; (right) LiFF features have well-defined scale and depth, measured as light field slope, revealing the 3D structure of the scene – note we do not employ depth in the SfM solution.
Comparison to SIFT: Features identified only by LiFF, only by SIFT, and by both are shown in green, red, and blue respectively. (top) LiFF rejects spurious features in low-contrast areas and to some extent those distorted through refraction; (center) LiFF rejects spurious features at occlusion boundaries – the inset highlights a SIFT-only detection caused by leaves at different depths; (bottom) LiFF detects partially occluded features missed by SIFT – note the increasing proportion of LiFF-only features toward the back of the scene, and the LiFF-only detections highlighted in the inset.
3D Scene Shape: In this work we establish LiFF’s ability to deliver more informative features by virtue of higher selectivity and ability to image through partial occlusion. We expect LiFF’s slope estimates will also be of substantial interest. Here we see the 3D shape of each scene revealed through the slopes of the detected LiFF features.