Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas
We introduce an interferometric technique for passive time-of-flight imaging and depth sensing at micrometer axial resolutions. Our technique uses a full-field Michelson interferometer, modified to use sunlight as the only light source. The large spectral bandwidth of sunlight makes it possible to acquire micrometer-resolution time-resolved scene responses, through a simple axial scanning operation. Additionally, the angular bandwidth of sunlight makes it possible to capture time-of-flight measurements insensitive to indirect illumination effects, such as interreflections and subsurface scattering. We build an experimental prototype that we operate outdoors, under direct sunlight, and in adverse environment conditions such as machine vibrations and vehicle traffic. We use this prototype to demonstrate, for the first time, passive imaging capabilities such as micrometer-scale depth sensing robust to indirect illumination, direct-only imaging, and imaging through diffusers.
Existing contactless imaging techniques for micron-scale 3D sensing, such as interferometry and microscopy, require active illumination, most commonly from a laser or other coherent sources. This requirement makes these techniques impractical for use outdoors, in the presence of strong ambient illumination that overwhelms the active light source, or in power-constrained applications.
We change this state of affairs by introducing a completely passive, interferometric micron-scale 3D sensing technique using sunlight as the only light source, to capture full-frame time-of-flight measurements at axial resolutions of 5 micrometers. The technique takes advantage of the spatial incoherence of sunlight, to enable robust 3D sensing in the presence of severe indirect illumination effects (interreflections, subsurface scattering), and even tasks such as imaging and 3D sensing through optically-thin scattering layers.
More broadly, our findings open the door for the deployment of interferometric techniques in uncontrolled outdoor environments, and for the development of passive computational light transport capabilities such as direct-only imaging, imaging through scattering, and transient imaging
Sunlight, and thus the illumination injected in the interferometer, is temporally and spatially incoherent. Temporal incoherence means that the illumination is broadband in spectrum, i.e., comprises multiple independent waves propagating with different wavelengths. Spatial incoherence means that the illumination comprises multiple planar wavefronts propagating along directions with different angular offsets from the optical axis. This is accurate for sunlight, as the Sun is a far-field area emitter subtending (when observed from the Earth) a small solid angle.
We show that, due to the incoherence of sunlight, we measure non-zero interference signal only in those parts of the image where the position of the reference mirror matches the scene depth. This measurement is the approximately equal to the direct-only transient response of the scene. Similar to Gkioulekas et al., we can use these measurements to reconstruct a micrometer-scale depth map of the scene.
We further quantify the spatial and axial resolution of the reconstructed depth by measuring the spatial and temporal coherence lengths. The temporal coherence length is of the order of 10 microns, and the spatial coherence length is of the order of 100 microns. This validates our claim of micron-resolution depth sensing.
We build an experimental prototype to demonstrate the capabilities of sunlight interferometry. Our prototype uses a camera equipped with a CCD sensor and a compound lens for imaging, and build a tracking assembly comprised of a mirror whose 3D orientation is controlled via two motorized rotation stages to redirects sunlight along the optical axis of the interferometer.
To perform experiments outdoors and mitigate the vibration effects, we build our setup on an optical breadboard mounted on a utility cart. The imaged scenes are on the ground or a tripod, and are fully immersed in ambient light. There are strong sources of environment noise, including moving vehicles, air, and ambient sound, conditions traditionally considered to preclude the use of interferometry. This is in stark contrast with prior works on interferometric computational imaging such as Gkioulekas et al., which require carefully-controlled environments (e.g., dark room, vibration-isolated optical tables, no air flow).
We performed most of our experiments under direct sunlight. However, we found that under scattered or very high-altitude clouds, we are still able to accurately track the sun and obtain high-quality depth, even if the clouds are fast-moving.
Our results emphasize the micron-scale depth detail our method can provide, competitive with setups in carefully-controlled lab conditions such as in Gkioulekas et al.
Our technique captures a video showing very faint speckle changes. From these, it computes the direct-only transient response and depth of the scene.
Here, we show scans for variety of scenes. These feature micrometer-scale geometric details, and pose a variety of challenges including low reflectivity (circuit board), specularities (coin, pawn) and strong sub-surface scattering (chocolate, soap, pill). We obtain high quality reconstructions, despite operating under very adverse environment conditions.
Besides depth, our technique also produces the direct-only transient response of scanned scenes, using only sunlight illumination.
A key challenge in sunlight interferometry is the very low SNR in captured images. To test the SNR limits of our technique, we scan a semi-transparent object (gummy bear) that backreflects very little light. Despite this, our technique still acquires accurate depth for most of the scene, except for parts observed at near-grazing angles (no backreflected light).
To test the ability of our technique to isolate direct-only (ballistic) light and measure multiple depth peaks, we scan a scene where a coin is occluded from the camera by a ground glass diffuser. The conventional images of the scene show strong blur because of the diffuser. Our technique can accurately acquire the depth and clean images of the occluded coin.
Even though our focus is on sunlight interferometry, we show an experiment testing the ability to perform time-of-flight imaging with passive indoor illumination. For this, we point the Sun tracking module of our setup to a ceiling light about five feet from the tracking mirror. We use as scene the same metallic coin as in the sunlight results, which backreflects a lot of light, resulting in accurate depth recovery.
For an in-depth description of the technology behind this work, please refer to our CVPR 2023 paper. A pre-print and the supplementary material can be found here.
Alankar Kotwal, Anat Levin and Ioannis Gkioulekas. "Passive Micron-scale Time-of-Flight with Sunlight Interferometry", CVPR 2023 (highlight!)
We thank Sudershan Boovaraghavan and Yuvraj Agrawal, who provided the samples for some of our experiments. This work was supported by NSF awards 1730147, 2047341, 2008123 (NSF-BSF 2019758), and a Sloan Research Fellowship for Ioannis Gkioulekas.