Looking Around Corners

Back to the main page.

Publication: Recovering Three-dimensional Shape Around a Corner using Ultrafast Time-of-Flight Imaging

Abstract:The recovery of objects obscured by scattering is an important goal in imaging and has been approached by exploiting coherence properties, ballistic photons or penetrating wavelengths, for example. Common methods use scattered light transmitted through an occluding material, although these fail if the occluder is opaque. Light is scattered by transmission through objects, but also by multiple reflection from diffuse surfaces in a scene. This reflected light contains information about the scene that becomes mixed by the diffuse reflections before reaching the image sensor. This mixing is difficult to decode using traditional cameras. Here we show the combination of a time-of-flight technique and computational reconstruction algorithms to untangle image information mixed by diffuse reflection. We demonstrate a 3D range camera able to look around a corner using diffusely reflected light that achieves sub-millimeter depth precision and centimeter lateral precision over 40 cm x 40 cm x 40 cm of hidden space.


Video on the Media Lab Labcast

Concept Artwork

Sketches by Tiago Allen, Vector graphics version is here

Cartoon describing concept

Cartoon describing concept

Cartoon describing concept

Toy example:
Revealing playing card

Applications: Robot
navigation and endoscopy

Application:
Search and rescue

Project Images

Experimental setup by Andreas Velten, photos taken by Christopher Barsi

Hidden object

Setup with hidden
object

The Laser

Streak Camera Image

Labeled image of the setup

Additional photos of the setup taken by Christopher Barsi

Videos

Created by Christopher Barsi and Andreas Velten
The Setup
The laser scanning the wall.
The Camera
The Camera (front view)
Computational reconstruction in progress while scanning the wall.
Combined video


Media Lab Labcast


Slideshow describing the method

Team

Photos by Everett Lawson

Andreas Velten (left)
and Ramesh Raskar

Andreas Velten and
Ramesh Raskar

Publication Figures

Images of the reconstructions and technical descriptions as a ppt

The experimental setup
Figure 1: Experimental Setup (a) The capture process. We capture a series of images by sequentially illuminating a single spot on the wall with a pulsed laser and recording an image of the dashed line segment on the wall with a streak camera. The laser pulse travels a distance r1 to strike the wall at a point L, some of the diffusely scattered light strikes the hidden object (for example at s after traveling a distance r2), returns to the wall (for example at w, after traveling over r3) and is collected by the camera after traveling the final distance r4 from w to the camera center of projection. The position of the laser beam on the wall is changed by a set of galvanometer actuated mirrors. (b) An example of streak images sequentially collected. Intensities are normalized against a calibration signal. Red corresponds to the maximum, blue to the minimum intensities. (c) The 2D projected view of the 3D shape of the hidden object, as recovered by the reconstruction algorithm. Here the same color map corresponds to backprojected filtered intensities or confidence values of finding an object surface at the corresponding voxel.

An example of a streak camera image
Figure 2: Streak image with calibration spot The calibration spot in a streak image (highlighted with an arrow). The calibration spot is created by an attenuated beam split off the laser beam that strikes the wall in the field of view of the camera. It allows to monitor the long term stability of the system and calibrate for drifts in timing synchronization.

An instructional reconstruction of a single hidden patch
Figure 3: Reconstruction Algorithm An illustrative example of geometric reconstruction using streak camera images. (a) Data capture. The object to be recovered consists of a 2 cm × 2 cm size square white patch beyond the line of sight (i.e. hidden). The patch is mounted in the scene and data is collected for different laser positions. The captured streak images corresponding to three different laser positions are displayed in the top row. Shapes and timings of the recorded response vary with laser positions and encode the position and shape of the hidden patch. (b) Contributing voxels in Cartesian space. For recovery of hidden position, consider the choices of contributing locations. The possible locations in Cartesian space that could have contributed intensity to the streak image pixels p, q, r are the ellipses p , q , r (ellipsoids in 3D). For illustration, these three ellipse sections are also shown in (a) bottom left in Cartesian coordinates. If there is a single world point contributing intensity to all 3 pixels, the corresponding ellipses intersect, as is the case here. The white bar corresponds to 2 cm in all subfigures. (c) Backprojection and heatmap. We use a back-projection algorithm that finds overlayed ellipses corresponding to all pixels, Here we show summation of elliptical curves from all pixels in the first streak image. (d) Backprojection using all pixels in a set of 59 streak images. (e) Filtering. After filtering with a second derivative, the patch location and 2 cm lateral size are recovered.

A reconstruction of a complex object
Figure 4: Depth in Reconstructions Demonstration of the depth and lateral resolution. (a) The hidden objects to be recovered are three letters, I, T, I at varying depths. The "I" is 1.5 cm in wide and all letters are 8.2 cm high. (b) 9 of 60 images collected by the streak camera. (c) Projection of the heatmap on the x-y plane created by the back projection algorithm. (d) Filtering after computing second derivative along depth (z). The color in these images represents the confidence of finding an object at the pixel position. (e) A rendering of the reconstructed 3D shape. Depth is color coded and semi-transparent planes are inserted to indicate the ground truth. The depth axis is scaled to aid visualization of the depth resolution.

A reconstruction of letters
Figure 5: Complex Object Reconstruction (a) Photo of the object. The mannequin is approximately 20 cm tall and is placed about 25 cm from the diffuser wall. (b) Nine of the 60 raw streak images. (c) Heatmap. Visualization of the heatmap after backprojection. The maximum value along the z direction for each x-y coordinate in Cartesian space. The hidden shape is barely discernible. (d) Filtering. The second derivative of the heatmap along depth (z) projected on the x-y plane reveals the hidden shape contour. (e) Depth map. Color encoded depth (distance from the diffuser wall) shows the left leg and right arm closer in depth compared to the torso and other leg and arm. (f) Confidence map. A rendered point cloud of confidence values after soft threshold. Images (g-h) show the object from different viewpoints after application of a volumetric blurring filter. (i) The stop-motion animation frames from multiple poses to demonstrate reproducability. See Supplementary Movie 1 for an animation. Shadows and the ground plane in images (f-i) have been added to aid visualization.

Documents and Related Work

The paper at Nature Communications(Subscription required)
Nature Video about the paper
Prepublished manuscript. Nature Communications, March 20, 2012
Original tech report
Prepublished related manuscript: arXiv, March 19, 2012
Slideshow describing the method
Related earlier publication: Estimating Motion and Size of Moving Non-Line-of-Sight Objects in Cluttered Environments

Media Coverage

MIT News
A camera that peers around corners article with video
USA Today
'See-around-a-corner' camera prototype unveiled article
Discovery News
Next Superpower: Seeing Around Corners article
Engadget
MIT's laser-powered camera can detect objects hidden around corners (video) article
MSNBC
At MIT, they find a way to see around corners article
New Scientist
Superfast laser camera peers around corners article
Handelsblatt (Germany)
Kamera schaut um die Ecke article with video
Focus (Germany)
Laserkamera kann um die Ecke fotografieren article with video
Wired (UK)
MIT camera uses lasers to capture images from around corners article
LiveScience.com
Why Seeing Around Corners May Become Next 'Superpower' article
Spectrum der Wissenschaft (Germany)
Um die Ecke geschaut article with video