Time-Of-Flight Imaging
 

Time-of-flight cameras are becoming an increasingly important 3D sensing technology due to their low cost, wide availability and rapidly-increasing spatial resolution. Although current practice has been to treat the raw sensor output of such cameras as pixe-wise depth measurements, in reality the signal being sensed by these cameras is not directly proportional to depth. This is because light received at an individual sensor pixel may travel along many different paths from the sensor's light source to its pixels, with each path contributing its own time delay and intesity. Intepreting the combined effect of these paths as depth causes severe scene-specific 3D distorions that are hard or even impossible to correct at post-capture time. Moreover, this ignores the fact that there is actually a great deal more information in the sensed time-of-flight signals that can be extremely valuable for understanding real scenes---their reflectance, the way they transport light, etc.

Working with my student Matt O'Toole and colleages at UBC and the University of Bonn, we have developed a new mathematical framework for time-of-flight imaging with two basic goals: (1) a deeper understanding of the raw data captured by time-of-flight sensors, and (2) acquiring far richer scene data from time-of-flight (ToF) sensors by slightly modifying both their hardware and their imaging pipeline. This framework, introduced in our recent SIGGRAPH 2014 paper, showed for the first time that it is possible to express sinusoidally-modulated ToF light sources and sensors as complex-valued entities that always have a linear relation between them, no matter how complex the scene's light transport properties. This makes it possible to think of a ToF camera as a complex-valued generalization of a traditional camera, onto which practically any conventional vision algorithm or imaging technique can be brought to bare.

As an initial demonstration of this principle, in SIGGRAPH 2014 we applied our recent work on Structured Light Transport (CVPR 2014) to time-of-flight imaging, in order to demonstrate acquisition of direct-only ToF images. These images are guaranteed to be free of multi-path interference even in very general settings (eg. scenes with deep concavities and/or mirrors).

We also demonstarted an ability to parse light transport, ie. to capture sub-nanosecond-timescale videos of "light in flight" where the individual contributions of direct, specular indirect, diffuse indirect and retro-reflected light are identified and shown separately for a variety of complex desktop-sized scenes.

 
Defocus Deblurring and Superresolution for Time-of-Flight Depth Cameras.
 
Lei Xiao, Felix Heide, Matthew P. O'Toole, Matthias Hullin, Kiriakos N. Kutulakos and Wolfgang Heidrich
Proc. Computer Vision and Pattern Recognition Conf., Boston, MA, 2015.
Project page   PDF (10424KB)    IEEEXplore
 
Temporal Frequency Probing for 5D Transient Analysis of Light Transport.
 
Matthew P. O'Toole, Felix Heide, Lei Xiao, Matthias Hullin, Wolfgang Heidrich and Kiriakos N. Kutulakos
Proc. ACM Siggraph, 2014.
Project page   PDF (9848KB)    ACM Portal
 
 
Computational Light Transport
 
Light interacts with the environment in very complex ways: it scatters off of rough surfaces or when passing through translucent media, it refracts and reflects, it causes surfaces to mutually illuminate each other, it gets attenuated through a variety of physical processes, and it interacts with the camera lens to produce defocus blur. These complex interactions are a major element of visual appearance but have been widely considered too hard to analyze computationally. Starting around 2005, I published a series of papers on the theoretical and algorithmic foundations of analyzing light transport in several settings---from simple diffuse scenes (ICCV 2005) to complex transparent ones (ICCV 2005, IJCV 2008). This line of work provided a detailed understanding of what is possible to recover from images of general refractive and mirror-like scenes; it also showed that many important and seemingly-difficult problems related to light transport analysis have simple and very efficient solutions (SIGGRAPH Asia 2010, SIGGRAPH 2012, CVPR 2014).
 
Epipolar Time-of-Flight Imaging.
 
Supreeth Achar, Joseph Bartels, William L. 'Red' Whittaker, Kiriakos N. Kutulakos and Srinivasa Narasimhan
Proc. ACM Siggraph, 2017.
Oral.
PDF (6047KB)    ACM Portal
 
Computational Imaging on the Electric Grid.
 
Mark Sheinin, Yoav Schechner and Kiriakos N. Kutulakos
Proc. Computer Vision and Pattern Recognition Conf., Honolulu, Hawaii, 2017.
Oral.
Winner, Best Student Paper Award.
Project page   PDF (5254KB)   
 
The Geometry of First-Returning Photons for Non-Line-of-Sight Imaging.
 
Chia-Yin Tsai, Kiriakos N. Kutulakos, Srinivasa Narasimhan and Aswin Sankaranarayanan
Proc. Computer Vision and Pattern Recognition Conf., Honolulu, Hawaii, 2017.
Spotlight.
Project page   PDF (3179KB)   
 
3D Shape and Indirect Appearance by Structured Light Transport.
 
Matthew P. O'Toole, John Mather and Kiriakos N. Kutulakos
IEEE Trans. Pattern Anal. Machine Intell., vol. 38, no. 7, pp. 1298-1312, 2016.
Project page   PDF (3221KB)    IEEEXplore
 
Technical Perspective: The Dawn of Computational Light Transport.
 
Kiriakos N. Kutulakos
ACM Comm., 2016.
PDF (27KB)    ACM Portal
 
Homogeneous Codes for Energy-Efficient Imaging and Illumination.
 
Matthew P. O'Toole, Supreeth Achar, Srinivasa Narasimhan and Kiriakos N. Kutulakos
Proc. ACM Siggraph, 2015.
Project page   PDF (30891KB)    ACM Portal
 
3D Shape and Indirect Appearance by Structured Light Transport.
 
Matthew P. O'Toole, John Mather and Kiriakos N. Kutulakos
Proc. Computer Vision and Pattern Recognition Conf., Columbus, OH, 2014.
Oral.
Winner, Best Paper Honorable Mention.
Project page   PDF (3380KB)    IEEEXplore
 
Primal-Dual Coding to Probe Light Transport.
 
Matthew P. O'Toole, Ramesh Raskar and Kiriakos N. Kutulakos
Proc. ACM Siggraph, 2012.
Project page   PDF (1188KB)    ACM Portal
 
Frequency Analysis of Transient Light Transport with Applications in Bare Sensor Imaging.
 
Di Wu, Gordon Wetzstein, Christopher Barsi, Thomas Willwacher, Matthew P. O'Toole, Nikhil Naik, Qionghai Dai, Kiriakos N. Kutulakos and Ramesh Raskar
Proc. 12th European Conf. on Computer Vision, Florence, Italy, 2012.
Poster.
PDF (7657KB)   
 
Dynamic Refraction Stereo.
 
Nigel J. Morris and Kiriakos N. Kutulakos
IEEE Trans. Pattern Anal. Machine Intell., vol. 33, no. 8, pp. 1518-1531, 2011.
PDF (9073KB)    IEEEXplore
 
Optical Computing for Fast Light Transport Analysis.
 
Matthew P. O'Toole and Kiriakos N. Kutulakos
Proc. ACM Siggraph Asia, 2010.
Project page   PDF (55931KB)    ACM Portal
 
Transparent and Specular Object Reconstruction.
 
Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P. Lensch, Marcus A. Magnor and Wolfgang Heidrich
Computer Graphics Forum, vol. 29, no. 8, pp. 2400-2426, 2010.
PDF (1224KB)   
 
A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation.
 
Eron Steger and Kiriakos N. Kutulakos
Int. J. Computer Vision, vol. 76, no. 1, pp. 13-29, 2008.
David Marr Prize Special Issue.
Project page   PDF (2354KB)    SpringerLink
 
Photoconsistent Reconstruction of Semitransparent Scenes by Density-Sheet Decomposition.
 
Samuel W. Hasinoff and Kiriakos N. Kutulakos
IEEE Trans. Pattern Anal. Machine Intell., vol. 29, no. 5, pp. 870-885, 2007.
Project page   PDF (2002KB)    IEEEXplore
 
Reconstructing the Surface of Inhomogeneous Transparent Scenes by Scatter Trace Photography.
 
Nigel J. Morris and Kiriakos N. Kutulakos
Proc. 11th Int. Conf. on Computer Vision, Rio de Janeiro, Brazil, 2007.
Oral.
Project page   PDF (549KB)    IEEEXplore
 
A Theory of Inverse Light Transport.
 
Steven M. Seitz, Yasuyuki Matsushita and Kiriakos N. Kutulakos
Proc. 10th Int. Conf. on Computer Vision, Beijing, China, pp. 1440-1447, 2005.
Oral.
PDF (4050KB)    IEEEXplore
 
 
Computational Photography
 
Over the last few years, I have been very interested in developing multi-shot imaging techniques that can improve the performance of cameras equipped with conventional lenses. One particular class of such techniques arises from a novel re-thinking of the photographic process: Instead of capturing one photo of a subject, the photographer specifies some target constraints (e.g., total exposure time, the subject depth of field, etc.) and then lets the camera decide how many shots to take, what settings to use for each shot (focus, aperture, exposure time) and how to merge shots into one. This reduces photo acquisition to a family of optimal resource allocation problems that can be rigorously analysed and solved (ECCV 2008, ICCV 2009, PAMI 2012).

A key result stemming from this work was to establish the potential performance advantage of focal-stack photography: much research in computational photography has been devoted to designing new computational cameras that seek to boost optical performance. Our performance analysis in ICCV 2009 showed that by capturing an optimally-chosen focal stack, the conventional camera becomes competitive with the best of these designs, approaching fundamental optical bounds that cannot be breached.

More recently, I have been interested in the use of optical aberrations for photography and defocus analysis (ICCP 2013, ACCV 2012), as well on advanced sensing methods for acquiring "physically impossible" photos, that cannot be acquired in one shot with conventional imaging devices (see papers listed under light transport analysis and time-of-flight imaging).

 
Rolling Shutter Imaging on the Electric Grid.
 
Mark Sheinin, Yoav Schechner and Kiriakos N. Kutulakos
Proc. 10th Int. Conf. on Computational Photography, Pittsburgh, PA, 2018.
Oral.
Project page   PDF (11091KB)    IEEEXplore
 
Depth from Defocus in the Wild.
 
Huixuan Tang, , Brian Price, Stephen Schiller and Kiriakos N. Kutulakos
Proc. Computer Vision and Pattern Recognition Conf., Honolulu, Hawaii, 2017.
Poster.
Project page   PDF (152444KB)   
 
High Resolution Photography with an RGB-Infrared Camera.
 
Huixuan Tang, Xiaopeng Zhang, Shaojie Zhuo, Feng Chen, Kiriakos N. Kutulakos and Liang Shen
Proc. 7th Int. Conf. on Computational Photography, Houston, TX, 2015.
PDF (2341KB)    IEEEXplore
 
What Does an Aberrated Photo Tell Us about the Lens and the Scene?.
 
Huixuan Tang and Kiriakos N. Kutulakos
Proc. 5th Int. Conf. on Computational Photography, Boston, MA, 2013.
Oral.
Project page   PDF (1607KB)    IEEEXplore
 
Utilizing Optical Aberrations for Extended-Depth-of-Field Panoramas.
 
Huixuan Tang and Kiriakos N. Kutulakos
Proc. Asian Conf. on Computer Vision, 2012.
Oral.
PDF (18261KB)   
 
Light-Efficient Photography.
 
Samuel W. Hasinoff and Kiriakos N. Kutulakos
IEEE Trans. Pattern Anal. Machine Intell., vol. 33, no. 11, pp. 2203-2214, 2011.
PDF (3221KB)    IEEEXplore
 
Focal Stack Photography: High-Performance Photography with Conventional Cameras.
 
Kiriakos N. Kutulakos and Samuel W. Hasinoff
Proc. 11th Int. Conf. on Machine Vision and Applications, Yokohama, Japan, pp. 332--337, 2009.
Invited paper.
PDF (2549KB)   
 
Confocal Stereo.
 
Samuel W. Hasinoff and Kiriakos N. Kutulakos
Int. J. Computer Vision, vol. 81, no. 1, pp. 82-104, 2009.
Special Issue on ECCV 2006 Best Papers.
Project page   PDF (4104KB)    SpringerLink
 
Time-Constrained Photography.
 
Samuel W. Hasinoff, Kiriakos N. Kutulakos, Fredo Durand and William T. Freeman
Proc. 12th Int. Conf. on Computer Vision, Kyoto, Japan, pp. 333-340, 2009.
Oral.
Project page   PDF (4996KB)    IEEEXplore
 
A Layer-Based Restoration Framework for Variable-Aperture Photography.
 
Samuel W. Hasinoff and Kiriakos N. Kutulakos
Proc. 11th Int. Conf. on Computer Vision, Rio de Janeiro, Brazil, 2007.
Poster.
Project page   PDF (1399KB)    IEEEXplore
 
Plenoptic Image Editing.
 
Steven M. Seitz and Kiriakos N. Kutulakos
Int. J. Computer Vision, vol. 48, no. 2, pp. 115-129, 2002.
PDF (875KB)    SpringerLink
 
 
Non-Rigid 3D Shape from Video
 
The problem of markerless motion capture,---using video to reconstruct the instantaneous 3D shape and 3D motion of deformable surfaces (faces, skin, cloth, human bodies, etc.)---has been an active topic in vision and graphics for several years now. Back in 2001, my student Rodrigo Carceroni and I were the first to demonstrate the multi-view acquisition of detailed 3D shape and 3D motion fields for complex surfaces such as skin, faces and clothing. The underlying algorithm, called surfel sampling, introduced the use of the surfel representation for multi-view stereo and multi-view motion analysis and was able to handle subtle deformations, occlusions and several complex lighting effects (e.g., self-shadowing that changed over time, non-Lambertial reflectance, etc.)

More recently, working with Allan Jepson and Jonathan Taylor, we developed a general geometric framework for reconstructing non-rigid 3D shapes from just a single video sequence. Our CVPR 2010 paper on the topic was the first to demonstrate accurate and fully-automatic reconstruction for a wide range of non-rigid motions---including deforming cloth, tearing paper, faces, and multiple independently-deforming surfaces---within a single algorithmic framework.

 
Linear sequence-to-sequence alighnment.
 
Rodrigo L. Carceroni, Flavio L. Padua, G. A. Santos and Kiriakos N. Kutulakos
IEEE Trans. Pattern Anal. Machine Intell., vol. 32, pp. 304-320, 2010.
PDF (4103KB)    IEEEXplore
 
Non-Rigid Structure from Locally-Rigid Motion.
 
Jonathan Taylor, Allan D. Jepson and Kiriakos N. Kutulakos
Proc. Computer Vision and Pattern Recognition Conf., San Francisco, CA, 2010.
Oral.
Project page   PDF (5070KB)    IEEEXplore
 
Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion, Shape and Reflectance.
 
Rodrigo L. Carceroni and Kiriakos N. Kutulakos
Int. J. Computer Vision, vol. 49, no. 2-3, pp. 175-214, 2002.
PDF (3209KB)    SpringerLink
 
Multi-view 3D shape and motion recovery on the spatio-temporal curve manifold.
 
Rodrigo L. Carceroni and Kiriakos N. Kutulakos
Proc. 7th Int. Conf. on Computer Vision, Corfu, Greece, pp. 520-527, 1999.
Poster.
PDF (3344KB)    IEEEXplore
 
 
Multi-View Stereo
 
Even though it is intuitively clear that photographs provide information about the 3D geometry of the physical world, this fact has proved surprisingly hard to quantify in realistic settings. Motivated by this fact, my work in ICCV 1999 with Steve Seitz was the first to provide an answer to a hitherto-unresolved question: Suppose we have N input photos of an arbitrarily-shaped, static and opaque 3D scene taken from arbitrary points of view. To what extent do these photos determine the scenes 3D shape? By studying the class of all 3D shapes that reproduce the input photographs, we proved the existence of a special member of this class, the Photo Hull, that is the tightest possible bound on the scene that can be computed from the input photos. Importantly, we introduced a simple volumetric algorithm called Space Carving that provably computes this shape and demonstrated its performance on a variety of scenes that were beyond the reach of existing reconstruction algorithms.

In practice, calibration errors, noise, as well as the discrete nature of computational algorithms and representations imply that only approximations to the true scene can ever be recovered. Motivated by these difficulties, I extended the basic multi-view stereo framework to include a probabilistic interpretation of occupancy (ECCV 2002, with David Fleet) as well as the ability to recover approximate geometries in a coarse-to-fine fashion when calibration errors, noise and non-rigid motion prevent a perfect reconstruction (ECCV 2000).

 
A Probabilistic Theory of Occupancy and Emptiness.
 
Rahul Bhotika, David J. Fleet and Kiriakos N. Kutulakos
Proc. 7th European Conf. on Computer Vision, Copenhagen, Denmark, pp. 112-130, 2002.
Oral.
PDF (556KB)    SpringerLink
 
A Theory of Shape by Space Carving.
 
Kiriakos N. Kutulakos and Steven M. Seitz
Int. J. Computer Vision, vol. 38, no. 3, pp. 199-218, 2000.
David Marr Prize Special Issue.
PDF (882KB)    SpringerLink
 
Approximate N-View Stereo.
 
Kiriakos N. Kutulakos
Proc. 6th European Conf. on Computer Vision, Dublin, Ireland, pp. 67-83, 2000.
Oral.
PDF (438KB)    SpringerLink
 
 
Fast Algorithms for Computer Vision
 
 
TurboPixels: Fast superpixels using geometric flows.
 
Alex Levinstein, Adrian Stere, Kiriakos N. Kutulakos, David J. Fleet, Sven Dickinson and Kaleem Siddiqi
IEEE Trans. Pattern Anal. Machine Intell., vol. 31, pp. 2290-2297, 2009.
PDF (4789KB)    IEEEXplore
 
Fast computation of the Euclidian distance maps for binary images.
 
Kiriakos N. Kutulakos and Mihail N. Kolountzakis
Information Processing Letters, pp. 181-184, 1992.
ACM Portal
 
 
Augmented Reality
 
In the mid-90s I became interested in augmented reality, a then-nascent technology for overlaying live video with computer-generated graphics. Inspired by vision research on non-Euclidean representations of 3D space, I developed a technique that simplified the overlay process by formulating it entirely in an affine 3D frame of reference. This eliminated the multitude of calibration steps and 3D position sensing devices used in previous augmented reality systems without sacrificing accuracy. These ideas were put into use in a real-time augmented reality system that I developed in collaboration with James Vallino at the University of Rochester.
 
Calibration-Free Augmented Reality.
 
Kiriakos N. Kutulakos and James R. Vallino
IEEE Trans. Visualization and Computer Graphics, vol. 4, no. 1, pp. 1-20, 1998.
PDF (3118KB)    IEEEXplore
 
 
View Selection, Visual Exploration and Active Vision
 
Humans are extremely good at picking up, manipulating, and inspecting objects even when these objects have a very complicated geometry. For instance, if we are handed a complicated object (e.g., an abstract art sculpture with many holes, etc) and are asked to visually scan its exposed surfaces, the task of ensuring that these surfaces are viewed completely is usually effortless, even for objects seen for the very first time. Moreover, we think of many objects as having a set of preferred vantage points (e.g., an upright view of a human figurine, a top or a side view of a cup) that we commonly use when viewing them. My thesis research grew out of my fascination with these intuitive---yet very hard for formalize---observations and was centered around a single question: how can we develop automatic object-scanning algorithms that move a camera around an object in order to reconstruct the objects entire surface? Using results from on-line path planning research in robotics and from the differential geometry and topology of smooth surfaces, I gave an answer to this question for smooth objects of arbitrary shape.

A key contribution of my thesis and early post-doctoral work was to show that by combining a small number of simple steps---moving on a line (ICRA 1994), rotating the object on a single axis (AIJ 1995), detecting its silhouette and its occluding contour (IJCV 1992)---one can provably scan an objects surface regardless of its geometric complexity. In fact, even though my algorithms were firmly rooted in geometry, the resulting motions bore striking similarity to motions that we, as humans, perform every day---turning an object, seeing through holes, moving to a preferred view.

 
Global surface reconstruction by purposive control of observer motion.
 
Kiriakos N. Kutulakos and Charles R. Dyer
Artificial Intelligence Journal, pp. 147-177, 1995.
Oral.
ACM Portal
 
Recovering shape by purposive viewpoint adjustment.
 
Kiriakos N. Kutulakos and Charles R. Dyer
Int. J. Computer Vision, vol. 12, no. 2-3, pp. 113-136, 1994.
Oral.
SpringerLink
 
Provable strategies for vision-guided exploration in three dimensions.
 
Kiriakos N. Kutulakos, Charles R. Dyer and Vladimir J. Lumelsky
Proc. Int. Conf. Robotics and Automation, San Diego, CA, vol. 2, pp. 1365-1372, 1994.
Oral.
PDF (1207KB)    IEEEXplore
 
Occluding contour detection using affine invariants and purposive viewpoint control.
 
Kiriakos N. Kutulakos and Charles R. Dyer
Proc. Computer Vision and Pattern Recognition Conf., Seattle, WA, pp. 323-330, 1994.
Oral.
Winner, Siemens Best Student Paper Award.
PDF (371KB)    IEEEXplore
 
 
Site last modified on Monday, April 29, 2019
Send questions or comments about this page to Kyros Kutulakos