Course Description

From the advanced depth cameras in today's smartphones to the 3D sensors of next-gen autonomous vehicles, 3D sensing has already began to revolutionize the way we sense the world. This, however, is just the beginning: cameras that can see in pitch-black darkness and around corners, depth sensors with sub-micron accuracy or a range of several kilometers, and cameras that can see under the skin are already becoming a reality. Key to these developments is the rapid convergence of optics, photonics, sensing and computation.

This seminar course will provide an in-depth look at 3D sensing and related technologies, and the computational techniques used to model and exploit them. Specific topics will include "extreme imaging" with single-photon detectors; principles of time-of-flight imaging and LiDAR; structured-light imaging; 3D imaging through scattering media; and imaging around corners. Class format will be discussion based, with approximately one paper covered per week and a term project.

Recommended preparation: Graduate-level exposure to one or more of computer vision, computer graphics, sensing or numerical optimization is desirable but not required. Prior enrollment in CSC2529 (Computational Imaging) is not necessary.

Teaching Assistant

Course Logistics

Lectures: Wednesdays 4:00am-6:00pm in SF3201.

Instructor in-person office hour (starting Jan 17): Wednesdays noon-1:00pm (BA7270).
Instructor zoom office hour: Tuesdays 3:30-4:30pm (zoom link posted on Quercus).

Contact: Course announcements and general information will be posted Quercus. Q&A related to papers discussed in lecture will be on Piazza.

Course dropbox: The one-stop shop for all course materials. See Quercus for the link.

Role assignments & schedule: See Quercus for the link to the google sheet with all the info.

Coursework

Role playing participation (60%)

Marks will be distributed equally across all lectures (5.4% per lecture). All role-specific written components are due at midnight before the paper is discussed in class (ie. Tue 11:59pm)

Term project (40%)

Project proposal is due February 14 (5% of mark). Final report is due at the end of classes (35% of mark).

Late policy

All homework is due at midnight on the due date. For role-specific homework, there will be a 30% marks deduction if you submit late, but before the start of that week's lecture (i.e., if you submit anytime between 12:01am and 4pm on Wednesday). No such homework will be accepted after the lecture's start since the homework's intent is to get your thoughts before any in-class discussions have occurred.

If you need more time to submit your project proposal or final project report, you will need to discuss your timeline with the instructor and get approval at least 1 week before the posted due date.

Lecture Schedule

Date Topic Paper(s) Event
Wed
10/1
Introduction
Course overview, role-playing class format, course components, grading, etc.
Part I: Active Sensing using Classical 3D Vision Cues
Wed
17/1
Laser
Triangulation
Main paper
Levoy et al., The digital Michelangelo project: 3D scanning of large statues (Proc. SIGGRAPH 2000).

Essential background
  • Curless, New methods for surface reconstruction from range images (PhD thesis). Chapters 1-2.

  • To probe further:
  • Curless, New methods for surface reconstruction from range images (PhD thesis). Chapters 3-4.
  • Curless et al., Better optical triangulation through spacetime analysis (Proc. ICCV 1995).
  • Wed
    24/1
    Structured-Light
    Triangulation
    Main paper
    Gupta & Nayar, Micro Phase Shifting (Proc. CVPR 2012)

    To probe further:
  • Nayar et al., Fast separation of direct and global components of a scene using high frequency illumination (Proc. SIGGRAPH 2006).
  • Szeliski & Scharstein, High-accuracy stereo depth maps using structured light (Proc. CVPR 2003).
  • Salvi et al., Pattern codification strategies in structured light systems (Pattern Recognition, 2004).
  • Zhang et al., Rapid shape acquisition using color structured light and multi-pass dynamic programming (Proc. 3DPVT 2002).
  • Wed
    31/1
    Photometric
    Stereo
    Main paper
    Johnson et al, Microgeometry capture using an elastomeric sensor (Proc. SIGGRAPH 2011).

    To probe further:
  • Hernandez et al., Non-rigid Photometric Stereo with Colored Lights (Proc. ICCV 2007).
  • Wed
    7/2
    Polarization
    Imaging
    Main paper
    Kadambi et al, Depth Sensing Using Geometrically Constrained Polarization Normals (IJCV 2017).

    To probe further:
  • Baek et al., Simultaneous acquisition of polarimetric SVBRDF and normals (SIGGRAPH Asia 2018).
  • Part II: Time-Resolved 3D Sensing
    Wed
    14/2
    Single-Photon
    LiDAR
    Main paper
    Kirmani et al., First-Photon Imaging (Science 2014).
    Project proposals due at 11:59pm

    To probe further:
  • Gupta et al., Asynchronous Single-Photon 3D Imaging (Proc. ICCV 2019).
  • Wed
    21/2
    Winter break (No Lecture)
    Wed
    28/2
    Indirect
    ToF
    Main paper
    Gupta et al., What Are Optimal Coding Functions for Time-of-Flight Imaging? (ACM TOG 2018).

    To probe further:
  • Lange & Seitz, Solid-state time-of-flight range camera (IEEE Trans. Quantum Electronics, 2001).
  • Achar et al., Epipolar Time of Flight (Proc. SIGGRAPH 2017).
  • Wed
    6/3
    Coherent
    ToF
    Main paper
    Kadambi et al, Rethinking Machine Vision Time of Flight With GHz Heterodyning (IEEE Access 2017).

    To probe further:
  • Behroozpour et al., Lidar system architectures and circuits (IEEE Communications Mag. 2017).
  • Wed
    13/3
    Non-Line-of-Sight 3D
    Imaging
    Main paper
    Velten et al, Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging (Nature Communications, 2012).

    To probe further:
  • Velten et al., Femto-photography: capturing and visualizing the propagation of light (SIGGRAPH 2013).
  • Wed
    20/3
    3D Imaging
    Through Scatter
    Main paper
    Zhao et al., High Resolution, Deep Imaging Using Confocal Time-of-Flight Diffuse Optical Tomography (IEEE PAMI 2021).

    To probe further:
  • Bertolotti et al, Non-invasive imaging through opaque scattering layers (Nature 2012).
  • Wed
    27/3
    Interferometric
    ToF
    Main paper
    Kotwal et al, Passive Micron-scale Time-of-Flight with Sunlight Interferometry (CVPR 2023).

    To probe further:
  • Gkioulekas et al, Micron-Scale Light Transport Decomposition Using Intererometry (SIGGRAPH 2015).
  • Wed
    3/4
    Ultra-Wideband
    Single-Photon Imaging
    Main paper
    Wei et al., Passive Ultra-wideband Single-Photon Imaging (ICCV 2023).
    Fri
    5/4

    Final project reports due at 11:59pm

    Additional Information

    Related courses at UofT