Course Description

Computational imaging systems have a wide range of applications in consumer electronics, scientific imaging, HCI, medical imaging, microscopy, and remote sensing. We discuss digital photography and basic image processing, convolutional neural networks for image processing, denoising, deconvolution, single pixel imaging, inverse problems in imaging, proximal gradient methods, introduction to wave optics, time-of-flight imaging, end-to-end optimization of optics and imaging processing. Emphasis is on applied image processing and solving inverse problems using classic algorithms, formal optimization, and modern artificial intelligence techniques. Students learn to apply material by implementing and investigating image processing algorithms in Python and completing a term project.

Topics include:

  • Human visual perception
  • Digital cameras and ISPs
  • Denoising, deconvolution, and other inverse problems in imaging
  • Convolutional neural networks for solving inverse problems
  • Proximal gradient methods / formal optimization for solving inverse problems
  • High dynamic range imaging
  • Light field imaging
  • Time-of-flight imaging, non-line-of-sight imaging
  • End-to-end optimization of optics and image processing
  • ... other interesting topics.

Instructor

Course Logistics

Lectures: Monday 10:00am-12:00pm in Galbraith Building 120. Lectures will be recorded and made available on Quercus.

Instructor office hours: Monday 1:30-2:30pm (BA7228) starting on September 12; discussion about projects, material, etc.

TA office hours: Tuesdays and Fridays in Bahen 3201 from 12:00-1:30pm starting on September 16; discussion about homework questions

Problem sessions: Wednesday 11:00am-12:00pm (SS1071) starting on September 14. Problem sessions will be recorded and made available on Quercus.

Textbook: There is no textbook for the course; links to readings and course notes are provided in the schedule.

Contact: Course announcements and general information will be posted on the course forum at Ed Discussion. For any emergency inquiries that cannot be handled on the forum via a private post, email the staff listserv: csc2529-fall2223-staff-l@listserv.utoronto.ca.

Coursework

Helpful Background

Related courses

Gradescope

All assignments and the final project proposal and report should be submitted on Gradescope using entry code MXDXRM. If you work as a team, make sure to indicate your team member in the submission.

Assignments (50%)

There will be 6 homework assignments (see schedule) in this class. These assignments will contain some theoretical questions and also implementations of techniques that we will discuss in class. Please refer to assignment writeups (available in the Files tab on Quercus) for details. After you finish, submit your code and report on Gradescope.

There are no "late days" for the assignments. If you choose to submit an assignment late, we will accept it for up to 24h after the submission deadline with a 30% penalty (final grade multiplied by 0.7).

Collaboration Policy: Students are permitted to work together on homework assignments in small groups. However, while students can discuss the assignments together, they must compose their solutions individually. Students must not view or copy code of other students or solutions available on the internet. Names of collaborators must be listed on submitted assignments. Submitting plagiarized solutions is an academic offense and can have severe penalties.

Final Project (50%)

The final project grade takes into account your poster presentation (organization of poster, clarify of presentation, ability to answer question), your source code submission (code organization and documentation), and your final project report (appropriate format and length, abstract, introduction, related work, description of your method, quantitative and qualitative evaluation of your method, results, discussion & conclusion, bibliography).

You can work in teams of up to 3 students for the project. Submit only one proposal and final report for each team. The expected amount of work is relative to the number of team members, so if two teams work on a similar project, we'd expect less work from a smaller team. Before you start to work on the proposal or the report, take a look at some of the past project proposals and reports to give you sense for what's expected (see link at the bottom of this page).

The project proposal is a 1-2 page document that should contain the following elements: clear motivation of your idea, a discussion of related work along at least 3 scientific references (i.e., scientific papers not blog articles or websites), an overview of what exactly your project is about and what the final goals are, milestones for your team with a timeline and intermediate goals. Once you send us your proposal, we may ask you to revise it and we will assign a project mentor to your team.

The final project report should look like a short (~6 pages) conference paper. We expect the following sections, which are standard practice for conference papers: abstract, introduction, related work, theory (i.e., your approach), analysis and evaluation, results, discussion and conclusion, references. To make your life easier, we provide an LaTex template that you can use to get started on your report (see schedule for link).

Schedule and Syllabus

Week Date Description Material Readings Event Deadline
Week 1 Mon
12/9
Lecture 1: Course intro, human visual system
Overview of class, human perception of color, depth, contrast, resolution
[slides]
Hybrid Images Paper HW1 out on Quercus
Wed
14/9
Problem session (HW1) [slides]
Fri
16/9
TA Office Hours (HW1)
Week 2 Mon
19/9
Lecture 2: Digital photography I
Ray optics, aperture, depth of field, exposure, sensor, noise
[slides] Marc Levoy's course on digital photography HW2 out on Quercus
Tue
20/9
TA Office Hours (HW1)
Wed
21/9
HW1 due at 11:59pm
Week 3 Mon
26/9
Lecture 3: Digital photography II
CameraISP, demosaicking, denoising, deconvolution
[slides] Demosaicking Paper
Non-local Means Paper
Intro to Bilateral Filtering
Tue
27/9
TA Office Hours (HW2)
Wed
28/9
Problem session (HW2) [slides]
Fri
30/9
TA Office Hours (HW2)
Week 4 Mon
3/10
Lecture 4: Math review
Quick review of sampling, optimization, deconvolution, ...
[slides] HW3 out on Quercus
Wed
5/10
Problem session (HW3) [slides] HW2 due at 11:59pm
Fri
7/10
TA Office Hours (HW3)
Week 5 Mon
10/10
Thanksgiving Day (No Lecture)
Week 6 Mon
17/10
Lecture 5: Great ideas in computational photography
HDR, tone mapping, coded apertures, flutter shutter
[slides] HDR Imaging Paper
Tone Mapping Paper
Ext. Depth of Field Paper
Flutter Shutter Paper
Learned Coded Apertures
Neural Sensors
HW4 out on Quercus
Tue
18/10
TA Office Hours (HW3)
Wed
19/10
Problem session (HW4) [slides] HW3 due at 11:59pm
Fri
21/10
TA Office Hours (HW4)
Week 7 Mon
24/10
Lecture 6: Introduction to neural networks
MLPs, CNNs, ResNets, denoising/deconvolution with CNNs
[slides] Linear classifiers Backpropagation notes
Optimization notes
HW5 out on Quercus
Tue
25/10
TA Office Hours (HW4)
Wed
26/10
Problem session (HW5) [slides] HW4 due at 11:59pm
Fri
28/10
TA Office Hours (HW5)
Week 8 Mon
31/10
Lecture 7: Solving regularized inverse problems with ADMM
Natural image priors, deconvolution, single-pixel imaging, ADMM, solving general inverse problems
[slides] Deconvolution notes
Compressive imaging notes
HW6 out on Quercus
Tue
1/11
TA Office Hours (HW5)
Wed
2/11
Problem session (HW6) [slides] HW5 due at 11:59pm
Fri
4/11
TA Office Hours (HW6)
Week 9 Mon
7/11
Fall Reading Week (No Lecture)
Week 10 Mon
14/11
Lecture 8: Light Field Imaging
plenoptic function, light field cameras, 3D displays
[slides]
Tues
15/11
TA Office Hours (HW6)
Wed
16/11
HW6, Project Proposal due at 11:59pm
Week 11 Mon
21/11
Lecture 9: Guest Lecture Mark Sheinin (CMU)
Imaging, Fast and Slow: Computational Imaging for Sensing High-speed Phenomena
[slides]
Week 12 Mon
28/11
Lecture 10: Time-of-flight imaging
lidar, single-photon imaging, non-line-of-sight imaging, scattering
[slides]
Week 13 Mon
5/12
Lecture 11: Neural signal representations
coordinate networks, neural scene representations, neural radiance fields
[slides]
Thu
8/12
Final Project - poster presentations in Bahen lobby
ProjectPosterTemplate.ppt
iccp2020_latex_template.zip
Project report and code due at 11:59pm

Miscellanea

Installing Python

We will only support Python 3.7 and recommend that you install it using Anaconda or Miniconda (see installation instructions here).

Poster Printing Instructions

See Quercus for instructions on how to print your poster for the final project poster session.

Previous Course Offerings

This course is adapted from the Computational Imaging course designed by Gordon Wetzstein and offered at Stanford University (EE367). Below you can find links to pinhole camera photos and course projects from these previous iterations of the course.

Pinhole Camera Gallery

Links to notable pinhole camera photos:

Previous Course Projects

Acknowledgements

The course is adapted from EE367 at Stanford University by Gordon Wetzstein. Some of the materials used in class build on that from other instructors, including Marc Levoy, Fredo Durand, Ramesh Raskar, Shree Nayar, Paul Debevec, Matthew O'Toole and others, as noted in the slides. Feel free to use these slides for academic or research purposes, but please maintain all acknowledgments. This webpage is based on the website for CS231N and EE367 at Stanford University.