CSC320 (Spring 2021)

Introduction to Visual Computing

General Information

Professor: Kyros Kutulakos

Office hour: T 1-2pm (online)

TAs: Esther Lin, Mian Wei, Juhan Bae, Wenzheng Chen, Jianfeng Li, Sam Sattarzadeh, Robin Swanson, Hafsa Zahid

Lectures: W 6-8pm (online synchronous)

Tutorial 1 (TA Esther Lin): W 8-9pm (Student last name: A-Liu, online synchronous)

Tutorial 2 (TA Mian Wei): W 8-9pm (Student last name: Lu-Z, online synchronous)

Term test 1: Mar 10 (W), 8-9pm (via Crowdmark)

Term test 2: TBD, during final assessment period (2-hour test + 30 mins for upload, via Crowdmark)

About the Course

Image synthesis and image analysis aimed at students with an interest in computer graphics, computer vision, or the visual arts. Focus on three major topics: (1) visual computing principles—computational and mathematical methods for creating, capturing, analyzing, and manipulating digital photographs (image acquisition, basic image processing, image warping, anti-aliasing); (2) digital special effects—applying these principles to create special effects found in movies and commercials; (3) visual programming—using Python and graphical user interface creation tools for synthesizing and manipulating photographs. The course requires the ability to use differential calculus in several variables and linear algebra.

Textbooks & other reading materials: There is no required textbook for the course. We will be using sections from Rick Szeliski's book on Computer Vision (2nd ed) as well a few research papers. These materials along with lecture slides, tutorial notes and assignments are available on the course dropbox.. Lecture slides for all lectures from the 2018 version of this course are already on dropbox. These will be updated over the term with this years' lecture slides.

Prerequisites: See the FAS calendar entry. Students taking the course are expected to be comfortable with linear algebra, elementary geometry, multivariate calculus (including partial differentiation). It is also assumed that students are very comfortable with Python programming.

Marking Scheme

Term test 1 is worth 19% and Term test 2 is worth 28%.

Class participation is worth 3%, assessed via zoom polls during lectures 2-12. Students will receive full marks for participation if they respond to all zoom polls in at least 8 lectures during the term.

There will be 5 assignments for a total weight of 50%. Each assignment includes a written report, and assignments 2-5 have a programming component. Due dates and weight distribution are as follows:

- Assignment 1 (5%): posted on dropbox Jan 13, due Jan 20 at noon
- Assignment 2 (12%): posted on dropbox Jan 20, due Feb 10 at noon
- Assignment 3 (12%): posted on dropbox Feb 10, due Mar 3 at noon
- Assignment 4a (12%): posted on dropbox Mar 10, due Apr 7 at noon
- Assignment 4b (9%): posted on dropbox Mar 24, due Apr 7 at noon

Late policy: A15% marks deduction applies for each day late, for up to five days. Deductions begin immediately after the due date & time (ie. after 12:01pm on the due date). For example, an assignment submitted late but less then 24hrs after the due date will receive a 15% deduction. Assignments submitted more than 120 hours late will not be accepted.

Lecture Calendar (tentative)

Part I: Representing images as 2D arrays of pixels

Date

Topic

Sub-topic

Readings (on dropbox)

Resources (on dropbox & web)

Jan 13

Introduction; Cameras and Images

Understanding digital images; basic camera controls; color image acquisition; image noise

Szeliski: 73-76, 78-82, 85-94

clarkvision.com: A very comprehensive website about photography, cameras and how to characterize their properties

Jan 20

HDR Imaging and Alpha Matting

Computing camera response functions from images; the matting equation

Szeliski: 600-607.

Debevec paper: Sections 1 and 2, up to Eq (2). As you read them, note that the film response curve corresponds to the camera response curve in the case of digital photography

Part II: Representing images as continuous 1D and 2D functions

Jan 27

Computing 1D image derivatives

Least-squares polynomial fitting; intensity derivatives; weighted least squares; RANSAC

polydemo.zip in the Demo Code directory on dropbox. Run the demo for a variety of fits (LS, WLS, 1st degree, 2nd degree, etc) to see their effect.

Feb 3

Representing 2D image curves

Local analysis of curves: the tangent & normal vectors, the moving frame

Feb 10

Edge detection

Local analysis of 1D and 2D image patches: the Image Gradient; case study: Painterly Rendering; Intelligent scissors

Litwinowicz paper (this is not required reading).

Feb 24

Basic linear filter theory

Linear filters; convolution; Fourier series and the continuous-time Fourier transform

Szeliski: 109-113, 139-145

Part III: Representing images as N-dimensional vectors

Mar 3

Template matching, correlation and patch-based image processing

Representing images as vectors; evaluating similarity using RMS distance error, cross-correlation and normalized cross-correlation; non-local means denoising

To run the demo shown in class: (1) unpack the file corrdemo.zip in the Demo Code directory, (2) run MATLAB, (3) change the current MATLAB directory to the directory you unpacked the code, (4) type corrdemo at the matlab prompt.

Mar 10

Principal Component Analysis

Face recognition using Eigenfaces

To run the demo shown in class: (1) unpack the file recognition_demo.zip in the Demo Code directory, (2) run MATLAB, (3) change the current MATLAB directory to the directory you unpacked the code, (4) type pca_recdemo at the matlab prompt.

Part IV: Multi-resolution image representations

Mar 17

Gaussian & Laplacian Pyramids

The Matlab demo scale_space1D_demo.zip in the Demo Code directory on Dropbox shows how a 1D image changes as we smooth it with a sequence of Gaussians of increasing standard deviation.

Mar 24

The Haar Wavelet Transform

Stollnitz paper.

The Matlab demo wavedemo.zip in the Demo Code/ directory is the demo shown in class. Type wavedemo at the Matlab prompt to run it.

Mar 31

Polynomial fitting vs. correlation; Matching images using SIFT

Analysis of WLS polynomial fitting and image smoothing as a template matching operation; template matching expressed as a multiplication of an image with a Toeplitz matrix; Gaussian image smoothing; SIFT-based feature detection; the SIFT descriptor; image matching using SIFT

Part V: Introduction to 2D Image Transformations

Apr 7

Homogeneous coordinates

Homography-based image warping; panoramic image stitching

How to make the most out of this course

- Browse the lecture slides before coming to class and consult the relevant readings, if applicable. While you are certainly not expected to digest the material before coming to each lecture, this should help you identify background you may want to consult before attending, as well as questions you may want to ask during class.
- Print a copy of the lecture slides (or have them on hand in digital form on a tablet) so you can write annotations on them during the lecture.
- Consult past exams often, throughout the course. This will help you assess your understanding of the material as well as prepare for the term tests. Note that my policy is to not provide solutions to past exam questions and to discourage students from posting their answers to the web. Nevertheless, you are encouraged to discuss your answers and/or solution strategies informally in small groups. I am more than happy to tell you or your group if you are on the right (or wrong) track for a past exam question, as long as you have made a reasonable effort to answer it.
- Create informal study groups to discuss the lecture materials and the research papers associated with assignments, to double-check your answers to past exam questions and to resolve points of confusion. Informal groups of 2-3 students work best.

Academic Honesty Policy

Each assignment will have a written component and most will also have a programming component. The course policy is as follows.

- Written reports: All reports submitted as part of your assignments in CSC320 are strictly individual work. No part of these reports should be shared with others, or taken from others. This includes verbatim text, paraphrased text, and/or images used. You are, however, allowed to discuss these components with others at the level of ideas, and indeed you are welcome to brainstorm together.
- Programming assignments: Collaboration on a programming component by individuals (whether or not they are taking the class) is encouraged at the level of ideas. Feel free to ask each other questions, brainstorm on algorithms, or work together on a (virtual or real) whiteboard. Be careful, however, about copying the actual code for programming assignments or merely adapting others' code. This sort of collaboration at the level of artifacts is permitted if explicitly acknowledged, but this is usually self-defeating. Specifically, you will get zero points for any portion of an artifact that you did not transform from concept into substance by yourself. If you neglect to label, clearly and prominently, any code that isn't your own or that you adapted from someone else's code, that's academic dishonesty for the purpose of this course and will be treated accordingly.
- There are some circumstances under which you may want to collaborate with someone else on the programming component of an assignment. You and a friend, for example, might create independent parts of an assignment, in which case you would each get the points pertaining to your portion, and you'd have the satisfaction of seeing the whole thing work. Or you might get totally stuck and copy one subroutine from someone else, in which case you could still get the points for the rest of the assignment (and the satisfaction of seeing the whole thing work). But if you want all the points, you have to write everything yourself. These collaborations must be explicitly acknowledged in the CHECKLIST.txt file you submit for each assignment.

The principle behind the above policies is simple: I want you to learn as much as possible. I don't care if you learn from me or from each other. The goal of artifacts (programming assignments) is simply to demonstrate what you have learned. So I'm happy to have you share ideas, but if you want your own points you have to internalize the ideas and them craft them into an artifact by yourself, without any direct assistance from anyone else, and without relying on any code taken from others (whether at this university or from the web).

Note: this policy, with minor modifications, was developed by Michael Scott at the University of Rochester