Machine learning is a powerful set of techniques that allow computers to learn from data rather than having a human expert program a behavior by hand. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They're at the heart of production systems at companies like Google and Facebook for face recognition, speech-to-text, and language understanding.

This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms. Roughly the first 2/3 of the course focuses on supervised learning -- training the network to produce a specified behavior when one has lots of labeled examples of that behavior. The last 1/3 focuses on unsupervised learning -- the algorithm isn't given any examples of the correct behavior, and the goal is instead to discover interesting regularities in the data.

**Winter term 2017****Instructor:**Roger Grosse**Office hours:**Mondays 10am-noon, Pratt 290F. (That's the D. L. Pratt Building, not the E. J. Pratt Library!)**Teaching assistants:**ChunHao Chang, Renjie Liao, Ladislav Rampasek, Paul Vicol, Yuhuai (Tony) Wu, Lisa Zhang**Staff e-mails:****TAs and instructor:**csc321ta [at] cs.toronto.edu**Instructor only:**csc321prof [at] cs.toronto.edu- Please do not contact us at our personal e-mails. We will not respond.
**Afternoon section:****Lectures:**Tuesdays and Thursdays, 1:10-2:00pm, in Sidney-Smith 1073**Tutorials:**Thursdays, 2:10-3:00pm, in Bahen 1200. There is no tutorial on January 5.**Night section:****Lectures:**Tuesdays, 6:10-7:50pm, in Bahen 1200.**Tutorials:**Tuesdays, 8:15-9:00pm, in Bahen 1200. (This allows a 25 minute dinner break between lecture and tutorial.)

All course-related announcements will be sent to the class mailing list, csc321h1s [at] teach.cs.toronto.edu.

We will use Discourse as the discussion forum. If you have a question you think would be relevant to the whole class, please post it there so that everyone gets the benefit of the answer. Please include the part of the course (e.g. "Homework 1", "Lecture 2") in the subject.

If you want to contact the course staff privately, please e-mail csc321ta [at] cs.toronto.edu (for the TAs and the instructor) or csc321prof [at] cs.toronto.edu (for only the instructor).

All assignment deadlines are at 11:59pm of the date listed. Please see the course information handout for detailed policies (marking, lateness, etc.).

**Marking scheme:**

- Midterm: 15%
- Final exam: 35%
- Weekly homeworks: 10% total
- 4 Programming assignments: 10% each

**Schedule:**

- Homework 1 (due 1/16)
- If you are on the waitlist, then you don't have a MarkUs account yet. In that case, please e-mail your solutions to the staff list.
- Homework 2 (due 1/23)
- Homework 3 (due 1/30)
**Programming Assignment 1**(due~~2/2~~2/6) [handout] [code]- TA office hours, in Pratt 290C:
- Wednesday, Feb. 1, 3-4pm
- Friday, Feb. 3, 3-4pm
- Monday, Feb. 6, 11am-noon

- TA office hours, in Pratt 290C:
- Homework 4
~~(due 2/6)~~(not marked) - Homework 5 (due 2/13)
**Programming Assignment 2**(due~~2/16~~~~2/20~~2/22) [handout] [code]- TA office hours, in Pratt 290C:
- Monday, Feb. 13, 11am-noon
- Wednesday, Feb. 15, 2-3pm
- Friday, Feb. 17, 2-3pm
~~Monday, Feb. 20, 11am-noon~~Tuesday, Feb. 21, 11am-noon

- TA office hours, in Pratt 290C:
- Homework 6
~~(due 3/6)~~(not marked) **Midterm****Afternoon:**The test will be held during lecture time, 2/28, 1:10-2:00pm. There will be lecture as usual on 3/2, but no tutorial.**Night:**The test will be held during lecture time, 2/28, 6:10-7:00pm. There will be a half-hour break, followed by a lecture from 7:30-8:30, and no tutorial.- Practice tests:
- 2013 Midterm
- Questions from 2014 Midterm
- 2015 Midterm
- 2016 Midterm and solutions
- Note that topics were covered in a different order in previous years, so some questions were on material we haven't covered yet.

- Midterm: afternoon, night, solutions
- Homework 7 (due 3/15)
**Programming Assignment 3**(due 3/22) [handout] [code]- TA office hours, in Pratt 290C:
- Friday, March 17, 1:30-2:30pm
- Monday, March 20, 11am-12
- Tuesday, March 21, 11am-12
- Wednesday, March 22, 10-11am

- TA office hours, in Pratt 290C:
- Homework 8 (due 3/29)
**Programming Assignment 4**(due 4/4) [handout] [code]- TA office hours, in Pratt 290C:
- Friday, March 31, 2-3pm
- Monday, April 3,
~~3-4pm~~4-5pm - Tuesday, April 4, 3-4pm

**Final Exam:**April 24

**Lecture 1: Introduction**[Slides] [Lecture Notes]Afternoon: 1/5, 1-2pm; Night: 1/10, 6-7pm

What are machine learning and neural networks, and what would you use them for? Supervised, unsupervised, and reinforcement learning. How this course is organized.

**Lecture 2: Linear Regression**[Slides] [Lecture Notes]Afternoon: 1/10, 1-2pm; Night: 1/10, 7-8pm

Linear regression, a supervised learning task where you want to predict a scalar valued target. Formulating it as an optimization problem, and solving either directly or with gradient descent. Vectorization. Feature maps and polynomial regression. Generalization: overfitting, underfitting, and validation.

**Lecture 3: Linear Classification**[Slides] [Lecture Notes]Afternoon: 1/12, 1-2pm; Night: 1/17, 6-7pm

Binary linear classification. Visualizing linear classifiers. The perceptron algorithm. Limits of linear classifiers.

**Lecture 4: Learning a Classifier**[Slides] [Lecture Notes]Afternoon: 1/17, 1-2pm; Night: 1/17, 7-8pm

Comparison of loss functions for binary classification. Cross-entropy loss, logistic activation function, and logistic regression. Hinge loss. Multiway classification. Convex loss functions. Gradient checking. (Note: this is really a lecture-and-a-half, and will run into what's scheduled as Lecture 5.)

**Lecture 5: Multilayer Perceptrons**[Slides] [Sorry, no notes.]Afternoon: 1/19, 1-2pm; Night: 1/24, 6-7pm

Multilayer perceptrons. Comparison of activation functions. Viewing deep neural nets as function composition and as feature learning. Limitations of linear networks and universality of nonlinear networks.

Suggested reading: Deep Learning Book, Sections 6.1-6.4

**Lecture 6: Backpropagation**[Slides] [Lecture Notes]Afternoon: 1/24, 1-2pm; Night: 1/24, 7-8pm

The backpropagation algorithm, a method for computing gradients which we use throughout the course.

**Lecture 7: Optimization**[Slides] [Sorry, no notes.]Afternoon: 1/26, 1-2pm; Night: 1/31, 6-7pm

How to use the gradients computed by backprop. Features of optimization landscapes: local optima, saddle points, plateaux, ravines. Stochastic gradient descent and momentum.

Suggested reading: Deep Learning Book, Chapter 8

**Lecture 8: Automatic Differentiation**[Slides: part 1, part 2] [Sorry, no notes.]Afternoon: 1/31, 1-2pm; Night: 1/31, 7-8pm

Guest lecture by David Duvenaud

**Lecture 9: Generalization**[Slides] [Sorry, no notes.]Afternoon: 2/2, 1-2pm; Night: 2/7, 6-7pm

Bias/variance decomposition, data augmentation, limiting capacity, early stopping, weight decay, ensembles, stochastic regularization, hyperparameter tuning.

Suggested reading: Deep Learning Book, Chapter 7

**Lecture 10: Distributed Representations**[Slides] [Sorry, no notes.]Afternoon: 2/7, 1-2pm; Night: 2/7, 7-8pm

Language modeling, n-gram models (a localist representation), neural language models (a distributed representation), and skip-grams (another distributed representation).

**Lecture 11: Convolutional Networks**[Slides] [Lecture Notes]Afternoon: 2/9, 1-2pm; Night: 2/14, 6-7pm

Convolution operation. Convolution layers and pooling layers. Equivariance and invariance. Backprop rules for conv nets.

**Lecture 12: Image Classification**[Slides] [Sorry, no notes.]Afternoon: 2/14, 1-2pm; Night: 2/14, 7-8pm

Conv net architectures applied to handwritten digit and object classification. Measuring the size of a conv net.

**Lecture 13: Fun with Conv Nets**[Slides (Michael Guerzhoy): 1, 2, 3] [Sorry, no notes.]Afternoon: 2/16, 1-2pm; Night: 2/28, 7:30-8:30pm

Conv net visualizations: guided backprop, gradient descent on inputs. Deep Dream. Neural style transfer.

**Lecture 14: Recurrent Neural Nets**[Slides] [Lecture Notes]Afternoon: 3/2, 1-2pm; Night: 3/7, 6-7pm

Recurrent neural nets. Backprop through time. Applying RNNs to language modeling and machine translation.

**Lecture 15: Exploding and Vanishing Gradients**[Slides] [Lecture Notes]Afternoon: 3/7, 1-2pm; Night: 3/7, 7-8pm

Why RNN gradients explode and vanish, both in terms of the mechanics of backprop, and conceptually in terms of the function the RNN computes. Ways to deal with it: gradient clipping, input reversal, LSTM.

**Lecture 16: ResNets and Attention**[Slides] [Notes coming soon]Afternoon: 3/9, 1-2pm; Night: 3/14, 6-7pm

Deep Residual Networks. Attention-based models for machine translation and caption generation. Neural Turing Machines.

**Lecture 17: Learning Probabilistic Models**[Slides] [Notes]Afternoon: 3/14, 1-2pm; Night: 3/14, 7-8pm

Maximum likelihood estimation. Basics of Bayesian parameter estimation and maximum a-posteriori estimation.

**Lecture 18: Mixture Modeling**[Slides] [Notes]Afternoon: 3/16, 1-2pm; Night: 3/21, 6-7pm

K-means. Mixture modeling: posterior inference and parameter learning.

**Lecture 19: Boltzmann Machines**[Slides] [Notes coming soon]Afternoon: 3/21, 1-2pm; Night: 3/21, 7-8pm

Boltzmann machines: definition; marginal and contitional probabilities; parameter learning. Restricted Boltzmann machines.

**Lecture 20: Autoencoders**[Slides: part 1, part 2 (Geoffrey Hinton)]Afternoon: 3/23, 1-2pm; Night: 3/28, 6-7pm

Principal component analysis; autoencoders; layerwise training; applying autoencoders to document and image retrieval

**Lecture 21: Bayesian Hyperparameter Optimization**[Slides]Afternoon: 3/28, 1-2pm; Night: 3/28, 7-8pm

Bayesian linear regression; Bayesian optimization.

**Lecture 22: Adversarial Learning**[Slides]Afternoon: 3/30, 1-2pm; Night: 4/4, 6-7pm

Adversarial examples; generative adversarial networks (GANs).

**Lecture 23: Go**[Slides]Afternoon: 4/4, 1-2pm; Night: 4/4, 7-8pm

**Tutorial 1: Linear Regression and Python**[PDF] [IPython Notebook]Afternoon: 1/12, 2-3pm; Night: 1/10, 8-9pm

**Tutorial 2: Classification**[PDF] [IPython Notebook]Afternoon: 1/19, 2-3pm; Night: 1/17, 8-9pm

**Tutorial 3: Backpropagation**[PDF]Afternoon: 1/26, 2-3pm; Night: 1/24, 8-9pm

**Tutorial 4: Autograd**[Slides] [IPython Notebook]Afternoon: 2/2, 2-3pm; Night: 1/31, 8-9pm

**Tutorial 5: Optimization and Generalization**[Slides]Afternoon: 2/9, 2-3pm; Night: 2/7, 8-9pm

**Tutorial 6: Convolutional Networks**[Slides]Afternoon: 2/16, 2-3pm; Night: 2/14, 8-9pm

**Tutorial 7: Recurrent Neural Networks**Afternoon: 3/9, 2-3pm; Night: 3/7, 8-9pm

**Tutorial 8: Maximum Likelihood**~~Afternoon: 3/16, 2-3pm; Night: 3/14, 8-9pm~~No tutorial this week!

**Tutorial 9: Mixture Modeling**[IPython Notebook] [PDF]Afternoon: 3/23, 2-3pm; Night: 3/21, 8-9pm

**Tutorial 10: Bayesian Learning**Afternoon: 3/30, 2-3pm; Night: 3/28, 8-9pm

No tutorial this week!

The programming assignments will all be done in Python using the NumPy scientific computing library, but prior knowledge of Python is not required. Basic Python will be taught in a tutorial. **We will be using Python 2, not Python 3**, since this is the version more commonly used in machine learning.

You have several options for how to use Python:

- You can install Python yourself on your own machine. For most of you, this will be the most convenient option. (Our assignments will not require especially heavy computation.)
- Anaconda provides a single-click installer for most common platforms, and this is likely the easiest way to install Python and the required libraries.
- You can install Python, NumPy, and Matplotlib manually. This takes a bit more work than using Anaconda.

- You can run it on the Teaching Labs machines. All required libraries are already installed. Accounts should have already been created for registered students by the start of the course. If you are having a problem with a CDF account, ask us.

Once Python is installed, there are two ways you can edit and run Python code:

- You can edit the code in a general-purpose text editor, such as Emacs, Vim, or GEdit, and run Python from the command line. (If you’re not already familiar with a text editor, GEdit is probably the easiest to start with.) We recommend IPython rather than the default Python console. If you’re already comfortable with one of these editors and with the command line, this may be the easiest way to go. For most of us in the machine learning research group at U of T, this is how we use Python on a day-to-day basis. If you’re new to this mode of programming, it may take 5-10 hours before you feel comfortable with it. But if you’re concentrating in computer science, you’ll need to learn this stuff eventually, so why not now?
- If you’re newer to programming, you may feel more comfortable with an IDE. We recommend Spyder because it’s included in Anaconda, and it’s intended for the sort of numerical computing we do in this class. Here are instructions for using it with Anaconda. There are lots of other IDEs for Python, though.

Here are some recommended background readings on Python and NumPy.

**If you haven’t taken a programming class**, you may need to spend some time learning the basics. You should watch Lectures 2, 3, 4, and 6 of MIT 6.001 on EdX. (Lectures 7 and 11 are also helpful.)**If you have programming experience but not in Python**, read Learn X in Y Minutes for a concise summary of the language. You can probably pick up Python quickly if you are familiar with another general-purpose language (C, Java, Matlab, etc.).- Read this tutorial on NumPy, the library we’ll use for array manipulation and linear algebra.