CSC321 Winter 2013 - Lectures

Tuesday January 8
First class meeting. Explaining how the course will be taught.

For the rest of this schedule, students are to study the listed material before the class meeting.
The 'lecture' meeting will always be a discussion of the material of that day.
The 'tutorial' meetings are used for a variety of purposes.

Thursday January 10
Tutorial: Introduction to Matlab, for novices
(This is the beginning of a section: Introduction)
Video 1a: Why do we need machine learning?
Video 1b: What are neural networks?
Video 1c: Some simple models of neurons
Video 1d: A simple example of learning

Tuesday January 15
Video 1e: Three types of learning
Video 2a: Types of neural network architectures
(This is the beginning of a section: Perceptrons)
Video 2b: Perceptrons: The first generation of neural networks

Thursday January 17
Tutorial: Introduction to Matlab, with focus on vectorizing
Video 2c: A geometrical view of perceptrons
Video 2d: Why the learning works
Video 2e: What perceptrons can't do

Tuesday January 22
(This is the beginning of a section: Modern NN basics)
Video 3a: Learning the weights of a linear neuron
Video 3b: The error surface for a linear neuron
Video 3c: Learning the weights of a logistic output neuron

Thursday January 24
Tutorial: The basics of probability theory and/or linear algebra
Video 3d: The backpropagation algorithm
Video 3e: Using the derivatives computed by backpropagation

Tuesday January 29
(This is the beginning of a section: Language modeling)
Video 4a: Learning to predict the next word
Video 4b: A brief diversion into cognitive science
Video 4c: Another diversion: The softmax output function
Written lecture: The math of softmax units
Video 4d: Neuro-probabilistic language models
(Assignment 1 is posted: learning a distributed representation of words)

Thursday January 31
Tutorial: Introducing assignment 1
Video 4e: Ways to deal with the large number of possible outputs
(This is the beginning of a section: Architectures and why they work)
Optionally revisit: Video 2a: Types of neural network architectures
Video 101a: Coarse coding 1
Video 101b: Coarse coding 2
Video 101c: Coarse coding 3

Tuesday February 5
(Assignment 1 is due)
Video 5a: Why object recognition is difficult
Video 5b: Achieving viewpoint invariance
Video 5c: Convolutional nets for digit recognition
Video 5d: Convolutional nets for object recognition

Thursday February 7
Tutorial: Post mortem on assignment 1
(This is the beginning of a section: Optimization)
Video 6a: Overview of mini-batch gradient descent
Video 6b: A bag of tricks for mini-batch gradient descent

Tuesday February 12
Video 6c: The momentum method
Video 6d: Adaptive learning rates for each connection
Video 6e: Rmsprop: Divide the gradient by a running average of its recent magnitude
(Assignment 2 is posted: optimization)

Thursday February 14
Tutorial: Introducing assignment 2
(This is the beginning of a section: Recurrent neural networks)
Video 7a: Modeling sequences: A brief overview
Video 7b: Training RNNs with back propagation

Tuesday February 19 and Thursday February 21, there are no class meetings (Reading week).

Tuesday February 26
Video 7c: A toy example of training an RNN
Video 7d: Why it is difficult to train an RNN
Video 7e: Long-term Short-term-memory

Thursday February 28
Tutorial: midterm
(This is the beginning of a section: Generalization)
Video 9a: Overview of ways to improve generalization
Video 9b: Limiting the size of the weights
Video 9c: Using noise as a regularizer

Tuesday March 5
(Assignment 2 is due)
Video 9d: Introduction to the full Bayesian approach
Video 9e: The Bayesian interpretation of weight decay
Video 10a: Why it helps to combine models
Video 10b: Mixtures of Experts

Thursday March 7
Tutorial: Post mortem on assignment 2
Video 10c: The idea of full Bayesian learning
Video 10d: Making full Bayesian learning practical
Video 10e: Dropout

Tuesday March 12
(This is the beginning of a section: Mixtures of Gaussians, as an introduction to generative models)
Video 102a: Mixtures of Gaussians 1
Video 102b: Mixtures of Gaussians 2
Video 102c: Mixtures of Gaussians 3
Video 102d: Mixtures of Gaussians 4
(Assignment 3 is posted: Mixtures of Gaussians)

Thursday March 14
Tutorial: Introducing assignment 3
(This is the beginning of a section: Undirected unsupervised stochastic models)
Video 11a: Hopfield Nets
Video 11b: Dealing with spurious minima
Video 11c: Hopfield nets with hidden units

Tuesday March 19
(Assignment 3 is due)
Video 11d: Using stochastic units to improve search
Video 11e: How a Boltzmann machine models data
Video 12a: Boltzmann machine learning
Video 12c: Restricted Boltmann Machines

Thursday March 21
Tutorial: Post mortem on assignment 3
Video 12d: An example of RBM learning
Video 12e: RBMs for collaborative filtering
Video 14d: Modeling real-valued data with an RBM

Tuesday March 26
(This is the beginning of a section: Directed unsupervised stochastic models)
Video 13a: The ups and downs of back propagation
Video 13b: Belief Nets
Video 13c: Learning sigmoid belief nets
Written lecture: The math of Sigmoid Belief Nets
(Assignment 4 is posted: RBMs for pretraining)

Thursday March 28
Tutorial: Introducing assignment 4
Video 13d: The wake-sleep algorithm
Video 14a: Learning layers of features by stacking RBMs
Video 14b: Discriminative learning for DBNs
Video 14c: What happens during discriminative fine-tuning?

Tuesday April 2
(Assignment 4 is due)
(This is the beginning of a section: Autoencoders)
Video 15a: From PCA to autoencoders
Video 15b: Deep autoencoders
Video 15c: Deep autoencoders for document retrieval

Thursday April 4
Tutorial: Post mortem on assignment 4
Video 15d: Semantic Hashing
Video 15e: Learning binary codes for image retrieval
Video 15f: Shallow autoencoders for pre-training

CSC321 - Computation In Neural Networks: || www.cs.toronto.edu/~tijmen/csc321/