CSC321 Winter 2015: Introduction to Neural Networks

Calendar

There are two sections for the course.

**For the afternoon section**, the first lecture each week is 1:10-2pm on Tuesday, and the second lecture is 1:10-2pm on Thursday. The tutorial is 12:10-1pm on Thursday. Lectures and tutorial are both in BA1200.**For the night section**, the two lectures are held back-to-back from 6:10-7:50pm. The tutorial is 8:20pm-9pm. Lectures and tutorial are both in BA1220.

Each week has some assigned Coursera videos. The quiz on those videos is **due by 11:59pm on Monday** of that week. Since the in-class meetings build on the material in the Coursera videos, it is important that you watch them before class.

**No lectures, tutorials, or office hours will be held for the duration of the CUPE 3902 Unit 1 strike.** You should have received an e-mail with further details.

**Background:**If it’s been a long time since you used any linear algebra, this is a good time for a refresher. Here are the relevant concepts on Metacademy. Each one gives a number of pointers, but the Khan Academy links are especially useful since they have auto-graded exercises you can use to check your understanding.**Lecture 1:**What are neural networks and machine learning? (Traditional lecture) pdf**Lecture 2:**Linear regression (Traditional lecture) pdf- No tutorial this week.
- Optional resources:

**Assigned Coursera videos:**Lectures A and B.**Remember that both quizzes are due at 11:59pm on Monday, Jan. 12.**You may find these lecture notes a helpful supplement.**Background:**plotting linear inequalities in 2 dimensions**Lecture 3:**Binary linear classification [pdf]**Lecture 4:**The perceptron learning algorithm [pdf] (We didn’t make it through all of these slides in lecture, so we will finish the rest next week.)**Tutorial:**Python programming [pdf]- This covers the basics of the Python language, NumPy arrays, and plotting with Matplotlib.

- Optional resources:

**Assigned Coursera videos:**Lecture C. The quiz is due 11:59pm Monday as usual. [notes]**Background:**- We compute a lot of derivatives this week. If you want a refresher, try the Khan Academy sequence on computing derivatives, which includes some auto-graded exercises. The sections on the chain rule and the product and quotient rules are especially relevant.
- We will also work with partial derivatives.
- This week is when we start to talk about implementing learning algorithms. See the Homework page for some recommended reading on Python and NumPy, as well as installation instructions.

**Lecture 5:**Learning in a single neuron [pdf]**Lecture 6:**Backpropagation [pdf]**Tutorial:**Implementing the learning in Python ([pdf] [code])- Optional resources:

**Assigned Coursera videos:**Lectures D and E. Note that we are going a bit out of order compared with how the Coursera course was originally taught. [notes]**Background:****Lecture 7:**Neural language models [pdf]**Lecture 8:**Optimization tricks [pdf]**Tutorial:**Introducing Assignment 1 [pdf]- Optional resources:
- notes on the math for softmax units
- the original paper on neural language models
- stochastic gradient descent

**Extra office hours for Assignment 1:**Friday from 3-4pm in BA4161

**Assigned Coursera videos:**Lecture F [notes]**Assignment 1 is due at the start of lecture on Tuesday.****Lecture 9:**Recurrent nets [pdf]**Lecture 10:**Training recurrent nets [pdf]- No tutorial this week.

**Assigned Coursera videos:**Lecture G [notes]**Lecture 11:**Convolutional nets [pdf]**Lecture 12:**Recent advances in convolutional nets (traditional lecture) [pdf]**Tutorial:**Assignment 1 post-mortem; introducing Assignment 2- Optional resources:

There is no class during reading week, but we will still hold office hours. There are no office hours on Monday, since it is a holiday. Instead, we’ll hold them at the following times:

- Tuesday, Feb 17, from 2-4pm in Pratt 290C (the usual room)
- Friday, Feb 20, from 3-5pm in Pratt 290C

This week, we will have an unusual schedule because of the midterm. See the announcements page for more information about the test.

- For the afternoon section:
**The midterm will be held during lecture, 1:10-2pm on Tuesday, Feb. 24, in the usual room.**- In place of the tutorial, there will be a double-lecture on generalization from 12:10-2pm on Thursday, Feb. 26.

- For the night section:
**The midterm will be held during lecture, 6:10-7pm on Tuesday, Feb. 24, in the usual room.**- In place of the tutorial, there will be a double-lecture on generalization from 7:10-9pm on Tuesday, Feb. 24.

This week’s content:

**Assigned Coursera videos:**Lectures J and K- You only have to do the quiz for Lecture J.
- You are still responsible for the first video in Lecture K, even though there is no quiz. The other videos for Lecture K are optional.
- Because of the midterm, we have extended the quiz deadline until
**11:59pm Thursday**. However, you will get more out of class if you watch the videos beforehand.

**Background:****Lecture 13 + 14:**Improving generalization [pdf] (solutions will be posted after we finish covering the material)**Tutorial:**introducing assignment 2 [slides]**Midterm solutions:**- Optional resources:
- generalization
- cross-validation
- regularized linear regression
- LASSO (sparse linear regression)
- bagging

**No lectures, tutorials, or office hours will be held for the duration of the CUPE 3902 Unit 1 strike.** You should have received an e-mail with further details.

**Assigned Readings:**instead of videos, we have required readings for this week. Even though there’s no quiz, you should do the readings before lecture. Don’t be intimidated by the length: most of the text is worked-through examples. There are fewer new ideas than in most of the video lectures.**Background:**look at the Week 7 background if you haven’t done so yet.**Lecture 15:**Learning probabilistic models**Lecture 16:**Mixture models**We no longer have tutorials**, since both sections voted to replace the tutorial with another hour of lecture.- Optional resources:

**Assigned Coursera videos:**Lectures L and M- Note that you have
**two**quizzes. Due to the assignment deadline, the quiz deadline is moved to**11:59pm on Thursday, March 12.**

- Note that you have
**Assignment 2 is due at the start of lecture on Tuesday.****Lecture 17:**Hopfield nets and Boltzmann machines**Lecture 18:**Learning Boltzmann machines

**Assigned Coursera videos:**Lecture N**Assignment 3 is due at the start of lecture on Tuesday.****Lecture 19:**Principal component analysis**Lecture 20:**Autoencoders

- We will have assigned readings rather than videos.
**Lecture 21:**Bayesian neural networks**Lecture 22:**Bayesian optimization**Tutorial:**Assignment 3 post-mortem; introducing Assignment 4

- We will have assigned readings rather than videos.
**Assignment 4 is due at the start of lecture on Tuesday.****Lecture 23:**Q-learning**Lecture 24:**Policy gradient**Tutorial:**final exam review