CSC 421/2516 Winter 2019

Neural Networks and Deep Learning

Overview

Machine learning is a powerful set of techniques that allow computers to learn from data rather than having a human expert program a behavior by hand. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They're at the heart of production systems at companies like Google and Facebook for face recognition, speech-to-text, and language understanding.

This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms. Roughly the first 2/3 of the course focuses on supervised learning --- training the network to produce a specified behavior when one has lots of labeled examples of that behavior. The last 1/3 focuses on unsupervised learning and reinforcement learning.

Policies (marking, prerequisites, etc.)

See the course information handout.

Where and When

There are two sections of the course. Since both sections are fully subscribed, please attend the one you are registered for.

Teaching Staff

• Instructors: Roger Grosse, Jimmy Ba
• Office Hours:
  • Roger: Monday 5-6pm in PT290F
  • Jimmy: Friday 1-2pm in PT290D
• Head TA: Alex Adam
• TAs: TBA
• Staff emails:
  • TAs and instructors: csc421staff [at] cs.toronto.edu
  • Instructors and head TA only: csc421instructors [at] cs.toronto.edu
  • Please do not contact us at our personal emails.
• We will use Piazza for the course forum. If your question is about the course material and doesn't give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.

Assignments

Most written homeworks and programming assignments will be due on Thursdays at 11:59pm. Please see the course information handout for detailed policies (marking, lateness, etc.).

The following schedule is subject to change.

Grad Student Projects

Grad students will do a final project in place of the final exam. Students must form teams of 2-3. The deadline for proposals is March 1, but you are encouraged to submit a proposal earlier so that you can receive feedback earlier. The deadline for final reports is April 18 26. You can find the full project requirements here.

Tests

Midterm

All students (undergrads and grad students) must take the midterm test. It will be held from 6:10-7:40pm on Friday, Feb. 15, in EX 200 (Exam Centre). It will be a 90 minute exam.

It will cover up through Lecture 9 (conv nets). Only material covered in lecture will be tested, so we won't test material that is only in the tutorials, readings, etc. However, we will place more emphasis on topics you've had an opportunity to practice in homeworks, tutorials, etc. There will be some conceptual questions, and some mathematical questions (similar to individual steps in the homeworks).

The format will be similar to CSC321 midterms from past years, so you might like to use these to practice. Note that the topics covered in different years might not correspond exactly.

• 2015 midterm: version 1 and solutions; version 2 and solutions. Note that this exam was too difficult, and the marks were adjusted upwards.
• 2017 midterm: version 1, version 2, and solutions.
• 2018 midterm: version 1, version 2, and solutions.

Here are the midterm questions and solutions.

Final Exam

Only undergrads will take the final exam. Grad students do a final project instead.

The exam will take place from 9am-noon on Thursday, April 25. The rooms are as follows:

• Surname A-G: Bahen (BA) 2159
• Surname H-Z: Medical Sciences (MS) 2158
• No, it's not a typo that it's two different buildings with adjacent room numbers.

Practice exams. These are from CSC321, a third-year version of this course. All but 2017 and 2018 were with different instructors, and topics varied from year to year.

• 2011
• 2012
• 2013
• 2016
• 2017, and solutions
• 2018, and solutions

Lectures

Here is a tentative schedule, which will likely change as the course goes on. Each "Lecture" corresponds to 50 minutes, so each 2-hour lecture session will cover 2 of them.

Suggested readings are just that: resources we recommend to help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the papers listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.

Goodfellow = I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning.

Tutorials