Required math background
Nikulin & Novak, Exploring the Neural Algorithm of Artistic Style
Instructor: Michael Guerzhoy. Office: BA5244, Email: guerzhoy at cs.toronto.edu (please include CSC411 in the subject, and please ask questions on Piazza if they are relevant to everyone.)
CSC411 TAs: Patricia Thaine, Aditya Bharagava, Yulia Rubanova, Yingzhou Wu, Shun Liao, Bin Yang, Jixuan Wang, Aleksei Shkurin, Alexey Strokach, Katherine Ge, Karo Castro-Wunsch, Farzaneh Mahdisoltani.
A sample report/LaTeX template containing advice on how to write project reports AI courses is here (see the pdf and tex files). (The example is based on Programming Computer Vision, pp27-30.) Key points: your code should generate all the figures used in the report; describe and analyze the inputs and the outputs; add your interpretation where feasible.Project 1: Face Recognition and Gender Classification Using Regression (10%) Due
Jan. 29Feb 1 at 11PM
Project 2: Deep Neural Networks (10% + up to 2% bonus) Due
Feb. 19March 5 at 11PM
Project 3: Supervised and Unsupervised Methods for Natural Language Processing (10% + up to 1.5% bonus) Due
Mar. 19March 21 at 11PM
Project 4: Reinforcement Learning using Policy Gradients (10%) Due
Apr. 2Apr. 4 at 11PM (Note: late projects only accepted by Wednesday Apr. 5 23:59PM)
Lateness penalty: 5% of the possible marks per day, rounded up. Projects are only accepted up to 72 hours (3 days) after the deadline.
The Winter 2017 exam paper
The Winter 2017 midterm paper. Solutions and marking scheme. Solutions for Q2b and Q6: intro, performance vs. k, breaking the assumptions behind k-NN.
Friday March 3, 6pm-8pm. Make-up midterm for those who have a documented (a screenshot and/or explanatory email is sufficient) conflict with the main timeslot: TBA. Please send me email if there is an issue.
Coverage: the lectures and the projects, focusing on the lectures.
Michael's office hours: Tuesday 6-7PM, Wednesday 5-6PM in BA5244. Or email for an appointment. Or drop by to see if I'm in. Feel free to chat with me after lecture.
Course forum on Piazza
The CSC411 study guide (continuously updated)
We will be using the Python 2 NumPy/SciPy stack in this course. It is installed in the Teaching Labs.
For the first two projects, the most convenient Python distribution to use is Anaconda. If you are using an IDE and download Anaconda, be sure to have your IDE use the Anaconda Python.
I recommend the Pyzo IDE available here. To run Pyzo in the Teaching Labs, simply type iep in the command line.
We will be using Google's TensorFlow later on in the course.
We will be using the CS Teaching Labs and AWS for GPU computing. Please sign up using AWS Educate. Use your CS Teaching Labs account when signing up for AWS Educate. You can use either the GPU-equipped machines in the Teaching Labs (30 machines in BA2210 and 6 machines in BA3200) or AWS.
Pattern Recognition and Machine Learning by Christopher M. Bishop is a very detailed and thorough book on the foundations of machine learning. A good textbook to buy to have as a reference.
The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman is also an excellent reference book, available on the web for free at the link.
An Introduction to Statistical Learning with Applications in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani is a more accessible version of The Elements of Statistical Learning.
Deep Learning by Yoshua Bengio, Ian Goodfellow, and Aaron Courville is an advanced textbook with good coverage of deep learning and a brief introduction to machine learning.
Learning Deep Architectures for AI by Yoshua Bengio is in some ways better than the Deep Learning book, in my opinion.
Reinforcement Learning: An Introduction by R. Sutton and A. Barto will be useful when we discuss Reinforcement Learning
Geoffrey Hinton's Coursera course contains great explanations for the intution behind neural networks.
The CS229 Lecture Notes by Andrew Ng are a concise introduction to machine learning.
Andrew Ng's Coursera course contains excellent explanations of basic topics (note: registration is free).
Pedro Domingos's CSE446 at UW (slides available here) is a somewhat more theorically-flavoured machine learning course. Highly recommended.
CS231n: Convolutional Neural Networks for Visual Recognition at Stanford (archived 2015 version) is an amazing advanced course, taught by Fei-Fei Li and Andrej Karpathy (a UofT alum). The course website contains a wealth of materials.
CS224d: Deep Learning for Natural Language Processing at Stanford, taught by Richard Socher. CS231, but for NLP rather than vision. More details on RNNs are given here.
Python Scientific Lecture Notes by Valentin Haenel, Emmanuelle Gouillart, and Gaël Varoquaux (eds) contains material on NumPy and working with image data in SciPy. (Free on the web.)
All announcement will be on Piazza.
Coming up: introduction to Numpy/SciPy, k-Nearest Neighbours, linear regression and gradient descent. See Andrew Ng's coursera course Weeks 1 and 2, Notes, part 1 from CS229, and Hastie et al. 2.1-2.3. Gradients: Understanding the Gradient, Understanding Pythagorean Distance and the Gradient
The needed multivariate calculus: see the first three videos here.
Q: for functions of two variables, what's the unit vector that points in the "downhill" direction on the surface plot? (answer.)
Coming up: Gradient descent in multiple variables, linear classifiers, multiple linear regression (see Weeks 1-3 from Andrew Ng's course). Learning with maximum likelihood (Goodfellow et al. 5.5).
Coming up: Inference using maximum likelihood and Bayesian inference (see here, Goodfellow at el. 5.5 and 5.6). Overfitting in regression. Intro to Neural Networks (Bishop Ch. 5, Week 4 from Andrew Ng's course.)
Just for fun: The first joke on Yann LeCun's Geoff Hinton Facts page is relevant.
Coming up: More neural networks, backpropagation. See the Bishop chapter on Neural Networks, week 5 of Andrew Ng's course, and Goodfellow et al Ch. 6.
Video tutorial: Forward propagation: setup, Forward propagation: Vectorization, Backprop for a specific weight, part 1, Backprop for a specific weight, part 2, Backprop for a generic weight, Backprop for a generic weight: vectorization
Training neural networks: Vectorizing neural networks.
Interpreting neural networks: How neural networks see.
Coming up: Overfitting and preventing overfitting (see Ch. 7-7.1 in Goodfellow et al). Convolutional neural networks (see the cs231n notes, and Chapter 9 in Goodfellow et al).
Demo: TensorFlow playground.
ConvNets: intro to convolutional networks
Coming up: a bit of neuroscience (not testable material). Modern ConvNet architectures: see here.
Coming up: A bit of optimization for neural networks (see here and here). Interpreting ConvNets (see here). Neural Style (see slides, or the paper for full details). Generative classifiers intro (see Bishop 1.5.3-1.5.4, 2.3.3-2.3.4, 4.2 and Goodfellow 5.5-5.7.1), Andrew Ng's notes, and my notes.
Intro to generative classifiers and Bayesian inference. See notes, Bishop 4.2.
Coming up: Mixtures of Gaussians and EM. See Andrew Ng's notes, Bishop 9.1-9.2. Principal Components Analysis (Bishop 12.1)
Just for fun: Radford Neal and Geoffrey Hinton, A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants.
Coming up: midterm review
Week 8.5 (to be covered at some point) PCA
Coming up (tentatively): Reinforcement Learning and Policy Gradient methods for RL. See Ch. 1 and Ch. 13.1-13.3 of Sutton and Barto.
Coming up (tentatively): Intro to Recurrent Neural Networks. See Ch. 10 of Goodfellow et al.
Just for fun: Deep Spreadsheets with ExcelNet.
Just for fun: Recurrent Donald Trump
Coming up: RNNs continued, Markov Chain Monte Carlo and Bayesian Inference. See this tutorial
Breaking news: RL with Evolutionary Strategies/Hill Climbing
Just for fun: to read more on babies learning word meanings with unsupervised learning, see e.g. Paul Bloom, How Children Learn The Meanings of Words (The MIT Press, 2002)
All project submission will be done electronically, using the MarkUs system. Log in to MarkUs.
For group projects, to submit as a group, one of you needs to "invite" the other to be partners, and then the other student needs to accept the invitation. To invite a partner, navigate to the appropriate Assignment page, find "Group Information", and click on "Invite". You will be prompted for the other student's CDF user name; enter it. To accept an invitation, find "Group Information" on the Assignment page, find the invitation listed there, and click on "Join". Only one student must invite the other: if both students send an invitation, then neither of you will be able to accept the other's invitation. So make sure to agree beforehand on who will send the invitation! Also, remember that, when working in a group, only one person must submit solutions.
To submit your work, again navigate to the appropriate Exercise or Assignment page, then click on the "Submissions" tab near the top. Click "Add a New File" and either type a file name or use the "Browse" button to choose one. Then click "Submit". You can submit a new version of any file at any time (though the lateness penalty applies if you submit after the deadline) — look in the "Replace" column. For the purposes of determining the lateness penalty, the submission time is considered to be the time of your latest submission.
Once you have submitted, click on the file's name to check that you have submitted the correct version.
Web-based LaTeX interfaces: Overleaf, ShareLaTeX
TeXworks, a cross-platform LaTeX front-end. To use it, install MikTeX on Windows and MacTeX on Mac
Detexify2 - LaTeX symbol classifier
The LaTeX Wikibook.
Additional LaTeX Documentation, from the home page of the LaTeX Project.
LaTeX on Wikipedia.
Policy on special consideration
Special consideration will be given in cases of documented and serious medical and personal cirumstances.