STA314H1F: Statistical Methods for Machine Learning I

Overview

Machine learning (ML) is a set of techniques that allow computers to learn from data and experience, rather than requiring humans to specify the desired behaviour by hand. ML has become increasingly central both in statistics as an academic discipline, and in the data science industry. This course provides a broad introduction to commonly used ML methods, as well as the key statistical concepts underlying ML. It serves as a foundation for more advanced courses, such as STA414 (Statistical Methods for Machine Learning II).

We will cover statistical methods for supervised and unsupervised learning from data: training error, test error and cross-validation; classification, regression, and logistic regression; principal components analysis; stochastic gradient descent; decision trees and random forests; k-means clustering and nearest neighbour methods. Computational tutorials will support the efficient application of these methods.

Announcements

• Midterm marks are out. If you did worse than you expected, I will be hosting extra office hours for you to come discuss with me on Tuesday Nov. 2 from 11AM-1PM.
• I made a mistake requiring HW2 groups on Quercus, so I am giving a blanket 24 hour extension for HW2. See Quercus for details.
• An important error in HW2 has been corrected, please get latest version (V2).
• Tutorial groups and in-person tutorial enrollments are now settled and on Quercus.
• HW1 is up.
• There is now a single Zoom meeting for all online delivery. See Quercus for details.
• Reminder that lecture and tutorials are fully online for the first two weeks.
• Syllabus for the course is available.

Course Information

Unless otherwise specified, lectures and tutorials will be held synchronously, either online via Zoom or in-person. There will be two mandatory tests held during the scheduled class time. Please see the syllabus for detailed policies (marking, lateness, etc.) and attendance instructions.

Staff

Lectures

Tutorials

Students are enrolled in tutorial groups based on their stated preferences for in-person vs. online. Tutorial groups can be checked on Quercus via the People tab. Do not attend an in-person tutorial section unless you are enrolled in that section on Quercus.

Homework

The course will have four homeworks, due at 11:59pm on Thursdays on the week that they are due. They will be submitted through Quercus. This is a tentative schedule, and any changes will be announced.

Tests

The course will have two tests, each with a duration of 1 hour and held during the normal class time. You must take the test with your assigned section, unless you have prior permission from the instructor. Please note, the lecture schedule on both days will be somewhat unusual; see details below.

Practice Midterms

Please be aware that these are from a different series of courses that may vary in difficulty and may have covered different material. Nevertheless, you should be able to get a sense for the style of questions that I may ask, and it will be very good practice to work through these.

• CSC411 F17 midterm.
• CSC411 W18 midterm with solutions.
• CSC411 F18 midterm 1 and CSC411 F18 midterm 2 with solutions.
• CSC311 F19 midterm with solutions. Just do Q1 and Q5.

Lectures

This is a preliminary schedule; it may change throughout the term. The suggested readings (see legend below) are completely optional, but recommended.

Suggested References

• PRML = Pattern Recognition and Machine Learning, by Chris Bishop.
• ESL = The Elements of Statistical Learning, by Hastie, Tibshirani, and Friedman.
• LTFP = Learning Theory from First Principles, by Fancis Bach.
• Information Theory, Inference, and Learning Algorithms, by David MacKay.
• Bayesian Reasoning and Machine Learning, by David Barber.
• Reinforcement Learning: An Introduction, by Sutton and Barto.

Computing

Python Programming Language

For the homework assignments, we will use Python, and libraries such as NumPy, SciPy, and scikit-learn. We will not be expecting you to know advanced Python programming, however we will expect that you are able to do the following.

• navigate the file structure of a basic Python project,
• execute and read through Python scripts while understanding the distinction between code that executes and comments,
• basic Python programming, including calling and defining functions; assigning variables, evaluating expressions, the basics of loops, etc.
• read NumPy, SciPy, or scikit-learn documentation to understand their API.

Python Tutorials

There are a number of great Python tutorials on the web.

• Real Python has a variety of great tutorials to get you started. In particular, this NumPy tutorial.
• The official Python tutorial is quite detailed and comprehensive (has more than you'll need to know).

Using Python

There are a few options for running Python yourself.

• The easiest option is probably to install everything yourself on your own machine.

  • If you don’t already have python, install it. We recommend using Anaconda. You can also install python directly using the instructions here.

  • Optionally, create a virtual environment for this class and step into it. If you have a conda distribution run the following commands:

    conda create --name sta314
    source activate sta314

  • Use pip to install the required packages pip install scipy numpy autograd matplotlib jupyter sklearn
• You may also consider using UofT Jupyter Hub or Google Colab (if you have access to a Google account).