Course logistics

Course introduction (Rmd): goals, structure, mini-projects, the course project, the culture-of-honor paper as a paradigmatic example.

Programming in R

R basics and functional programming (Rmd): why functional programming, literals, printing, variables, conditionals, functions, print-vs-return.

Vectors (Rmd): c(), indexing, boolean masks, combining with & / | / !, the pie-or-ice-cream gotcha, the doctor-salaries example.

Data frames (Rmd): building data frames, the babynames dataset, accessing rows / columns / cells, $ vs [ ], subsetting with masks.

Wrangling data with the pipe and tidyverse (Rmd): function composition and %>%; filter / select / arrange / rename / mutate / summarize / group_by / distinct / n_distinct / which.max; gapminder worked examples; sapply.

Statistical inference

Probability review (Rmd): random variables and events, conditional probability, independence, Bayes' rule, law of total probability, the boy-and-girl paradox.

Maximum likelihood (Rmd): on paper for Bernoulli, in code via grid search, linear regression as MLE under Gaussian noise.

Bayesian inference (Rmd): prior, likelihood, posterior; the coin example in code; Bayesian inference for the difference of two means (continued in Week 2 with the finches).

Reading: R4DS Ch. 4 (Data transformation); Poldrack Ch. 6 (Probability).

Just for fun: Physician salary data; Tyche, the Greek goddess of chance.

Pre-course slides: MIE1626 introduction slide deck.

Lecture recordings: May 6 — Part 1, Part 2, Part 3; May 8 — Part 1, Part 2.

Hypothesis testing

Hypothesis testing: p-values from first principles (Rmd): Darwin's finches; the null hypothesis; the test statistic; computing p-values by simulation with replicate; one-sided vs two-sided tests.

Interpreting p-values, errors, and the replication crisis (Rmd): the 5% threshold; the ASA statement; Type I / II / S / M errors; publication bias, p-hacking, the garden of forking paths; the dead-salmon fMRI study.

Inference continued

The Bayesian inference topic from Week 1 also covers the posterior over the difference of finch means (worked through during the May 13 lecture).

Reading: Andrew Gelman, "Type 1, type 2, type S, and type M errors"; Bennett et al, “Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon”; OpenIntro Statistics, chapters on inference (free PDF).

Sources for finches data: Sleuth3::case0201.

Lecture recordings: May 13 — Part 1, Part 2.

Predictive modeling

Predictive modeling: linear regression and categorical predictors (Rmd): Portland housing example; cost functions vs probabilistic interpretation; simple vs multiple regression; log-transforms on gapminder; indicator variables for categorical predictors; lm, predict, and manual SSE.

Logistic regression (Rmd): Titanic case study; the sigmoid; glm(..., family = binomial); classification accuracy and the base rate; FPR / FNR / PPV / NPV.

Workflow and a project preview

Scraping data with a coding agent (Rmd): Arctic Shift for Reddit; using an agent inside VS Code; auditing silent stop-word removal; a worked example.

Mini-project 2 walkthrough — ICU adverse outcomes (Rmd): predicting mortality from ICU data; ROC curves; the “doctor with 25 charts” cost model and threshold selection.

Inference with linear regression

Confidence intervals (Rmd): what a CI actually means (and what it does not); reading CIs off summary(glm(...)); the CI ↔ p-value duality.

Regression diagnostics (Rmd): assumptions (linearity, normality, independence of residuals); Q–Q, residuals-vs-fitted, scale-location, residuals-vs-leverage; outliers; multiple comparisons and the F-test.

Correlation, R², and causation (Rmd): reverse causation, common cause, indirect causation, coincidence; R² as comparison to the constant-prediction baseline; Anscombe's quartet.

Training, validation, and test sets (Rmd): why training accuracy lies; three-way split; why categorical encoding cannot hurt training fit; using replicate to estimate validation-accuracy variability.

“The truth about linear regression” (Rmd): Shalizi's “lies”; collinearity; omitted-variable bias (Fisher and smoking; ice cream and drownings); the gender wage gap as a controls question; the Harvard admissions lawsuit; errors in variables; interactions (avocado / bacon).

Reading: Cosma Shalizi, “The Truth About Linear Regression”; OpenIntro Statistics, regression chapters.

Datasets: gapminder, titanic, the ICU data from mini-project 2.

Lecture recordings: May 20 — Part 1, Part 2; May 22 — Part 1, Part 2.

Test statistics and their null distributions

Revisiting hypothesis testing (Rmd) under a unifying frame: two ways to get a null distribution — enumerate it (Lady Tasting Tea, Fisher's exact test) and simulate it (the finches walkthrough from Week 2).

Workshop: analyzing scraped text with an LLM

Classifying and clustering text with an LLM (Rmd): the r/UofT “why are students (un)happy” example; why a single sentiment percentage is uninteresting; keyword vs. fine-tuning vs. just-ask-the-model; zero-shot vs. few-shot (in-context learning); calling an API reproducibly instead of the chat window; ground-truth labelling; embeddings and clustering for a thematic breakdown.

Hierarchical models

Hierarchical (multilevel) models (Rmd): grouped data and the pooling trichotomy (complete pooling, no pooling, partial pooling); the Minnesota radon case study with lmer; Lac Qui Parle and shrinkage with n=2; adding predictors and random slopes; prediction in known vs. new counties; the varying-slopes voting case study (Red State, Blue State, Rich State, Poor State); the Bayesian connection.

Causal inference

Causal inference (Rmd): DAGs as a language for causal structure; the three d-separation patterns (common cause, chain, collider/“explaining away”) verified by simulation on the smoking/asbestos/cancer DAG; the trap of regressing on the wrong variable; P(Y | X) vs. P(Y | do(X)); identification strategies — backdoor adjustment, instrumental variables, matching.

Reading: Gelman & Hill (2007), Data Analysis Using Regression and Multilevel/Hierarchical Models, Ch. 12; Cosma Shalizi, Advanced Data Analysis from an Elementary Point of View, Ch. 21–24.

Datasets: Minnesota radon — topics/data/srrs2.dat (cached locally for reproducibility).

Further reading: Andrew Gelman, Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do (PU Press, 2009); Pearl, Glymour & Jewell, Causal Inference in Statistics: A Primer.

Lecture recordings: May 27 — Part 1, Part 2.

Causal inference, revisited

Causal inference (Rmd): extended with the storks-vs-obstetricians warm-up (per-capita vs. total, additive “confounders,” epistemic humility, mechanistic understanding), the grad-admissions collider (“math people vs. humanities people” as a selection effect), and a real research example where backdoor adjustment paid off.

Machine learning: neural networks

Neural networks (Rmd): faces as pixel vectors; one-nearest-neighbour; cosine similarity vs. Euclidean distance; logistic regression as one template per class; hidden layers as many templates; deep “features of features” and what deep units detect (ImageNet); the deep-learning hypothesis; the Hubel & Wiesel biological inspiration.

How large language models work (Rmd): autoregressive next-word prediction; word embeddings and the king − man + woman ≈ queen arithmetic; the transformer and attention; in-context learning, chain-of-thought, tool use, and agents; the Bender & Koller octopus test.

Communicating results

Data visualization (Rmd): comprehensible / non-misleading / memorable; chart junk; truncated axes; colour vs. shape pop-out; seeing patterns in random data; dashboards. Also: the multiplicative-Poisson restaurant-ranking case study added to the multilevel notes.

Reading: Shalizi, Advanced Data Analysis from an Elementary Point of View, Ch. 21–24 (causal inference); Claus O. Wilke, Fundamentals of Data Visualization (free online).

Just for fun: Matthews (2000), “Storks Deliver Babies (p = 0.008)”; Bender & Koller (2020), “Climbing towards NLU.”

Lecture recordings: June 3 — Part 1; June 5 — Part 1, Part 2.

Multilevel inference, one more example

Shaquille O’Neal’s free throws (Rmd): a binomial multilevel logistic model (glmer with the cbind(makes, misses) syntax); partial pooling across games; reading off the good-day/bad-day spread.

Fairness in machine learning

Fairness in machine learning (Rmd): the COMPAS / ProPublica story; demographic parity, false-positive parity, accuracy parity, calibration, anti-classification; the impossibility theorem when base rates differ; why dropping the protected attribute fails; label bias and feedback loops; causal / counterfactual fairness; model cards.

More models: trees

Decision trees (and k-NN) (Rmd): how kNN, logistic regression, neural nets, and trees carve up the feature space; axis-parallel splits; XOR and parity; greedy learning; random forests; interpretability.

Stepping back: the two cultures

The two cultures, and a predictive workflow (Rmd): Breiman’s data-modeling vs. algorithmic cultures; a recap of the whole course; prediction markets; a Monte-Carlo World Cup bracket simulation built with a coding agent.

Reading: Breiman (2001), “Statistical Modeling: The Two Cultures”; Corbett-Davies et al. (2017), “Algorithmic Decision Making and the Cost of Fairness.”

Datasets: COMPAS / Broward County (from the mini-project); Statistical Sleuth3.

Lecture recordings: June 10 — Part 1, Part 2; June 12 — recording.

Project presentations (June 17)

Three-minute progress presentations; feedback and discussion.

A modern analysis workflow

A modern analysis workflow (with an LLM) (Rmd): using an LLM as a fast — but verified — statistical consultant. The inference workflow via the Pygmalion effect (Sleuth3::case1302, a multilevel model plus model checking); auditing a published active-learning effect by simulation; the prediction workflow (sequestering the test set; random forests; zero-/few-shot LLM classification; feature engineering and stacking on Kaggle; classifying subreddit posts).

Reading: Ramsey & Schafer, The Statistical Sleuth (Pygmalion, case 1302); Deslauriers et al. (2019, PNAS), “Measuring actual learning versus feeling of learning…”

Lecture recordings: June 19 — recording.