Artificial neural networks are the models behind most of what people mean by “AI” today. They are particularly useful on large, high-dimensional data — images, video, audio, text. We’ll build them up as a statistical model (something in the same family as linear and logistic regression) and also gesture at the biological inspiration that gives them their name.
The plan: start from the simplest possible classifier (nearest neighbour), see that logistic regression is already a tiny neural network, and then add a hidden layer to get a real one. Face recognition is the running example.
Everything in this course that predicts a label has had the same shape. You have a training set of input–output pairs
\[(x^{(1)}, y^{(1)}), \; (x^{(2)}, y^{(2)}), \; \ldots\]
and a function \(f_\theta\) with some coefficients \(\theta\). You choose \(\theta\) so that \(f_\theta(x^{(i)}) \approx y^{(i)}\) on the training set, and you hope that the same \(\theta\) predicts well on new, unseen data (the test set).
This is worth pausing on, because it is genuinely strange that it works. In an intro programming course, “write some code, see that it’s a bit wrong, tweak it, see if it’s better” is a terrible way to write a program — you usually just break it in a new way. In machine learning that loop is exactly the method: guess \(\theta\), measure the error on the training set, nudge \(\theta\) in a direction that lowers the error, repeat. Eventually you land on a good \(\theta\). So far we have found coefficients with lm and glm; neural networks are fit with an algorithm called gradient descent, which we won’t cover — the point is only that finding good coefficients is possible.
Say you have photographs, each labelled with whose face it is, and you want to identify new photos. How does a computer “see” a photo?
A grayscale image is a grid of pixels, each with a brightness — often an integer from 0 (black) to 255 (white). Stack the rows end to end and you get a vector: a \(64 \times 64\) image becomes a vector of \(64 \times 64 = 4096\) numbers. A colour image is the same idea with three numbers (red, green, blue) per pixel; think grayscale for now.
So an image is a point in a high-dimensional space, and “these two photos look alike” should mean “these two vectors are close.” That single idea carries us all the way to neural networks.
The simplest classifier there is. Keep the whole training set around. To label a new image, find the single training image it looks most like, and copy that image’s label.
That’s it. If the new photo is closest to a training photo of Paul, you guess Paul.
To make “looks most like” precise we need a way to compare two vectors.
The obvious measure is Euclidean distance — the straight-line distance between the two endpoints:
\[d(a, b) = \sqrt{\sum_j (a_j - b_j)^2}.\]
For images this has a nasty flaw. Take an image, multiply every pixel by 0.5. It is the same picture, just darker — visually the same face — but every coordinate has moved halfway toward the origin (all-black), so the Euclidean distance from the original can be large. Distance is polluted by overall brightness.
Cosine similarity fixes this by looking at the angle between the vectors instead of their endpoints. Two vectors that point the same way have angle 0 regardless of length:
\[\cos\theta = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}, \qquad a \cdot b = \sum_j a_j b_j.\]
The numerator \(a \cdot b\) is the dot product. The denominator divides out the magnitudes, so scaling an image up or down (changing its overall brightness) doesn’t change the cosine. If all your images happen to have roughly the same average brightness — the same magnitude — you can even skip the denominator and just rank by the dot product: large dot product = similar.
Let’s see the brightness problem directly. Two little “images” as vectors:
a <- c(10, 200, 50, 180) # some image
b <- a * 0.5 # the SAME image, half as bright
euclidean <- function(u, v) sqrt(sum((u - v)^2))
cosine <- function(u, v) sum(u * v) / (sqrt(sum(u^2)) * sqrt(sum(v^2)))
euclidean(a, b) # large, even though it's the same picture## [1] 136.9306cosine(a, b) # exactly 1: same direction, angle 0## [1] 1Euclidean distance calls the darker copy “far”; cosine similarity correctly calls it identical.
A tiny, honest version of the whole algorithm. Four 3×3 “images” (flattened to length-9 vectors) as the training set, one new image to classify:
# Training "images": each row is a flattened 3x3 grayscale grid.
train_imgs <- rbind(
vertical = c(0,9,0, 0,9,0, 0,9,0), # a vertical stroke
horizontal= c(0,0,0, 9,9,9, 0,0,0), # a horizontal stroke
diagonal = c(9,0,0, 0,9,0, 0,0,9) # a diagonal stroke
)
labels <- rownames(train_imgs)
new_img <- c(0,8,0, 0,7,0, 1,9,0) # looks like a vertical stroke
# Score the new image against each training image by cosine similarity,
# then copy the label of the best match.
sims <- apply(train_imgs, 1, function(row) cosine(row, new_img))
sims## vertical horizontal diagonal
## 0.9922779 0.2894144 0.2894144labels[which.max(sims)]## [1] "vertical"The new image matches the vertical template best, so 1-NN calls it vertical. Nearest-neighbour is dead simple and surprisingly hard to beat, but it has to store and search the entire training set, and raw pixel similarity is a crude notion of “looks alike.” Neural networks learn a better notion.
Now suppose there are six people to recognise. Here is the simplest network — it is exactly multinomial logistic regression (logistic regression with more than two classes).
Plug in the image \(x = (x_1, \ldots, x_{4096})\). For each person \(k\) compute a score
\[z_k = \sigma\!\Big(\sum_j w_{kj}\, x_j + b_k\Big),\]
one weight \(w_{kj}\) per pixel per person, squashed by the sigmoid \(\sigma\) from the logistic regression notes. You get one \(z_k\) per possible name. In a picture:
x1 ─┐
x2 ─┼─► z1 (score for person 1)
. ├─► z2 (score for person 2)
. │ ...
x4096┴─► z6 (score for person 6)Each arrow is a weight \(w_{kj}\). We choose all the weights \(W\) and biases \(b\) so that, on the training set, plugging in a photo of person 1 makes \(z_1\) high and the other scores low; plugging in person 6 makes \(z_6\) high; and so on — just like glm finds coefficients, only there are a lot more of them. To classify a new photo, compute all six scores and report whichever \(z_k\) is largest.
Here’s the interpretation that makes the rest of the lecture click. The score for person \(k\) is \(\sigma(w_k \cdot x + b_k)\) — it is large exactly when the dot product \(w_k \cdot x\) is large, i.e. when the image \(x\) points in the same direction as the weight vector \(w_k\). So \(w_k\) acts as a template for person \(k\): the network says “person \(k\)” when the input looks like person \(k\)’s template, measured by the same dot product we used for nearest-neighbour.
If that story is right, we should be able to see it. Take a trained network, pull out the weights \(w_k\) that compute each person’s score, reshape that length- 4096 vector back into a \(64 \times 64\) grid, and look at it. What you get are ghostly, blurry faces — you can squint and see a face-like blob, but no individual is sharp. One template per person is simply too few: a single template can’t cover you facing left, facing right, smiling, in shadow. It “kind of works but kind of not.”
(Our eyes over-cooperate here — humans see faces in clouds and burnt toast — so be a little skeptical of how face-like the blobs look.)
The fix: allow many templates, not just one per person. That’s a hidden layer.
Before computing the six output scores, compute an intermediate layer of \(K\) values \(h_1, \ldots, h_K\) — the hidden layer — each one its own template match:
\[h_m = \sigma\!\Big(\sum_j w^{(1)}_{mj}\, x_j + b^{(1)}_m\Big), \qquad z_k = \sigma\!\Big(\sum_m w^{(2)}_{km}\, h_m + b^{(2)}_k\Big).\]
In a picture, with the hidden layer in the middle:
x1 ─┐ ┌─► h1 ─┐
x2 ─┼──────► ┤ h2 ├──────► z1
. │ │ .. │ z2
. │ └─► hK ─┘ ...
x4096┘ z6
inputs hidden outputs
(pixels) (templates) (names)Read it the same way. Each hidden unit \(h_m\) is a template compared against the raw image; the outputs \(z_k\) are then computed from how well the image matched each of the \(K\) templates, rather than directly from the pixels. With, say, \(K = 200\) hidden units you have 200 templates to spread across all the poses and lighting conditions of all six people.
Visualise the hidden templates — reshape each \(w^{(1)}_m\) back into an image — and now they look strikingly like faces. Some are generic face-parts; some look like one specific training photo. That second kind is the network memorising: it sets a template equal to a hard training image so that \(h_m\) lights up only for that exact image, then reads the answer off that. Memorising training images makes training accuracy excellent and says nothing about new photos — this is overfitting, and it is why we always judge a model on a held-out set.
Everything is still “just a formula with coefficients we fit on the training set.” The template story is only an interpretation — but it’s a good one, and it holds up when you look at the learned weights.
The one-hidden-layer picture — global templates, one comparison, done — is true but simplified. Real deep networks stack many hidden layers:
\[x \to h^{(1)} \to h^{(2)} \to \cdots \to h^{(L)} \to z.\]
Two ways to read this:
For a first-layer unit we could reshape its weights into an image and look. A deeper unit’s inputs are the \(h\)’s, not pixels, so its weights don’t form a picture. So we borrow Hubel & Wiesel’s trick (below): feed the network many images and see which inputs make the unit fire hardest — those images are its “preferred stimulus.”
Do this for a network trained on ImageNet (the 1000-category, ~million-image dataset that kicked off the deep-learning era in 2012) and a clean hierarchy appears:
The deeper you go, the more abstract the concept. (Caveat: a “dog” unit hasn’t learned what dogs are; among only 1000 options, a few characteristic patches are enough to bet “dog.” Don’t over-read it.) The same principle runs modern image-understanding — paste a photo into an LLM and it will categorise it.
Why should stacking layers work so well? An argument due to Ilya Sutskever (a co-founder of OpenAI). Distinguish two properties a model can have:
Deep networks are rare in being both. The hypothesis then rides on a fact about the brain: humans do perceptual tasks — “dog / cat / unicycle” — in about a tenth of a second. Neurons communicate by firing rate and can only fire on the order of 100 times per second, so in 0.1 s a chain of neurons can be at most ~10 steps deep. If a person can do a task in that budget, a network ~10 layers deep ought to be able to as well. It has held up: fast networks with modest depth now classify objects, and even judge “shape” (position quality) in the board game Go better than expert humans — who themselves judge it by fast intuition, not explicit search.
The name is a metaphor, but not an empty one.
A neuron is a cell that, when stimulated enough, “fires” an electrical pulse that in turn stimulates other neurons. The human brain has on the order of 100 billion of them, and the idea is that cognition arises from neurons firing and triggering each other in cascades.
The clearest evidence comes from the visual cortex — the brain region that handles early vision. In a Nobel-winning experiment, Hubel and Wiesel recorded from a single neuron in a cat’s visual cortex while showing it patterns, and could hear the neuron fire (as clicks) only for a specific stimulus — e.g. a bar at one particular orientation, moving one particular way. There is, quite literally, a neuron that fires for a diagonal edge and stays quiet for a horizontal one.
Map that onto the network:
For early vision this picture is well supported; deeper in the brain, neurons do respond to more abstract things, but general cognition is not understood — there is no single neuron that lights up when someone says “neural network” to you. The biology inspired the model; it is not a literal description of it.
lm/glm, but it’s the same idea.