CIAR Summer School Tutorial
Lecture 2b

Learning a Deep Belief Net

A neural network model of digit recognition

The generative model

Learning by dividing and conquering

Another way to divide and conquer

Why its hard to learn one layer at a time

Using complementary priors to eliminate explaining away

An example of a complementary prior

Inference in a DAG with replicated weights

A picture of the Boltzmann machine learning algorithm for an RBM

"The learning rule for a..."

Pro’s and con’s of replicating the weights

Contrastive divergence learning:
 A quick way to learn an RBM

Multilayer contrastive divergence

A simplified version with all hidden layers the same size

Why the hidden configurations should be treated as data when learning the next layer of weights

Why greedy learning works

Back-fitting

Samples generated by running the top-level RBM with one label clamped. There are 1000 iterations of alternating Gibbs sampling between samples.

Examples of correctly recognized MNIST test digits (the 49 closest calls)

How well does it discriminate on MNIST test set with no extra information about geometric distortions?

Slide 22

Samples generated by running top-level RBM with one label clamped. Initialized by an up-pass from a random binary image. 20 iterations between samples.

The wake-sleep algorithm

The flaws in the wake-sleep algorithm

The up-down algorithm:
A contrastive divergence version of wake-sleep

Mode averaging

The receptive fields of the first hidden layer

The generative fields of the first hidden layer

A different way to capture low-dimensional manifolds