CIAR Second Summer School Tutorial
Lecture 1b

Contrastive Divergence
and
Deterministic Energy-Based Models

Restricted Boltzmann Machines

A picture of the Boltzmann machine learning algorithm for an RBM

The short-cut

Contrastive divergence

Contrastive divergence

How to learn a set of features that are good for reconstructing images of the digit 2

Slide 8

How well can we reconstruct the digit images from the binary feature activations?

Another use of contrastive divergence

Energy-Based Models with deterministic hidden units

Frequently Approximately Satisfied constraints

Reminder:
Maximum likelihood learning is hard

Hybrid Monte Carlo

Slide 15

Backpropagation can compute the gradient that Hybrid Monte Carlo needs

The online HMC learning procedure

The shortcut

A simple 2-D dataset

The network for the 4 squares task

Slide 21

Slide 22

Slide 23

Slide 24

Slide 25

Slide 26

Slide 27

Slide 28

Slide 29

Slide 30

Slide 31

Slide 32

Learning the constraints on an arm

Slide 34

Superimposing constraints

Dealing with missing inputs

Learning constraints from natural images
(Yee-Whye Teh)

Slide 38

Slide 39

How to learn a topographic map

Slide 41

Slide 42

THE END

Independence relationships of hidden variables
 in three types of model that have one hidden layer

Faster mixing chains

Pro’s and Con’s of Gibbs sampling

Slide 47

Over-complete ICA
using a causal model

Over-complete ICA
using an energy-based model