CSC 2535: Advanced Machine
Learning
Lecture 5
Energy-Based Models
Using energies to define probabilities
How to combine simple density models
A picture of the two combination methods
Products of Experts and energies
How sharp are products of experts?
Generating from a product of experts
Relationship to causal generative models
Ways to deal with the intractable sum
The Markov chain for unigauss experts
Good and bad properties of the shortcut
15 axis-aligned uni-gauss experts fitted to 24 clusters (one cluster is missing from the grid)
Fantasies from the
model
(it fills in the missing cluster)
Energy-Based Models with deterministic hidden units
Reminder:
Maximum likelihood learning is hard
The leapfrog method for keeping numerical errors small.
Combining the last move of one interval with the first move of the next interval
Dealing with the remaining numerical error
Backpropagation can compute the gradient that Hybrid Monte Carlo needs
The online HMC learning procedure
The network for the 4 squares task
A different kind of hidden structure
Frequently Approximately Satisfied constraints
Frequently Approximately Satisfied constraints
Learning the constraints on an arm
Learning constraints from
natural images
(Yee-Whye Teh)
How to learn a topographic map
Pro’s and Con’s of Gibbs sampling
Independent Components Analysis
Independence relationships
of hidden variables
in three types of model that have one
hidden layer