• We will combine two types of unsupervised neural net:
– Undirected model  =  Boltzmann Machine
– Directed model      =  Sigmoid Belief Net
• Boltzmann Machine learning is made efficient by
restricting the connectivity & using contrastive divergence.
• Restricted Boltzmann Machines are shown to be
equivalent to infinite Sigmoid Belief Nets with tied weights.
– This equivalence suggests a novel way to learn deep
directed belief nets one layer at a time.
– This new method is fast and learns very good models,
provided we do some fine-tuning afterwards
• We can now learn a really good generative model of the
joint distribution of handwritten digit images and their
– It is better at recognizing handwritten digits than
discriminative methods like SVM’s or backpropagation.