home page
people
research
publications
seminars
travel
search
UCL
Recognizing Hand-written Digits Using Hierarchical products of Experts
Guy Mayraz & Geoffrey Hinton
Gatsby Computational Neuroscience Unit
University College London
17 Queen Square, London WC1N 3AR, UKAbstract
The product of experts learning procedure [1] can discover a set of stochastic binary features that constitute a non-linear generative model of hand-written images of digits. The quality of generative models learned in this way can be assessed by learning a separate model for each class of digit and then comparing the unnormalized probabilities of test images under the 10 different class-specific models. To improve discriminative performance, each of the 10 digit models can be given more layers of feature detectors. The layers are trained sequentially and each layer learns a generative model of the patterns of feature activities in the preceding layer. After training, each layer of feature detectors produce a separate, unnormalized log probability score. With three layers of feature detectors in each of the 10 digit models, a test image produces 30 scores which can be used as inputs to a supervised, logistic classification network that is trained on separate data. On the MNIST database, our system is comparable with current state-of-the-art discriminative methods, demonstrating that the product of experts learning procedure can produce effective generative models of high-dimensional data.
Submitted to Advances in Neural Information Processing Systems 13, MIT Press, Cambridge, MA