Unsupervised “pre-training” also helps for
models that have more data and better priors
Ranzato et. al. (NIPS 2006) used an additional
600,000 distorted digits.
They also used convolutional multilayer neural
networks that have some built-in, local
pre-training followed by backprop: