Dept. of Computer Science, University of Toronto

An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up ``recognition'' connections convert the input into representations in successive hidden layers, and top-down ``generative'' connections reconstruct the representation in one layer from the representation in the layer above. In the ``wake'' phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the ``sleep'' phase, neurons are driven by generative connections, and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above.

*Science*, vol. 268, pp. 1158-1161 (1995).

Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The Helmholtz machine'',The wake-sleep algorithm is used to learn factor analysis models in the following technical report:Neural Computation, vol. 7, pp. 1022-1037: abstract.

Neal, R. M. and Dayan, P. (1996) ``Factor analysis using delta-rule wake-sleep learning'', Technical Report No. 9607, Dept. of Statistics, University of Toronto, 23 pages: abstract, postscript, pdf, associated references, associated software.The wake-sleep algorithm is used to model temporal sequences in the following conference paper:

Hinton, G. E., Dayan, P., To, A., and Neal, R. (1995) ``The Helmholtz machine through time'', in F. Fogelman-Soulie and R. Gallinari (editors)ICANN-95, pp. 483-490: abstract.