Mean Field Networks that Learn to
Discriminate Temporally Distorted Strings
Christopher Williams, Geoffrey Hinton
Department of Computer Science
University of Toronto
Abstract
Neural networks can be used to discriminate between very similar
phonemes and they can handle the variability in time of occurrence by using a time-delay
architecture followed by a termporal integration (Land, Hinton and Waibel, 1990). So
far, however, neural networks have been less successful at handling longerduration events
that require something equivalent to 'time warping' in order to match stores knowledge to
the data. We present a type of mean field network (MFN) with tied weights that is
capable of approximating the recognizer for a hidden markov model (HMM). In the
process of settling to a stable state, the MFN finds a blend of likely ways of generating
the input string given its internal model of the probabilities of transitions between
hidden states and the probabilities of input symbols given a hidden state. This
blend is a heuristic approximation to the full set of path probabilities that is
implicitly represented by an HMM recognizer. The learning algorithm for the MFN is
less efficient than for an HMM of the same size. However, the MFN is capable of
using distributed representations of the hidden state, and this can make it exponentially
more efficient than an HMM when modelling strings produced by a generator that itself has
componential states. We view this type of MFN as a way of allowing more powerful
representations without abandoning the automatic parameter estimation procedures that have
allowed relatively simple models like HMM's to outperform complex AI representations on
real tasks.
Download [ps] [pdf]
Touretzky, D. S., Elman, J. L., Sejnowski, T. J. and
Hinton, G. E. (Eds.) Connectionist Models: Proceedings of the 1990 Connectionist Summer
School. Morgan Kauffman: San Mateo, CA.
[home page] [publications]