Glove-Talk: A Neural Network Interface Between a
Data-Glove and a Speech Synthesizer
S. Sidney Fells and Geoffrey E. Hinton
Department of Computer Science
University of Toronto
Abstract
To illustrate the potential of multilayer neural networks for adaptive interfaces, we used
a VPL Data-Glove connected to a DECtalk speech synthesized via five neural networks to
implement a hand-gesture to speech system. Using minor variations of the standard
back-propagation learning procedure, the complex mapping of hand movements to speech is
learned using data obtained from a single 'speaker' in a simple training phase. With
a 203 gesture-to-word vocabulary, the wrong is produced less than 1% of the time, and no
word is produced about 5% of the time. Adaptive control of the speaking rate and
word stress is also available. The training times and final performance speed are
improved by using small, separate networks for each naturally defined subtask. The
system demonstrates that neural networks can be used to develop the complex mappings
required in a high bandwidth interface that adapts to the individual user.
Download: [ps] [pdf]
[home page] [publications]