SMEM Algorithm for Mixture Models
Naonori Ueda, Ryohei Nakano
NTT Communication Science Laboratories
Hikaridai, Seika-cho, Soraku-gun
Kyoto 619-0237, Japan
Zoubin Ghahramani, Geoffrey Hinton
Gatsby Computational Neuroscience Unit
University College London
17 Queen Square, Alexandra House
London WC1N 3AR, UK
Abstract
We Present a split and merge EM (SMEM) algorithm to overcome the
local maxima problem in parameter estimation of finite mixture models. In the case
of mixture models, local maxima often involve having too many components of a mixture
model in one part of the space and too few in another, widely separated part of the space.
To escape from such configurations we repeatedly perform simultaneous split and
merge operations using a new criterion for efficiently selecting the split and merge
candidates. We apply the proposed algorithm to the training of Gaussian mixtures and
mixtures of factor analyzers using synthetic and real data and show the effectiveness of
using the split and merge operations to improve the likelihood of both the training data
and of held-out test data. We also show the practical usefulness of the proposed
algorithm by applying it to image compression and pattern recognition problems.
Download [ps.gz] [pdf]
Neural Computation (in press)
[home page] [publications]