NIPS*03 Workshop Saturday December 13th
Nonparametric Bayesian Methods and Infinite Models

Opening Remarks - Matt Beal and Yee Whye Teh

Introductory Tutorial on Nonparametric Methods and Infinite Models
Radford M. Neal (Toronto)

Talk slides: [pdf]

Hierarchical models with multiple mixtures of Dirichlet processes
Michael Escobar, George Tomlinson and Christine McLaren (Toronto, Toronto, UCI)

This talk will discuss the use of hierarchical models with multiple layers of mixtures of Dirichlet processes. For example, with modern diagnostic equipment, one can measure the individual cell sizes from a sample of ones blood. One can then use a mixture of Dirchlet process to model the distribution cell sizes for each distribution. This model is somewhat equivalent to putting a distribution on the family of kernel density estimates. Escobar and West (1995) showed how the kernel density estimator approximates a Bayesian method of estimating denisties based on a mixture of Dirichlet processes (MDP). However, since the MDP method is a proper Bayesian model, one can use hierarchical priors and calculate posterior distributions of functionals of interest.

Using these techniques, we develop a highly flexible hierarchical model in the space of distributions. This model allows us to model samples of densities and to find outliers in the space of distributions. This talk will discuss the methods used to compute these models and to assess outliers. These techniques will be used to identify diseased subjects based on the distribution of the size of the subject's red blood cells.
Extended abstract: [pdf]

The Hierarchical Dirichlet Process
Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei (Berkeley x2, Toronto, Berkeley)

Certain interesting data sets can naturally be grouped into subsets, each of which having its own distinct features. Some of these data sets can be well characterized as having arisen from a model consisting of several mixture models, and the aim is to extract features (mixture components) that explain the data well.
Dirichlet processes (DPs) are often used as a nonparametric replacements for mixture models, but there is the problem that naively combining independent DPs as components of a multiple mixtures model produces a probabilistically unsound model. We describe the hierarchical Dirichlet process (HDP), a new Bayesian nonparametric model which is well suited to this modelling task and can solve this problem. An HDP consists of a local Dirichlet process mixture model for each subset of the data, and a global Dirichlet process which describes the properties of the whole data set, and which ties together the different local Dirichlet processes. We explore the properties of this model, give Markov chain Monte Carlo sampling methods for posterior inference, and show it working on some problems.
Talk slides: [ppt]

Break

Application of nonparametric Bayesian methods in genetic inference
Eric P. Xing, Roded Sharan, Michael I. Jordan (Berkeley)

The problem of inferring haplotypes from genotypes of single nucleotide polymorphisms (SNPs) is essential for the understanding of genetic variation within and among populations, with important applications to the genetic analysis of disease propensities and other complex traits. In this paper we present a novel statistical model for haplotype inference. Our model is a Bayesian model based on a prior known as the Dirichlet process, a nonparametric prior which provides control over the size of the unknown pool of population haplotypes. The model also incorporates a likelihood that allows statistical errors in the haplotype/genotype relationship, trading off these errors against the size of the pool of haplotypes. We describe an algorithm based on Markov chain Monte Carlo for posterior inference. The overall result is a flexible Bayesian model that is reminiscent of parsimony methods in its preference for small haplotype pools. We apply this new approach to the analysis of both simulated and real genotype data, and compare to extant methods.
Extended abstract: [ps]

Approximate Solutions to Nonparametric Bayesian Hierarchical Modelling with Applications to Information Filtering
Volker Tresp, Kai Yu and Anton Schwaighofer (Siemens AG, Munich, Graz U.of Technology)

In some cases---for example in applications involving hierarchical Bayes---one learns a ``prior'' parameter distribution from repeated experiments. It often occurs that the ``learned prior'' does not correspond to a family of distributions which can easily be specified. In this paper, we present a solution to this problem by formulating our prior in terms on an infinite-dimensional Dirichlet process. This usage of a Dirichlet process in a statistical model is sometimes referred to as Dirichlet enhancement and is a main approach in nonparametric hierarchical Bayesian modeling. Typically, nonparametric hierarchical modeling relies on an efficient implementation of Gibbs sampling. We demonstrate how Gibbs sampling can be used in our context. In addition we present two novel non sample-based approximations. The first approximation is based on a MAP approximation using an EM algorithm. The second one is based on a variational approximation. We apply a Dirichlet enhanced hierarchical model to information filtering where nonparametric hierarchical modeling allows the principled combination of both content filtering and collaborative filtering. We demonstrate the effectiveness of our approximation using two applications: the retrieval of art images and the information filtering of Reuters news data.
Extended abstract: UAI paper [pdf]

Extended Bayesian Statistical Inference and Renormalization Group
Toshiaki Aida (Okayama Japan)

The work discusses the improvement and the generalization of the predictive distribution in Bayesian statistical inference in a most general setting.
(1) I have made clear what is the origin of the effectiveness of Bayesian statistical inference, applying a field theoretical argument to a few examples of non-parametric models, which also let us know the connection between Bayesian framework and renormalization group.
(2) Following the above, the predictive distribution in Bayesian statistical inference is generalized from the point of view of renormalization group, which is similar (but not equal ) to the modification of a prior distribution and leads to a general prescription in order to obtain faster decay of prediction error.
(3) These are discussed generally without our assuming any specific models. Our argument is so general that it can be applied to any type of models: non-parametric, parametric, linear, non-linear...
Extended abstract: [ppt]