NIPS 2010 Workshop on Transfer Learning Via Rich Generative Models.

Organizers:

• Ruslan Salakhutdinov , Department of Brain and Cognitive Sciences and CSAIL, MIT
• Ryan Adams , Department of Computer Science, University of Toronto.
• Josh Tenenbaum , Department of Brain and Cognitive Sciences and CSAIL, MIT.
• Zoubin Ghahramani , Department of Engineering, University of Cambridge.
• Tom Griffiths , Department of Psychology, University of California, Berkeley.

Motivation and Topics:

Intelligent systems must be capable of transferring previously-learned abstract knowledge to new concepts, given only a small or noisy set of examples. This transfer of higher order information to new learning tasks lies at the core of many problems in the fields of computer vision, cognitive science, machine learning, speech perception and natural language processing.

Over the last decade, there has been considerable progress in developing cross-task transfer (e.g., multi-task learning and semi-supervised learning) using both discriminative and generative approaches. However, many existing learning systems today can not cope with new tasks for which they have not been specifically trained. Even when applied to related tasks, trained systems often display unstable behavior.

More recently, researchers have begun developing new approaches to building rich generative models that are capable of extracting useful, high-level structured representations from high-dimensional sensory input. The learned representations have been shown to give promising results for solving a multitude of novel learning tasks, even though these tasks may be unknown when the generative model is being trained. A few notable examples include learning of Deep Belief Networks (Hinton et.al.), Deep Boltzmann Machines (Salakhutdinov and Hinton), deep nonparametric Bayesian models (Adams, Wallach and Ghahramani), as well as Bayesian models inspired by human learning (Perfos and Tenenbaum, Canini and Griffiths).

"Learning to learn" new concepts via rich generative models has emerged as one of the most promising areas of research in both machine learning and cognitive science. Although there has been recent progress, existing computational models are still far from being able to represent, identify and learn the wide variety of possible patterns and structure in real-world data. The goal of this workshop is to assess the current state of the field and explore new directions in both theoretical foundations and empirical applications.

During the course of the workshop, we shall be interested in discussing the following topics:

• State of the field: What are the existing methods and what is the relationship between them? Which problems can be solved using existing learning algorithms and which require fundamentally different approaches?
• Learning structured representations: How can machines extract invariant representations from a large supply of high-dimensional highly-structured unlabeled data? How can these representations be used to represent and learn tens of thousands of different concepts (e.g., visual object categories) and expand on them without disrupting previously-learning concepts? How can these representations be used in multiple applications?
• Transfer Learning: How can previously-learned representations help learning new tasks so that less labeled supervision is needed? How can this facilitate knowledge representation for transfer learning tasks?
• One-shot learning: For many traditional machine classification algorithms, learning curves are measured in tens, hundreds or thousands of training examples. For humans learners, however, just a few training examples is often sufficient to grasp a new concept. Even if these examples belong to an entirely different domain, it is possible for brains to make meaningful generalizations to novel instances. Can we develop rich generative models that are capable of efficiently leveraging background knowledge in order to learn novel categories based on a single training example? Are there models suitable for deep transfer, or generalizing across domains, when presented with one or few examples?
• Deep learning: Recently, there has been notable progress in learning deep probabilistic generative models, including Deep Belief Networks, Deep Boltzmann Machines, deep nonparametric Bayesian models, that contain many layers of hidden variables. These models have been successfully used in many application domains including visual object recognition, dimensionality reduction, information retrieval, and robotics. Can these models be extended to transfer learning tasks as well as learning new concepts with only one or few examples? Can we use representations learned by the deep models as an input to more structured hierarchical Bayesian models?
• Scalability and success in real-world applications: How well do existing transfer learning models scale to large-scale problems including problems in computer vision, natural language processing, and speech perception? How well do these algorithms perform when applied to modeling high-dimensional real-world distributions (e.g. the distribution of natural images)?
• Theoretical Foundations: What are the theoretical guarantees of learning good generative models? Under what conditions it is possible to provide performance guarantees for such algorithms?
• Suitable tasks and datasets: What are the right datasets and tasks that could be used in future research on the topic and to facilitate comparisons between methods?

Through having a series of invited talks, poster session, and a panel discussion, our workshop is targeted at researchers both within the NIPS community and outside, in the fields of machine learning, computer vision, natural language processing, and cognitive science focusing on learning generative models with a particular emphasis on transfer and multi-task learning. Our hope for this workshop is to catalyze the growing community of researchers working on learning rich generative models, assess the current state of the field, discuss key challenges, and identify future promising directions of investigation.