Program

 

Yoshua Bengio and Yann LeCun
Learning Deep Architectures

This short tutorial on deep learning will review a variety of methods for learning multi-level, hierarchical representations, emphasizing their common traits. While deep architectures have theoretical advantages in terms of expressive power and efficiency of representation, they also provide a possible model for information processing in the mammalian cortex, which seems to rely on representations with multiple levels of abstractions. A number of deep learning methods have been proposed since 2005, that have yielded surprisingly good performance in several areas, particularly in vision (object recognition), and natural language processing. They all learn multiple levels of representation using some form of unsupervised learning. Hypotheses to explain why these algorithms work well will be discussed in the light of new experimental results. Many of these algorithms can be cast in the framework of the energy-based view of unsupervised learning, which generalizes graphical models used as building blocks for deep architectures, such as the Restricted Boltzmann Machines (RBM) and variations of regularized auto-encoders. Old and new algorithms will be presented for training, sampling, and estimating the partition function of RBMs and Deep Belief Networks. Applications of deep architectures to computer vision and natural language processing will be described. A number of open problems and future research avenues will be discussed, with active participation from the audience.


Brief Bio.

Yoshua Bengio (PhD’1991, McGill University) is professor at the Department of Computer Science and Operations Research, Universite de Montreal, and Canada Research Chair in Statistical Learning Algorithms, as well as NSERC-CGI Chair, and Fellow of the Canadian Institute for Advanced Research. His main ambition is to understand how learning can give rise to intelligence. He has been an early proponent of deep architectures and distributed representations as tools to bypass the curse of dimensionality and learn complex tasks. He contributed to many machine learning areas: neural networks, recurrent neural networks, probabilistic graphical models (especially for temporal data), kernel machines, semi-supervised learning, unsupervised learning and manifold learning, pattern recognition, data-mining, natural language processing, machine vision, time-series prediction.

Yann LeCun is Silver Professor of Computer Science at the Courant Institute of Mathematical Sciences of New York University. He received the Electrical Engineer Diploma from Ecole Superieure d'Ingenieurs en Electrotechnique et Electronique (ESIEE), Paris in 1983, and a PhD in Computer Science from Universite Pierre et Marie Curie (Paris) in 1987. After a postdoctoral fellowship at the University of Toronto, he joined the Adaptive Systems Research Department at AT\&T Bell Laboratories in Holmdel, NJ, in 1988. He was named head of the Image Processing Research Department at AT\&T Labs-Research in 1996. In 2002, he became a Fellow of the NEC Research Institute in Princeton, before moving to NYU in 2003. Yann's research interests include computational and biological models of learning and perception, computer vision, mobile robotics, computational neuroscience, data compression, document image analysis, digital libraries, and the physical basis of computation. He has published over 130 technical papers on these topics. His image compression technology, called DjVu, is used by numerous digital libraries and publishers to distribute scanned documents on-line, and his handwriting recognition technology is used to process a large percentage of bank checks in the US. His learning-based image understanding techniques are used in many industrial applications including video surveillance, document understanding, and human-computer interaction. He has been general chair of the annual Learning workshop since 1997, and program chair of the 2006 CVPR conference.