Yoshua Bengio and Yann LeCun
Learning Deep Architectures
This short tutorial on deep learning will review a variety of methods
for learning multi-level, hierarchical representations, emphasizing
their common traits. While deep architectures have theoretical
advantages in terms of expressive power and efficiency of
representation, they also provide a possible model for information
processing in the mammalian cortex, which seems to rely on
representations with multiple levels of abstractions. A number of
deep learning methods have been proposed since 2005, that have yielded
surprisingly good performance in several areas, particularly in vision
(object recognition), and natural language processing. They all learn
multiple levels of representation using some form of unsupervised
learning. Hypotheses to explain why these algorithms work well will be
discussed in the light of new experimental results. Many of these
algorithms can be cast in the framework of the energy-based view of
unsupervised learning, which generalizes graphical models used as
building blocks for deep architectures, such as the Restricted
Boltzmann Machines (RBM) and variations of regularized
auto-encoders. Old and new algorithms will be presented for training,
sampling, and estimating the partition function of RBMs and Deep
Belief Networks. Applications of deep architectures to computer vision
and natural language processing will be described. A number of open
problems and future research avenues will be discussed, with active
participation from the audience.
Brief Bio.
Yoshua Bengio (PhD’1991, McGill University) is professor
at the Department of Computer Science and
Operations Research, Universite de Montreal, and
Canada Research Chair in Statistical Learning Algorithms,
as well as NSERC-CGI Chair, and Fellow of
the Canadian Institute for Advanced Research. His
main ambition is to understand how learning can give
rise to intelligence. He has been an early proponent of
deep architectures and distributed representations as
tools to bypass the curse of dimensionality and learn
complex tasks. He contributed to many machine learning
areas: neural networks, recurrent neural networks,
probabilistic graphical models (especially for temporal
data), kernel machines, semi-supervised learning,
unsupervised learning and manifold learning, pattern
recognition, data-mining, natural language processing,
machine vision, time-series prediction.
Yann LeCun is Silver Professor of Computer Science at the Courant
Institute of Mathematical Sciences of New York University. He
received the Electrical Engineer Diploma from Ecole
Superieure d'Ingenieurs en Electrotechnique et
Electronique (ESIEE), Paris in 1983, and a PhD in Computer
Science from Universite Pierre et Marie Curie (Paris) in 1987.
After a postdoctoral fellowship at the University of Toronto, he
joined the Adaptive Systems Research Department at AT\&T Bell
Laboratories in Holmdel, NJ, in 1988. He was named head of the
Image Processing Research Department at AT\&T Labs-Research in 1996.
In 2002, he became a Fellow of the NEC Research Institute in
Princeton, before moving to NYU in 2003. Yann's research interests
include computational and biological models of learning and
perception, computer vision, mobile robotics, computational
neuroscience, data compression, document image analysis, digital
libraries, and the physical basis of computation. He has published
over 130 technical papers on these topics. His image compression
technology, called DjVu, is used by numerous digital libraries and
publishers to distribute scanned documents on-line, and his
handwriting recognition technology is used to process a large
percentage of bank checks in the US. His learning-based image
understanding techniques are used in many industrial applications
including video surveillance, document understanding, and
human-computer interaction. He has been general chair of the annual
Learning workshop since 1997, and program chair of the 2006 CVPR
conference.