Deep Learning

KDD 2014 Tutorial

9:00am-12:30am, Sunday August 24, 2014

Schedule

9:00 - 10:30am Deep Learning I pdf
11:00- 12:30am Deep Learning II pdf
Video of the tutorial is available here


Overview

Building intelligent systems that are capable of extracting high-level representations from high-dimensional sensory data lies at the core of solving many AI related tasks, including visual object or pattern recognition, speech perception, and language understanding. Theoretical and biological arguments strongly suggest that building such systems requires deep architectures that involve many layers of nonlinear processing.

Many existing learning algorithms use shallow architectures, including neural networks with only one hidden layer, support vector machines, kernel logistic regression, and many others. The internal representations learned by such systems are necessarily simple and are incapable of extracting some types of complex structure from high-dimensional input.

In the past few years, researchers across many different communities, from applied statistics to engineering, computer science and neuroscience, have proposed several deep (hierarchical) models that are capable of extracting useful, high-level structured representations.An important property ofthese models is that they can extract complex statistical dependencies from high-dimensional sensory input and efficiently learn high-level representations by re-using and combining intermediate concepts, allowing these models to generalize well across a wide variety of tasks. The learned high-level representationshave been shown to give state-of-the-art results in many challenging learning problems,where data patterns often exhibit a high degree of variations, and have been successfully applied in a wide variety of application domains, including visual object recognition, information retrieval, natural language processing, and speech perception. A few notable examples of such models include Deep Belief Networks, Deep Boltzmann Machines, Deep Autoencoders, and sparse coding-based methods.

The goal of the tutorial is to introduce the recent and exciting developments of various deep learning methods to the KDD community. The core focus will be placed on algorithms that can learn multi-layer hierarchies of representations.


The tutorial will be split into two parts. The first part will provide a gentle introduction into graphical models and deep learning models. Topics will include:
  • Unsupervised learning methods, including autoencoders, de-noising autoencoders, restricted Boltzmann machines, stacked restricted Boltzmann machines, sparse coding, and methods for learning over-complete representations.
  • Supervised methods for deep models, including deep convolutional models and their applications to image and video analysis.

The second part of the tutorial will introduce more advanced models, including Deep Boltzmann Machines and Deep Belief Networks. We will also address mathematical issues, focusing on efficient large-scale optimization methods for inference and learning, as well as training density models with intractable partition functions.

The final part of the tutorial will also highlight the current performances obtained by deep approaches on various applications in image/video analysis, language modeling, and various other domains.