NIPS 2014 Workshop: Perturbations, Optimization, and Statistics

Confirmed Speakers

• Andreea Gane (MIT)

• Manfred Warmuth (UCSC)

• Ian Goodfellow (Google)

• Jacob Abernethy (Michigan)

• Yury Makarychev (TTI-C)

Description

In nearly all machine learning tasks, decisions must be made given current knowledge (e.g., choose which label to predict). Perhaps surprisingly, always making the best decision is not always the best strategy, particularly while learning. Recently, there is an emerging body of work on learning under different rules that apply perturbations to the decision procedure. These works provide simple and efficient learning rules with improved theoretical guarantees. This workshop will bring together the growing community of researchers interested in different aspects of this area, and it will broaden our understanding of why and how perturbation methods can be useful.

In the last couple of years, at the highly successful NIPS workshops on Perturbations, Optimization, and Statistics, we looked at how injecting perturbations (whether it be random or adversarial “noise”) into learning and inference procedures can be beneficial. The focus was on two angles: first, on how stochastic perturbations can be used to construct new types of probability models for structured data; and second, how deterministic perturbations affect the regularization and the generalization properties of learning algorithms.

The goal of this workshop is to expand the scope of previous workshops and also explore different ways to apply perturbations within optimization and statistics to enhance and improve machine learning approaches. This year, we would like to look at exciting new developments related to the above core themes.

More generally, we shall specifically be interested in understanding the following issues:

• Modeling: which models lend efficient learning by perturbations?

• Regularization: whether randomness can be replaced by other mathematical object while keeping the computational and statistical guarantees?

• Robust optimization: how stochastic and adversarial perturbations affect the learning outcome?

• Dropout: How stochastic dropout regularizes online learning tasks?

• Sampling: new ways to utilize perturbations for sampling from probabilistic models.

Target Audience: The workshop should appeal to NIPS attendees interested in both theoretical aspects such as Bayesian modeling, Monte Carlo sampling, optimization, inference, and learning, as well as practical applications in computer vision and language modeling.

Call for Papers

In addition to a program of invited presentations, we solicit contribution of short papers that explore perturbation-based methods in the context of topics such as: statistical modeling, sampling, inference, estimation, theory, robust optimization, robust learning. We are interested in both theoretical and application-oriented works. We also welcome papers that explore connections between alternative ways of using perturbations.

Contributed papers should adhere to the NIPS format and are encouraged to be up to four pages long (without counting the list of references). Papers submitted for review do not need to be anonymized. There will be no official proceedings. Thus, apart from papers reporting novel unpublished work, we also welcome submissions describing work in progress or summarizing a longer paper under review for a journal or conference (this should be clearly stated though). Accepted papers will be presented as posters; some may also be selected for spotlight talks.

Please submit papers in PDF format by email to posNIPS2014@gmail.com. The submission deadline is Nov 9, 2014 (please email us if you need to be notified of a decision earlier for the sake of travel arrangements). At least one of the authors must be attending the workshop to present the work.

Organizers

• Tamir Hazan, University of Haifa

• George Papandreou, TTI-Chicago

• Danny Tarlow, Microsoft Research

Previous Workshops

This is the third year of this workshop. The 2012 POS Workshop website is here. The 2013 POS Workshop website is here.

References

Machine learning

• Efficient algorithms for online decision problems (J. of Comp. and System Sci., 2005)
  A. Kalai, S. Vempala
• Extracting and composing robust features with denoising autoencoders (ICML, 2008)
  P. Vincent, H. Larochelle, Y. Bengio, P. Manzagol
• Herding dynamical weights to learn (ICML, 2009)
  M. Welling
• Gaussian sampling by local perturbations (NIPS, 2010)
  G. Papandreou, A. Yuille
• Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models (ICCV, 2011)
  G. Papandreou, A. Yuille
• Robust Max-Product Belief Propagation (arXiv, 2011)
  M. Ibrahimi, A. Javanmard, Y. Kanoria, A. Montanari
• Robust Optimization and Machine Learning (book chapter, 2011)
  C. Caramanis and S. Mannor and H. Xu
• Randomized Optimum Models for Structured Prediction (AISTATS, 2012)
  D. Tarlow, R. Adams, R. Zemel
• On the Partition Function and Random Maximum A-Posteriori Perturbations (ICML, 2012)
  T. Hazan, T. Jaakkola
• A Simple Geometric Interpretation of SVM using Stochastic Adversaries (AISTATS, 2012)
  R. Livni, K. Crammer, A. Globerson

Extreme value statistics

• Extremes and Related Properties of Random Sequences and Processes (Springer, 1983)
  M. Leadbetter, G. Lindgren, H. Rootzen
• Extreme value distributions: theory and applications (Imperial College Press, 2000)
  S. Kotz, S. Nadarajah

Discrete choice in psychology and economics

• Individual choice behavior (Wiley, 1959)
  D. Luce
• Conditional logit analysis of qualitative choice behavior (Book chapter, 1974)
  D. McFadden

Mathematics and statistical physics

• Information, Physics, and Computation (Oxford Univ. Press, 2009)
  M. Mezard and A. Montanari
• The supremum of some canonical processes (Am. J. of Math, 1994)
  M. Talagrand
• Random weighting, asymptotic counting, and inverse isoperimetry (Isr. J. Math, 2007)
  A Barvinok, A. Samorodnitsky
• Cover times, blanket times, and majorizing measures (arXiv, 2010)
  J. Ding, J.R. Lee, Y. Peres
• Smoothed Analysis: An Attempt to Explain the Behavior of Algorithms in Practice (CACM, 2009)
  D.A. Spielman, S. Teng
• Online Vertex-Weighted Bipartite Matching and Single-bid Budgeted Allocations (arXiv-10)
  G. Aggarwal, G. Goel, C. Karande, A. Mehta

Optimization

• Fast approximate energy minimization via graph cuts (PAMI, 2001)
  Y. Boykov, O. Veksler, R. Zabih
• Dynamic programming and graph algorithms in computer vision (PAMI, 2011)
  P.F. Felzenszwalb, R. Zabih
• Introduction to dual decomposition for inference (MIT Press, 2011)
  D. Sontag, A. Globerson, T. Jaakkola
• Optimization for Machine Learning (MIT Press, 2011)
  S. Sra, S. Nowozin, S. Wright., Editors, .
• Markov Random Fields for Vision and Image Processing (MIT Press, 2011)
  A. Blake, P. Kohli, and C. Rother (eds.)
• Stochastic Programming (Kluwer, 1995)
  A. Prekopa
• Robust Optimization (Princeton Univ. Press, 2009)
  A. Bental, L. El Ghaoui, A. Nemirovski