Decision Making under Uncertainty: Readings and Slides

Decision Making Under Uncertainty
readings and lecture slides: fall/winter 2006-07

Class Schedule (with Readings and Lecture Slides)

Readings for specific classes will be usually be posted here a few weeks before class. Occasionally some supplemental readings will be added after the class. Some readings will not be available online: as appropriate, I will make hardcopy available for copying. AIMA refers to Russell and Norvig's Artificial Intelligence: A Modern Approach (see General Readings below).

At the bottom of this page are some some general background readings that you can use for specific aspects of the course.

Sept.25, 2006: Introduction; Basic Probabilistic Inference

Readings: no readings (but see next week's readings)

Notes: Lecture 1 Slides
Oct.23, 2006: Probabilistic Inference and Bayesian Networks; we may begin introduction to decision theory as well (see next class).

Readings:
AIMA: Chapter 14 (short, skip if familiar)
AIMA: Chapter 15: focus on 15.1, 15.2 and skim 15.3,15.4
Notes by N. Friedman and D. Koller on the variable elimination for inference in Bayes nets. Note: an updated version of these notes (a draft chapter of a textbook) is available. I can distribute hardcopy (but not an electronic copy).
(optional) R. Dechter, Bucket Elimination: A Unifying Framework for Probabilistic Inference. UAI-96. Read up to 3.3 (5 pages). Available online

Notes: Lecture 2 Slides (same as Lecture 1)
October 30, 2006: Intro to Decision Theory

Readings:
AIMA: Chapter 16 (skim, skip 16.5 and 16.7)
Follow up reading: S. French, 1986, "Decision Theory", Chap.5.6 (and Ch.4). (See Background References list below.)
Follow up reading: F. Bacchus and A. Grove, Graphical models for preference and utility, Uncertainty in Artificial Intelligence (UAI-95), pp.3-10, 1995. Available Online

Notes: No electronic notes/slides.
November 20, 2006: Multiattribute Utility Theory

Readings:
AIMA: Chapter 16
Follow up reading: S. French, 1986, "Decision Theory", Chap.5.6 (an d Ch.4). (See Background References list below.)
Follow up reading: F. Bacchus and A. Grove, Graphical models for preference and utility, Uncertainty in Artificial Intelligence (UAI-95), pp.3-10, 1995. Available Online

Notes: No electronic notes/slides.
December 4, 2006: Preference Elicitation

Readings:
C. Boutilier, R. Patrascu, P. Poupart, and D. Schuurmans (2006). Constraint-based Optimization and Utility Elicitation using the Minimax Decision Criterion, Artificial Intelligence 170(8--9), pp.686-713. Available Online
Chajewska, U., L. Getoor, J. Norman and Y. Shahar (1998). Utility Elicitation as a Classification Problem. In Uncertainty in Artificial Intelligence 14 (UAI '98), pp. 79-88. Available Online

Notes: TBA.
January 15, 2006: Sequential Decision Making; Markov Decision Processes: Introduction

Readings:
C. Boutilier, T. Dean and S Hanks. Decision Theoretic Planning: Structural Assumptions and Computational Leverage. JAIR 11:1--94 (1999). Read Sections 1 through 3. Available Online
Follow up reading:S. French, Decision Trees and Multistage Problems (Ch.7 of Decision Theory, S. French, Haltsed Press, 1986).

Notes: MDPs and Factored Representations, slides for this week (and next class) will be posted.
February 12, 2007: Wrap Up MDPs; Structured Representations of MDPs

Readings:
C. Boutilier, T. Dean and S Hanks. Decision Theoretic Planning: Structural Assumptions and Computational Leverage. JAIR 11:1--94 (1999). Read Sections 4-5. Available Online
Optional Reading if you want some more detail: C. Boutilier, R. Dearden and M. Goldszmidt. Stochastic Dynamic Programming with Factored Representations, Artificial Intelligence 121(1):49-107, 2000. Available Online.
Optional Reading if you want some more detail): R. Dearden and C. Boutilier. Abstraction and Approximate Decision Theoretic Planning, Artificial Intelligence 89(1):219-283 (1997). Available On line.
Optional Reading if you want some more detail: SPUDD: Stochastic Planning using Decision Diagrams. In Proc. 15th Conference on Uncertainty in Artificial Intelligence (UAI '99), pp.279-288, Stockholm, 2001. Available On line.

Notes: MDPs and Factored Representations, slides for this week (and previous class) will be posted.
February 26: MDPs: Function Approximation

Readings:
Carlos Guestrin, Daphne Koller and Ronald Parr. Max-norm Projections for Factored MDPs. IJCAI-01, pp. 673 - 680, Seattle, 2001. Available Online.
(Optional/Supplemental) D. Schuurmans and R. Patrascu. Direct value-approximation for factored MDPs. In Advances in Neural Information Processing 14 (NIPS*2001). Available Online.
N. Meuleau, M. Hauskrecht, K.E. Kim, L. Peshkin, L.P. Kaelbling, T. Dean and C. Boutilier. Solving Very Large Weakly Coupled Markov Decision Processes. AAAI-98, pp.165--172, Madison, WI, 1998. Available Online.
(Optional/Supplemental) M. Kearns, Y. Mansour, A. Ng. A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes, IJCAI-99, (1999). Available Online.
Notes: Weakly Coupled MDPs, and Linear Approximations, slides for this week will be posted.
March 26: Partially-Observable Markov Decision Processes

Readings:
Leslie Pack Kaelbling, Michael L. Littman and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence, 101: 1-2, pages 99-134, 1998. Available Online. If you have trouble, try to grab an earlier, but quite similar version.
Notes: TBA.
April 9: Partially-Observable Markov Decision Processes: Approximate Monitoring and Solution

Readings:
Xavier Boyen and Daphne Koller. Tractable inference for complex stochastic processes. UAI-98. Available Online. (Don't get too hung up on the details in Sections 3 and 4; just try to extract the main intuitions and we'll discuss things.)
Matthijs T. J. Spaan and Nikos Vlassis. Perseus: Randomized Point-based Value Iteration for POMDPs. JAIR 24, pp.195-220 (2005). Available Online.

General Readings and Background

An useful background reference for the course (especially for those without significant AI background) is Artificial Intelligence: A Modern Approach, by Stuart Russell and Peter Norvig (Prentice-Hall, 1995). This is a comprehensive text with reasonably concise, but very readable and well-motivated, chapters on a number of topics we will discuss. I'll refer to sections from AIMA as background reading from time to time and make available several copies of the text.

We will also be using, for the last half of the course, a draft text by Yoav Shoham and Kevin Leyton-Brown, Multiagent Systems. This will be made available electronically in January.

Some further general background readings on various topics.

Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann, 1988. Probabilistic Reasoning and Bayes nets
T. Dean and M. Wellman. Planning and Control. Morgan-Kaufmann, 1991. Planning and decision theoretic planning
C. Boutilier, T. Dean and S Hanks. Decision Theoretic Planning: Structural Assumptions and Computational Leverage, Journal of AI Research 11:1--94 (1999) Survey of MDPs and the use of AI techniques to solve them
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994. General MDPs
Dimitri P. Bertsekas. Dynamic Programming: Deterministic and Stochastic Models. Prentice-Hall, 1987. General MDPs
Simon French. Decision Theory. Halsted Press, 1986. Good general decision theory
L. Savage. The Foundations of Statistics. Wiley, 1954. Foundations of decision theory
W. Lovejoy. A Survey of Algorithmic Methods for Partially Observed Markov Decision Processes, Annals of Operations Research 28:47--66 (1991). Survey of POMDP solution techniques
R. Sutton and A. Barto. Reinforcement Learning: An Introduction, MIT Press, 1998. Great introductory text for reinforcement learning. Very accessible, easy reading. Lacking a bit in depth. Entire book is online.
L. Kaelbling, M Littman, A. Moore. Reinforcement Learning: A Survey, Journal of Artificial Intelligence Research 4:237--285, 1996. Nice survey article.
D. Bertsekas and J. Tsitsiklis. Neuro-dynamic Programming, Athena, 1996. Another great text for RL. Less accessible than Sutton and Barto, but goes into much more technical depth.
R. Myerson. Game Theory: Analysis of Conflict. Harvard, 1991. Decent game theory text
D. Fudenberg and J. Tirole. Game Theory. MIT, 1991. Another decent game theory text
S. Rosenschein and G. Zlotkin. Rules of Encounter. MIT, 1994. A monograph on application of game theoretic methods to AI style problems