The specification of a Markov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function.

Most models of utility elicitation in decision support and interactive optimization assume a predefined set of "catalog" features over which user preferences are expressed. However, users may differ in the features over which they are most comfortable expressing their preferences. In this work we consider the problem of feature elicitation: a user's utility function is expressed using features whose definitions (in terms of "catalog" features) are unknown. We cast this as a problem of concept learning, but whose goal is to identify only enough about the concept to enable a good decision to be recommended. We describe computational procedures for identifying optimal alternatives w.r.t. minimax regret in the presence of concept uncertainty; and describe several heuristic query strategies that focus on reduction of relevant concept uncertainty.

We present a model for buying agents in e-marketplaces to interpret evaluations of sellers provided by other buying agents, known as advisors. The interpretation of seller evaluations is complicated by the inherent subjectivity of each advisor, the possibility that advisors may deliberately provide misleading evaluations to deceive competitors and the dynamic nature of seller and advisor behaviors that may naturally change seller evaluations over time. Using a Bayesian approach, we demonstrate how to cope with subjectivity, deception and change in a principled way. More specifically, by modeling seller properties and advisor evaluation functions as dynamic random variables, buyers can progressively learn a probabilistic model that naturally and ``correctly'' calibrate the interpretation of seller evaluations without having to resort to heuristics to explicitely detect and filter/discount unreliable seller evaluations. Our model, called BLADE, is shown empirically to achieve lower mean error in the estimation of seller properties when compared to other models for reasoning about advisor ratings of sellers in electronic maketplaces.