| 
| |
| (a) Adaptive typing assistance | (b) Adaptive highlighting assistance |
Software Customization Assistance (SCA)
This work began as an online solution to the general
software customization problem.
To design software such that it is flexible enough to allow the user to
tweak it after buying the product is called making the software
adaptable -- the approach that most HCI researchers take. To design the
software such that it can learn the user's preferences by observing what the
user does over time and adapt itself to them is called making the software
adaptive. Some current work combines these two in ad hoc ways, but
ideally, we want to view users on a continuous spectrum with adaptable
solutions on one end and adaptive solutions on the other, and have the system
figure out the optimal behaviour for a particular user.
My approach is to develop a general, decision-theoretic framework called
SCA, which stands for software customization assistance.
This framework consists of a Bayesian user model and a parametric reward
model, so that the system can take actions by maximizing (long-term) expected
utility. Specifying the reward model has been a hard task, because it requires
taking existing HCI phenonmena and formalizing it in our framework, as well as
eliciting subjective preferences for those interaction settings.
Learning various features in our user model is a difficult task as well,
because the personality psychology literature generally do not deal with
personality variables in a computing environment. One major aspect of this
work is to develop and conduct experiments with real people to learn the
models needed in the framework.
The other major aspect is to systematically evaluate each model's contribution
toward the overall utility/usability of the system.
This is on-going PhD work with Craig Boutilier. Interesting issues that arise for modeling the uncertainty of the user include preference elicitation, inverse reinforcement learning, and user modeling. It also has some flavour of mixed initiative to it because of the style of interaction between an adaptive system and its users. Currently, the system action strategy is myopic, and it is implemented in the context of providing adaptive help for a typing task (see Figure (a) below). Later, I intend to extend it as a POMDP (partially observable Markov decision processes), so that the system can act in the best interest of the user over time. The intended context is adaptive help during a highlighting task (see Figure (b) below, with a more detailed description in the APPA section).
|
|
| |
| (a) Adaptive typing assistance | (b) Adaptive highlighting assistance |
Papers and Presentations:
Adaptive PowerPoint Assistance (APPA)
This is an application testbed for the SCA framework
described above.
Description under construction
Click here to download the demo video.
Here's a demo video (duration 4:49). It shows the implemented suggestion toolbar for the highlighting task. The toolbar's parameters are LENGTH (the number of icons) and QUALITY (the quality of the icon suggestions defined as the objective number of steps that can save the user from manually executing the task). A pop-up window is used to set these parameters. The interface also allows you the change the vocabulary of fonts and colours, making the task more difficulty and thus, the user more NEEDY. Mouse and key events are logged in this application. The debug window icon (left of the "X") is used to view the event and error logs. The parameter window icon (right of the "N") is used to set the parameters of the suggestions.
In this video, the first file corresponds to the setting of not needy, and various settings for length and quality are shown. First, we see L10/Q4: it can be easy to pick out the correct icon in this case. Next we see L1/Q4: it is extremely easy to simply select the correct icon here because no search is involved. Next we see L1/Q2: selecting it only applies half of the correct features, so the user must complete the other half (hopefully correctly induced). Finally, we see L1/Q0: selecting it doesn't help, and still requires the user to click on at least as many features to fix the style as the user would have needed to do manually.
In the second file of the video, the setting is needy. We show some L/Q settings just to give a sense of what the suggestions are like and to contrast the difficulty of the highlighting task in this case.
In a standard gamble query, the user has to decide whether alternative A or B is better, or the user is indifferent between the two. Alternative A is a lottery of the form [p,o_best; 1-p,o_worst], while Alternative B is a definitive outcome, o_i. In the parameter window that sets LENGTH/QUALITY, there is also a box for setting the PROBABILITY in Alternative A. In that case, since there are always 10 slides per file, p of the 10 slides will have o_best kind of suggestions, while 1-p of the slides will have o_worst kind of suggestions.
Software Design and Customizability
To tailor software to user needs and preferences before delivering the
software to the user is called software customization. The
state-of-the-art for customization is manually tweaking the software based on
interview feedback from user or user groups. Some of the initial work in
developing a methodology for customization is done with John Mylopoulos.
This project began as part of a larger cross-Canadian University project to
develop cognitive orthodics to help Alzheimer's patients,
which later evolved as a collaboration with
the University of Oregon's
Think-and-Link project for
survivors of Traumatic Brain Injury,
led by Steve Fickas.
The need to tailor software to individuals is especially clear when we
consider the wide spectrum of skills that humans
are capable of. Modeling these skills under a psychological framework is done
jointly with Eileen
Wood and Teena
Willoughby. We are interested in applying this skills hierarchy to
categorize and develop specialized children games
(cf. edutainment for more details).
Papers and Presentations:
Behavioural Decision Making
My interest in this area started when I took a course on Thinking and Deciding
from
Derek
Koehler during my undergrad.
What interest me most is how people make decisions, and what affects their
decision making process.
Since there is always a discrepancy between a normative theory and a
descriptive theory, I wanted to know how computer systems can be designed to
accomodate descriptive theories. For example, a normative theory in decision
making is utility theory. But nobody thinks in utilities or computes
expected utilities in their daily lives. This is because people do not have an
actual number attached to events and outcomes in their daily activities.
But utility theory makes sense. Since computers can have numbers
associated with each event and outcome, utility theory is the right thing to
do. The trick is, how to we get "meaningful" numbers? The answer comes from
the area known as preference elicitation in Computer Science and
econometrics in Psychology and Economics. Still, the answers that the
literature has to offer are limited in applicability, because the "queries"
that computers are currently designed to use generally don't make any sense to
people. Most of this work has not considered psychological effects that affect
people's decision making process, the existing literature is mostly concerned
with computational feasibility. It's time we start bringing the two together.
Most importantly, if the computational approaches are designed to deal with
human users (rather than simulations), they need to start modeling (at least)
the following well-known effects:
anchoring, framing, endowment, calibratin, neglect, and intertemporal.
Until then, we still have a long ways to go.
Papers and Presentations:
Online Speech Processing
In traditional syntax, a parse tree is built given a string of words. However,
empirical evidence shows that people have partial parses for incomplete
sentences. Furthermore, evidence points to Japanese/German (S-O-V) and English
(S-V-O) having same parse structures. Therefore, a more accurate theory needs
to account for the incrementality of online processing -- building parse trees
as incoming words are heard. Much of this work was explored in Dave
Schneider's thesis. Under
the direction of Colin Phillips, I
further developed the computational model, SPARSE, to work with additional
lexical features and syntactic reanalysis.
A more basic question is, given a stream of sounds as input, how does this sequence transform into an understandable utterance? First, we need to map the clearer sounds that we are certain of and then guess the remaining sounds. But what exactly are the mapping and guessing functions? In particular, relevant questions to bare in mind are:
Papers and Presentations:
Information Extraction and Retrieval
I started working on information retrieval (IR) and information extraction
(IE) through courses I took with Robin Cohen. There, I was
interested in the (non-)use of linguistic theories and language processing
techniques in domains where robustness, scalability, and speed are crucial.
Later, these techniques were applied in the knowledge management domain, under
the direction of Eric Yu. An
interesting application is the automatic extraction and presentation of
concepts in patents and scientific articles for researchers and business
analysts. This was the focus of my Master's. Future work
outlines the potential of the system to incorporate automatic
reasoning/verification of user queries and to expand to managing document
collections. Another demanding area of application is the summarization of
medical documents, which is an interest of John Mylopoulos.
Papers and Presentations:
Lexical Things
For as long as I can remember, I have always had a passion for understanding
everything there is to know about words. How are they represented? What
do they mean? What do the entries look like in our heads? Should computer
applications use the same representation? The main application I was
interested in was machine translation, which evolved from a reading course I
took with Nick Cercone. In that
context, I was interested in questions such as: How do multilinguals perform
search and retrieval through their lexicons? Are entries of the same concept in
different languages linked together? My undergraduate years were spent
researching these questions, including my Honours thesis.
Papers and Presentations:
Mixed Initiative and Discourse Structure
In a dialogue of multiple participants, there are speakers and hearers who
interactively exchange utterances. How long does an individual piece of
dialogue go on for and how are interruptions modeled? How many turns do
speakers take before the topic shifts? Are there some speakers who always
initiate the dialogue? Do passive listeners get bored, lose attention, or get
anything worthwhile out of the dialogue? Put into the context of computer and
user interaction, would a mixed initiative computational model apply? Does the
model extend across different domains, such as a trains system, a planning
system, and an intelligent tutoring system? This is also a line of interest
that started through a graduate seminar course with
Robin Cohen at the
University of Waterloo.
Papers:
Edutainment
Edutainment is about education as a form of entertainment. I first
started working in this area under the direction of Maria Klawe, in her
lab called Electronic Games for
Education in Mathematics and Sciences. There, we surveyed how girls and
boys interact with computer games and collaboratively designed pen-and-paper
as well as computer prototypes with them. A game I developed was the Coins puzzle, where the user is presented with a
set of identical gold coins and the task is to determine the fake one. In the
same context, I worked on some features of a conversational agent named Julie, which was based on activation networks.
Our team effort on the Julie personality resulted in third place in the 1997
Loebner contest. These components were all part of the game, Phoenix Quest.
In another summer, I developed two language games, Bingo and Canoeing, for learners of Sliammon -- a native Indian language spoken in northern BC. In Bingo, the user has to listen to numbers with different endings (nominal markers) and put a chip on the box with the same number if the number has the same ending as the class of objects represented in that column. In Canoeing, the user has to move a canoe from the starting location to the destination. An advancement in the correct direction is granted if the user can differentiate a pair of similar sounding words (a minimal pair). All the words were displayed using the First Nations New font, which was especially developed for the family of native Indian languages. This work was done under the direction of Pat Shaw.
Since 2002, I have been affiliated with the Canadian Language and Literacy Research Network. Although my background overlaps greatly with the phonology and developmental linguistics work of the members of the network, my involvement there was to develop software literacy under the guidance of Eileen Wood and Teena Willoughby. The first part of this work analyzes current edutainment software and proposes an underlying canonical navigation structure for the various games. The second component of this work (in progress) analyzes individual game activities for required and exercised skills -- motor and cognitive (not emotional or social). The third part is to combine the results from the first two phases and incorporate it into the design structure of a software that tailors toward individual users.
More recently, I undertook an RA in pediatrics with Caroline Fusco and Tom Chau to document the vast array of works about children and multimedia. This work is part of a bigger vision of creating multimedia technology to help patients express their emotions and explore visual creativity.
Patents and Copyrights:
Papers and Presentations:
Miscellany
Topics include conceptual modeling, human-computer interaction, compilers,
and cognitive science.
