Recording Results

At this point, you have completed the question-answering task and should have a sense of how good the system outputs are as summaries. Now, we ask that you evaluate the output with respect to each of the principles. In particular, comments you have about the system can be typed up in an email and sent to the experimentor (bowen). For each principle, please also assign a number of how well you think the system output has conformed with it. Use a scale of 1 to 5, where 1 is the worst and 5 is the best. Also, use "not applicable" if you feel that a certain principle does not apply.

You may take a look at the original documents or go back to revisit the graphical interface if you wish.
Files Excerpts Interface
Playing structure and modules therefor Extracted Sections Graphical Layout
Method for remediation based on knowledge and/or functionality Extracted Sections Graphical Layout
Counting on Frank: Postmortem of an Edutainment Product Extracted Sections Graphical Layout
Children as Our Technology Design Partners Extracted Sections Graphical Layout

The Principles

  1. Conciseness: Components should not contain information that is irrelevant or redundant. Every extra unit of information competes with the relevant units of information and diminishes their relative visibility. All information should appear in a natural and logical order.
  2. Retention: Information retained in the system output should be representative of the key concepts and main points made in the original document. Are the major objectives of the paper captured in the summary? What about the major steps in the proposed solution and the results?
  3. Coherence: All information should be coherent within each component as well as the overall summary. Sentences need not be perfectly grammatical, but each point should make sense in its context.
  4. Consistency: Each component should be expressed clearly in words, phrases, and concepts consistent with those in the original document. Users should not have to wonder whether different words, situations, or actions mean the same thing.
  5. Informativeness: Information should be presented in a useful and easily acessible way. Some interface issues may be influential here as well. Irrelevant information should be omitted and words should not clutter the display of the information.
  6. Comprehensibility: Each point of information should be easy to understand. Users should not have to look up related information in another part of the system in order to understand a particular component.
  7. Fit For Audience: The information and the style of presentation fits for the intended audience. Audience may vary in their experience with domain knowledge. Access to different kinds of information should be easy and clear. The ability to show, modify, and hide information should be made obvious to the users.
  8. Fit For Purpose: The information and the style of presentation fits for the intended task or purpose. The particular task in this experiment is a question-answering task, similar to that of a reviewer for a conference. The system's overall objective is to make perceiving and learning of information more easy and fast for users. These two points should be taken into consdieration.

When You Are Done

Please email your feedback for both cases to the experimentor.

Go to the last page.