1 / 20

Evaluation

Evaluation. Chapter 9. Evaluation. A very significant aspect of HCI design that separates it from SE To test usability and usefulness Can be done in the lab and/or in the field Evaluate the design (early) and the implementation (later)

maile-dixon
Download Presentation

Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Evaluation Chapter 9

  2. Evaluation • A very significant aspect of HCI design that separates it from SE • To test usability and usefulness • Can be done in the lab and/or in the field • Evaluate the design (early) and the implementation (later) • Evaluate the implementation during design (formative) and after it has been deployed and used by customers (summative)

  3. Goals • Assess the functionality and usefulness of the interactive system: • Does it match requirements specifications? • Does it match the expectations of designers and users? • Does it meet the set out performance goals? • Assess the effect of the interface on the user: usability • Identify specific problems with the system and its interface, and develop remedies.

  4. Styles • Laboratory Studies • Advantages • specialized equipment can be used • environment can be controlled • Disadvantages • Cost • Unnatural/intimidating • Difficult to observe users in their “natural state” • Use when • collecting data in actual environment is impractical • if environment needs to be controlled • if experimental psychology techniques are to be used

  5. Styles • Field Studies • Advantages • Observe users in their “natural” setting • Can observe effects of context on system use • For longitudinal studies that require several days/weeks/months • Disadvantages • Cost • Can’t control environment: noise, interruptions etc. • Use when • A longitudinal study is required • Data on actual use conditions is desired • Strict control of the experimental condition unimportant

  6. Techniques • Cognitive Walkthrough • Heuristic Evaluation • Review-based Evaluation • Model-based Evaluation • Usability Evaluation • Observational Methods • Query Techniques • Experimental Evaluation

  7. Cognitive Walkthrough • Like code inspection/walkthrough in SE • Evaluates the design through a prototype - how well does it support the user in learning how to do the task - through exploration by experts • Usually performed by an expert who “walks through” the design to identify potential problems

  8. Cognitive Walkthrough • Starts with the expert being provided with: • A prototype of the system • Task, action and user descriptions developed during the design • For each task identified in task analysis, CW considers the following: • What impact will interaction have on the user? • What cognitive processes are required? • What learning/interaction problems may occur?

  9. Cognitive Walkthrough • For each action the user needs to carry out to accomplish a goal, the expert asks: • Can the user recognize the correct action that is required? • Is the action (i.e. how to carry it out) visible at the interface? • Is there a difference between its intended effect (what the user wants to happen) and its actual effect (what the system does)? • Is the user able to carry out the action successfully? • Can the user then successfully interpret the feedback provided by the system? • A negative answer indicates a potential usability problem • Understand the example in 11.4.1

  10. Heuristic Evaluation • A set of usability criteria (called heuristics) are identified • see the list on p. 413 • E.g. • System behaves in a predictable way in response to all user actions • System behaves in a consistent way • System provides feedback for all correct and incorrect actions • Then the design and/or the prototype is examined by experts to see if these are violated • This is called a “usability inspection technique”

  11. Heuristic Evaluation • Select the heuristics from those proposed in the literature by experts like Jakob Nielsen • Develop system and task specific questions to verify if the design/prototype satisfies each selected heuristic • Have multiple experts independently evaluate the system/prototype using these questions

  12. Other Evaluation Techniques • Review-based Evaluation • Review the technical HCI literature to see if similar designs/systems have been evaluated • Not commonly practiced • Model-based evaluation • An analytical approach in which models developed during the design process, such as ATN, GOMS, TDH etc, are analyzed to discover potential problems • Usability Specification & Evaluation • Selecting, setting target levels, and measuring specific usability attributes: usability specification table • Already covered; Read Chapter 8 of Reference if you haven’t yet done so!

  13. Observational Methods • A class of techniques called “protocol analyses” • Experimenter note-taking • cheap but limited • User notebooks • subjective, coarse level data • but useful user insights • good for longitudinal studies • beepers/PDAs used for reminding • Audio-taping • may miss actions, gestures etc. • transcription difficult

  14. Protocol Analyses • Video-taping • more complete record • but special equipment needed • obtrusive • transcription difficult • Computer-logging • automatic and unobtrusive • but voluminous data • Some combination of these is typically used

  15. Concurrent Think-aloud Protocols • Method • User observed while doing the task • User asked to talk aloud what he/she is doing, why, and what he/she is thinking/expecting to happen, etc. • Advantages • Simple technique • Can provide insights into user’s cognitive processes • Can reveal causes of errors • Disadvantages • Highly subjective • Voluminous raw data • Talking may alter performance

  16. Cooperative Protocols • Method • Variation of think-aloud in which the subject cooperates with the experimenter in asking and answering questions • Advantages • Advantages of think-aloud • Less constrained than think-aloud • User is encouraged to criticize the system and provide clarifications • Disadvantages • Disadvantages of think-aloud

  17. Retrospective Protocols • Also called Post-task Walkthrough • Method • User reflects on what happened after doing the task. • User asked questions to fill in details. • Sometimes combined with a during-task protocol collection • Advantages • Experimenter can focus on relevant incidents • Task interruption due to talking is avoided • Disadvantages • Memory limitations • Post-hoc interpretation of what happened is likely to be subjective

  18. Query Techniques • Advantages: informal, cheap, simple • Main disadvantage: subjective • Techniques • Structured Interviews • Experimenter questions each user after working with the system using prepared questions. • Written/Oral • Advantages: • Different questions for different users and tasks • Issues can be explored fully • Provides significant user input • Disadvantages • Time consuming

  19. Query Techniques • Questionnaires/Surveys • Fixed, typically multiple choice, written questionnaire given to users to fill out. • Careful design of questions and data analysis methods needed. • Advantages: • Quick, useful for large numbers of users • Can quantify data and statistical analyses possible • Provides significant user input • Disadvantages • Less flexible than interviews • Less deeply probing

  20. Query Techniques • Questionnaires/Surveys • Careful design of questions and data analysis methods needed. • Question Styles: • General: to characterize the subject • Open-ended: to elicit opinions/suggestions • Scalar: Likert Scale • Multiple-choice • Ranked • Examples • See worked exercise p. 433 of text • See p. 228 of reference

More Related