Evaluation in HCI

1. Evaluation in HCI Angela Kessell Oct. 13, 2005

2. Evaluation • Heuristic Evaluation • Measuring API Usability • Methodology Matters: Doing Research in the Behavioral and Social Sciences

3. Evaluation • Heuristic Evaluation • “Discount usability engineering method” • Measuring API Usability • Methodology Matters: Doing Research in the Behavioral and Social Sciences

4. Evaluation • Heuristic Evaluation • “Discount usability engineering method” • Measuring API Usability • Usability applied to APIs • Methodology Matters: Doing Research in the Behavioral and Social Sciences

5. Evaluation • Heuristic Evaluation • “Discount usability engineering method” • Measuring API Usability • Usability applied to APIs • Methodology Matters: Doing Research in the Behavioral and Social Sciences • Designing, carrying out, and evaluating human subjects studies

6. Heuristic EvaluationJakob Nielsen

7. Heuristic EvaluationJakob Nielsen Most usability engineering methods will contribute substantially to the usability of an interface …

8. Heuristic EvaluationJakob Nielsen Most usability engineering methods will contribute substantially to the usability of an interface … …if they are actually used.

9. Heuristic Evaluation • What is it?

10. Heuristic Evaluation • What is it? A discount usability engineering method

11. Heuristic Evaluation • What is it? A discount usability engineering method - Easy (can be taught in ½ day seminar) - Fast (about a day for most evaluations) - Cheap (e.g. $(4,000 + 600i))

12. Heuristic Evaluation • How does it work?

13. Heuristic Evaluation • How does it work? • Evaluators use a checklist of basic usability heuristics • Evaluators go through an interface twice • 1st pass get a feel for the flow and general scope • 2nd pass refer to checklist of usability heuristics and focus on individual elements • The findings of evaluators are combined and assessed

14. Heuristic EvaluationUsability Heuristics (original, unrevised list) • Simple and natural dialogue • Speak the users’ language • Minimize the users’ memory load • Consistency • Feedback • Clearly marked exits • Shortcuts • Precise and constructive error messages • Prevent errors • Help and documentation

15. Heuristic EvaluationUsability Heuristics (original, unrevised list) • Simple and natural dialogue • Speak the users’ language • Minimize the users’ memory load • Consistency • Feedback • Clearly marked exits • Shortcuts • Precise and constructive error messages • Prevent errors • Help and documentation COMMENTS?

16. Heuristic Evaluation • One expert won’t due • Need 3 - 5 evaluators • Exact number needed depends on cost-benefit analysis

17. Heuristic Evaluation • Who are these evaluators? • Typically not domain experts / real users • No official “usability specialist” certification exists • Optimal performance requires double experts

18. Heuristic Evaluation • Debriefing session • Conducted in brain-storming mode • Evaluators rate the severity of all problems identified • Use a 0 – 4, absolute scale • 0 I don’t agree that this is a prob at all • 1 Cosmetic prob only • 2 Minor prob – low priority • 3 Major prob – high priority • 4 Usability catastrophe – imperative to fix

19. Heuristic Evaluation • Debriefing session • Conducted in brain-storming mode • Evaluators rate the severity of all problems identified • Use a 0 – 4, absolute scale • 0 I don’t agree that this is a prob at all • 1 Cosmetic prob only • 2 Minor prob – low priority • 3 Major prob – high priority • 4 Usability catastrophe – imperative to fix COMMENTS?

20. Heuristic Evaluation • How does H.E. differ from User Testing?

21. Heuristic Evaluation • How does H.E. differ from User Testing? • Evaluators have checklists • Evaluators are not the target users • Evaluators decide on their own how they want to proceed • Observer can answer evaluators’ questions about the domain or give hints for using the interface • Evaluators say what they didn’t like and why; observer doesn’t interpret evaluators’ actions

22. Heuristic Evaluation • What are the shortcomings of H.E.?

23. Heuristic Evaluation • What are the shortcomings of H.E.? • Identifies usability problems without indicating how they are to be fixed. • “Ideas for appropriate redesigns have to appear magically in the heads of designers on the basis of their sheer creative powers.” • Cannot expect it to address all usability issues when evaluators are not domain experts / actual users

24. Measuring API UsabilitySteven Clarke

25. Measuring API UsabilitySteven Clarke • User-centered design approach • Understanding both your users and the way they work • Scenario-based design approach • Ensures API reflects the tasks that users want to perform • Use Cognitive Dimensions Framework

26. Cognitive dimensions framework describes: What users expect What the API actually provides Cognitive dimensions framework provides: A common vocabulary for developers Draws attention to important aspects The Dimensions: Abstraction level Learning style Working framework Work-step unit Progressive evaluation Premature commitment Penetrability API elaboration API viscosity Consistency Role expressiveness Domain correspondence Measuring API Usability

27. Cognitive dimensions framework describes: What users expect What the API actually provides Cognitive dimensions framework provides: A common vocabulary for developers Draws attention to important aspects The Dimensions: Abstraction level Learning style Working framework Work-step unit Progressive evaluation Premature commitment Penetrability API elaboration API viscosity Consistency Role expressiveness Domain correspondence Measuring API Usability COMMENTS?

28. Measuring API Usability • Use Personas: • Profiles describing the stereotypical behavior of three main developer groups (Opportunistic, Pragmatic, Systematic) • Compare API evaluation with the profile requirements

29. Measuring API Usability • Use Personas: • Profiles describing the stereotypical behavior of three main developer groups (Opportunistic, Pragmatic, Systematic) • Compare API evaluation with the profile requirements COMMENTS?

30. Methodology Matters: Doing Research in the Behavioral and Social SciencesJoseph McGrath

31. Methodology Matters: Doing Research in the Behavioral and Social Sciences Key points: • All methods are valuable, but all have limitations/weaknesses • Offset the weaknesses by using multiple methods

32. Methodology Matters: Doing Research in the Behavioral and Social Sciences In conducting research, try to maximize: • Generalizability • Precision • Realism

33. Methodology Matters: Doing Research in the Behavioral and Social Sciences In conducting research, try to maximize: • Generalizability • Precision • Realism -You cannot maximize all three simultaneously.

34. Methodology Matters: Doing Research in the Behavioral and Social Sciences From http://pages.cpsc.ucalgary.ca/%7Esaul/hci_educ_papers/bgbg95/mcgrath-summary.pdf

35. So… • 1st 2 papers focus on computer programs / GUIs • 3rd paper presents the whole gamut of methodologies available to study any human behavior

36. But… what’s missing?

37. But… • Where are the statistics? • Are there objective “right” answers in HCI? • How do we evaluate other kinds of interfaces? • Other thoughts on what’s missing?

38. How do we evaluate… • “Embodied virtuality” / ubiquitous computing “interfaces” • (Aura video… http://www.cs.cmu.edu/~aura/) • Try to pick out one capability presented, and think about how you might evaluate it

39. Evaluating Aura • Do we evaluate the whole system at once? Or bit by bit? • Where / What is the interface? • Is anyone not a target user?

40. From http://www.usability.uk.com/images/cartoons/cart5.htm