User Interface Evaluation

1. User Interface Evaluation Usability Testing Methods

2. Usability Testing Methods • Conduct experiments to find specific information about a design and/or product. • Basis comes from experimental psychology. • Uses statistical data methods • Quantitative and Qualitative

3. Usability Testing Methods • During usability testing, users work on specific tasks using the interface/product and evaluators use the results to evaluate and modify the interface/product. • Widely used in practice, but not appropriately used. • Often abused by developers that consider themselves to be usability experts. • Can be very expensive and time consuming.

4. Usability Testing Methods • Performance Measurement • Thinking-aloud Protocol • Question-asking Protocol • Coaching Method

5. Usability Testing Methods • Co-discovery Learning • Teaching Method • Retrospective Testing • Remote Testing

6. Performance Measurement • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 6.

7. Performance Measurement • Usability Issues Covered • Effectiveness: Yes • Efficiency: Yes • Satisfaction: No • Quantitative Data is collected. • Can NOT be conducted remotely. • Can be used on any system.

8. Performance Measurement • What is it? • Used to collect quantitative data. • Typically, you will be looking for benchmark data. • Objectives MUST be quantifiable • 75% of users shall be able to complete the basic task in less than 30 minutes.

9. Performance Measurement • How can I do it? • Define the goals that you expect users to perform

10. Performance Measurement • How can I do it? • Quantify the goals • The time users take to complete a specific task. • The Ratio between successful interactions and errors. • The time spent recovering from errors. • The number of user errors. • The number of commands or other features that were never used by the user. • The number of system features the user can remember during a debriefing after the test. • The proportion of users who say that they would prefer using the system over some specified competitor.

11. Performance Measurement • How can I do it? • Get participants for the experiments • Conduct very controlled experiments • All variables must remain consistent across users

12. Performance Measurement • Problems With Performance Measurement • No qualitative data.

13. Thinking-aloud Protocol • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 4.

14. Thinking-aloud Protocol • Usability Issues Covered • Effectiveness: Yes • Efficiency: No • Satisfaction: Yes • Quantitative Data is NOT collected. • Can NOT be conducted remotely. • Can be used on any system.

15. Thinking-aloud Protocol • What is it? • Technique where the participant is asked to vocalize his or her thoughts, feelings, and opinions while interacting with the product. .

16. Thinking-aloud Protocol • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Ask the participant to perform a task using the software.

17. Thinking-aloud Protocol • How can I do it? • During the task, ask the user to vocalize • Thoughts, opinions, feelings, etc. • http://www.cs.umd.edu/~zzj/ThinkAlo.htm

18. Thinking-aloud Protocol • Problem With Thinking-Aloud Protocol • Cognitive Overload • Can you walk & chew gum at the same time? • Asking the participants to do too much.

19. Question-asking Protocol • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 4.

20. Question-asking Protocol • Usability Issues Covered • Effectiveness: Yes • Efficiency: No • Satisfaction: Yes • Quantitative Data is NOT collected. • Can NOT be conducted remotely. • Can be used on any system.

21. Question-asking Protocol • What is it? • Similar to Thinking-aloud protocol. • Instead of participant saying what they are thinking, the evaluator prompts the participant with questions while using the system.

22. Question-asking Protocol • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Ask the participant to perform a task using the software.

23. Question-asking Protocol • How can I do it? • During the task, ask the user to questions about the product • Thoughts, opinions, feelings, etc. • http://www.cs.umd.edu/~zzj/Question.htm

24. Question-asking Protocol • Problem With Thinking-Aloud Protocol • Cognitive Overload++ • Can you walk, chew gum & talk at the same time? • Asking the participants to do too much. • Added pressure when the evaluator asks questions. • Can be frustrating on novice users.

25. Coaching Method • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 4.

26. Coaching Method • Usability Issues Covered • Effectiveness: Yes • Efficiency: No • Satisfaction: Yes • Quantitative Data is NOT collected. • Can NOT be conducted remotely. • Can be used on any system.

27. Coaching Method • What is it? • A system expert sits with the participant and acts as a coach. • Expert answers the participant’s questions. • The evaluator observes their interaction.

28. Coaching Method • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Ask the participant to perform a task using the software in the presence of a coach/expert.

29. Coaching Method • How can I do it? • During the task, the user will ask the expert questions about the product • http://www.cs.umd.edu/~zzj/Coaching.htm

30. Coaching Method • Problem With Coaching Method • In reality, there will not be a coach present. • This is good for creating a coaching system, but not for evaluating the interface.

31. Co-Discovery Learning • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 6.

32. Co-Discovery Learning • Usability Issues Covered • Effectiveness: Yes • Efficiency: No • Satisfaction: Yes • Quantitative Data is NOT collected. • Can NOT be conducted remotely. • Can be used on any system.

33. Co-Discovery Learning • What is it? • Two test users attempt to perform tasks together while being observed. • They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product. • They are encouraged to explain what they are thinking about while working on the tasks. • Thinking Aloud, but more natural because of partner.

34. Co-Discovery Learning • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Ask the participants to perform a task using the software.

35. Co-Discovery Learning • How can I do it? • During the task, the users will help each other and voice their thoughts by talking to each other. • http://www.cs.umd.edu/~zzj/Codiscov.htm

36. Co-Discovery Learning • Problem With Co-Discovery Learning • Neither is an expert • The blind leading the blind.

37. Teaching Method • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 4.

38. Teaching Method • Usability Issues Covered • Effectiveness: Yes • Efficiency: No • Satisfaction: Yes • Quantitative Data is NOT collected. • Can NOT be conducted remotely. • Can be used on any system.

39. Teaching Method • What is it? • You have 1 participant use the system. • Ask the participant to teach a new novice participant how to use the system.

40. Teaching Method • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Ask the 1st participant to perform a task using the software. • Ask the 1st participant to teach a new participant.

41. Teaching Method • How can I do it? • Observe their interactions. • http://www.cs.umd.edu/~zzj/Teaching.htm

42. Teaching Method • Problem With Teaching Method • Neither is an expert • The blind leading the blind. • Possible to discover some interesting things about the learn-ability of your interfaces.

43. Retrospective Testing • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 4.

44. Retrospective Testing • Usability Issues Covered • Effectiveness: Yes • Efficiency: Yes • Satisfaction: Yes • Quantitative Data can be collected. • Can NOT be conducted remotely. • Can be used on any system.

45. Retrospective Testing • What is it? • A videotape of the session is observed by the usability expert and the participants.

46. Retrospective Testing • How can I do it? • Select the participants, who will be involved? • Select the tasks and design scenarios. • Use one of the usability testing methods that we have discussed. • Videotape the session.

47. Retrospective Testing • How can I do it? • Review the videotape with the users. • http://www.cs.umd.edu/~zzj/Retrospe.htm

48. Retrospective Testing • Problem With Retrospective Testing • Extremely time consuming!

49. Remote Testing • Applicable Stages: • Design, Code, Test & Deployment • Personnel • Usability Experts, approximately 1. • Developers, 0. • Users, 5.

50. Remote Testing • Usability Issues Covered • Effectiveness: Yes • Efficiency: Yes • Satisfaction: Yes • Quantitative Data can be collected. • Can be conducted remotely. • Can be used on any system.