Chapter 7 Usability and Evaluation

1. Chapter 7 Usability and Evaluation Dov Te’eni Jane M. Carey

2. Chapter 7 Usability and Evaluation • Usability and evaluation are closely related concepts • Usability means that the HCI design of the system supports the user’s cognitive and ergonomic limitations and is easy to use and learn • Evaluation means to assess the system for functionality as well as usability • We explore how to compare, assess, and improve interfaces

3. Chapter 7 Usability and Evaluation • Mini-Case 1984 Olympic message system demonstrates: • You will not get it right the first time • You will get different answers when you use different methods of observation • You will need to come up with integrative solutions to problems

4. Chapter 7 Usability and Evaluation • Designers of the OMS (Olympic Message System) followed 3 principles: • Early focus on user and tasks • Empirical measure on user’s behavior • Iterative design

5. Chapter 7 Usability and Evaluation Figure 7.2 Attitudes, use, performance, and satisfaction relationship

6. Chapter 7 Usability and Evaluation • ISO (International Standards Organization) defines usability as “ a concept comprising the effectiveness, efficiency, and satisfaction with which specified users can achieve specified goals in a particular environment”

7. Chapter 7 Usability and Evaluation Table 7-1: Usability indicators based on performance 1. Goal achievement indicators (success rate, failure rate, accuracy, effectiveness).2. Work rate indicators (speed, completion rate, efficiency, productivity, productivity gain).3. Operability indicators of the user's ability to make use of the systems features (error rate, problem rate, function usage)4. Knowledge acquisition indicators of the user's ability and effort in learning to use the system (learnability and learning).

8. Chapter 7 Usability and Evaluation Table 7-2: Usability measures based on performance and HCI 1. Time to complete a task2. Number of user commands to complete task3. Fraction of task completed4. Fraction of task completed in a given time5. Number of errors6. Time spent on errors7. Frequency of online help used8. Number of available commands not used9. When task is repeated, ratio of successes to failures10. Fraction of positive comments made by user11. Fraction of good to bad features recalled by user12. Number of expressions of frustration and satisfaction13. Number of times user loses control over system14. Number of times the user needs to devise a way of working around the problem/system.

9. Chapter 7 Usability and Evaluation Table 7-3: The usability engineering life cycle · Know the user· Analyze competing products· Set usability goals· Consider alternative designs· Engage in participatory design· Coordinate the total interface· Check against heuristic guidelines· Prototype· Evaluate interface· Design in iterations· Follow up with studies of installed systems

10. Chapter 7 Usability and Evaluation • Methods of usability assessments include: • Thinking aloud (protocol analysis) • Observation • Interviews • Focus groups • Automatic logs • Questionnaires

11. Table 7-4: Attitude questionnaire 1. Using a microcomputer could provide me with information that would lead to a better decisions.2. I wouldn't use a microcomputer because programming it would take too much time.3. I'd like to use a microcomputer because it is oriented to user needs.4. I wouldn't use a microcomputer because it is too time consuming.5. Using a microcomputer would take too much time away from my normal duties.6. Using a microcomputer would involve too much time doing mechanical operations (e.g., programming, inputting data) to allow sufficient time for managerial analysis.7. A microcomputer would be of no use to me because of its limited computing power.8. I'd like to learn about ways that microcomputers can be used as aids in managerial tasks.9. Using a microcomputer would result in a tendency to over design simple tasks.10. I wouldn't want to have a microcomputer at work because it would distract me from my normal job duties.11. A microcomputer would give me more opportunities to obtain the information that I need12. I wouldn't favor using a microcomputer because there would be a tendency to use it even when it was more time consuming than manual methods.13. I'd like to have a microcomputer because it is so easy to use.14. I'd hesitate to acquire a microcomputer for my use at work because of the difficulty of integrating it with existing information systems.15. I'd discourage my company from acquiring microcomputers because most application packages would need to be modified before they could be useful in our specific situation.16. It is easy to access and store data in a microcomputer.17. A microcomputer would be of no use to me because of the limited availability of application program packages.18. A microcomputer would be of no use to me because of its small storage capacity.19. It is easy to retrieve or store information from/to a microcomputer.20. Using a microcomputer would give me much greater control over important information.

12. Chapter 7 Usability and Evaluation Step 7 in the engineering life cycle is check against heuristic guidelines Table 7-5: Heuristic guidelines 1. Create simple and natural dialog2. Speak the user's language3. Minimize the user's memory load4. Be consistent5. Provide feedback6. Provide clearly marked exits7. Provide shortcuts8. Provide specific, corrective and positive error messages9. Minimize propensity for error

13. Chapter 7 Usability and Evaluation • Evaluation is a broader term than usability – it includes several goals: 1) assess the system's functionality against the intended specifications, 2) assess the system's effect on the user's behavior and attitude, 3) assess the system's impact on measures of performance that are related to the user or the objective of the system, 4) discover unintended problems and perhaps opportunities.

14. Chapter 7 Usability and Evaluation • Evaluation techniques include: 1) Exploratory vs. model based.2) Design or implementation.3) Field study vs. laboratory testing.4) Design vs. use.5) Level of performance measures.6) Degree of designed manipulation and intrusion.

15. Chapter 7 Usability and Evaluation Cognitive Walkthrough (Polson et al, 1992) * Task description from the first-time user's viewpoint. Include any special assumptions about the state of the system assumed when the user begins work. * Action sequence: Make a numbered list of the atomic actions that the user should perform to accomplish the task. * Anticipated users: Briefly describe the class of users who will use this system. Note what experience they are expected to have with similar or previous versions. * User's initial goals: List the goals the user is likely to form when starting the task. If there are other likely goals list them, and estimate for each what percentage of user are likely to have them.

16. Chapter 7 Usability and Evaluation Table 7-6: Measures for comparing displays (Scott & Findlay, 1991) · Typing mistakes made before correction· Mistakes remaining· Pauses of 3 seconds or more immediately before mode change· Other pauses of 3 seconds or more· Length of pause immediately before mode change· Attempting to type-over whilst in insert mode· Attempting to insert whilst in type-over mode

17. Chapter 7 Usability and Evaluation • Ben Shneiderman’s Questionnaire for user satisfaction

18. Chapter 7 Usability and Evaluation • Other methods of evaluation • Metrics (Rengger, 1991)

19. Chapter 7 Usability and Evaluation • Conclusion – Flow diagram from Olson and Moran (1996)

20. Chapter 7 Usability and Evaluation Section Status1 Introduction draft2 accessibility draft3 presentation of information ISO document4 user guidance ISO document5 direct manipulation ISO document6 color draft7 forms fill-in ISO document8 command languages ISO document9 voice i/o voice recognition draftnon-speech auditory output draftinteractive voice response draft10 visually displayed menus re-drafted ISO doc Table 7.8 ANSI/HFS 200 outline and status (1997)