Testing and Modeling Users

1. Testing and Modeling Users Kristina Winbladh & Ramzi Nasr

2. Usability • Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks • We want to build usable products • How do we know whether our product is usable? • Testing

3. Usability Testing • Planning a usability test • Defining goals and concerns • Choosing participants • Design • Predictive Models

4. Planning a Usability Test • How long does usability testing take? • Important for clients and managers to know • Development does not have to stop • Time factors • Complexity of the product? • How much of the product is to be tested? • Number of participants? • Test preparation? • Test evaluation/report?

5. Examples • Microsoft • 8-12 weeks • Testers work together with developers • Formal reports • General Electric Information Services • 4-6 weeks • 3 days to determine participants and tasks • Week 1&2 test: scenario & questionnaire design • End of week 2: pilot test • Week 3&4: conducting the test w 12 participants • Week 5: test data analysis • Week 6: finished the report

6. Can you do it faster? • 1 Week • In-house usability specialists conduct test • Each test covers only a few features • Participants background is already known • Tasks are short • Generate short report • 1.5 days… • Special case

7. What can you do to make testing successful? • Planning! • Starting testing w/o thinking about: • What aspects might not be as usable as they should be? • How well do the participants represent the actual users? • What tasks will we ask them to do in the short time they have? • What information are we going to collect? • How are we going to analyze the information? • What are we going to do with the analyzed information?

8. The steps • Define the goals and concerns that are driving the test • Decide who should be participants • Recruit participants • Select and organize tasks to test • Create task scenarios • Decide how to measure usability • Prepare other materials • Prepare testing environment • Prepare test team • Conduct pilot test

9. Team Work • Usability specialists • Designers and developers • Technical communicators • Trainers • Marketing • Helpdesk or customer assistance

10. Usability Testing • Planning a usability test • Defining goals and concerns • Choosing participants • Design • Predictive Models

11. Defining Goals and Concerns • You cannot do exhaustive testing • Make choices among goals and concerns • Move from general concerns to specific ones • Understand sources of goals and concerns

12. Making Choices Among Goals and Concerns • Clearly define the concern that the study will answer • Prioritize all the concerns

13. Moving From General Concerns to Specific Ones • Refinement • Example • General: how easy is it to use the system? • Specific: how long does this specific task take?

14. Understanding Source of Goals and Concerns • Sources: • Task analysis and quantitative usability goals • Timely issues • A heuristic analysis • Previous tests of this or other products

15. Usability Testing • Planning a usability test • Defining goals and concerns • Choosing participants • Design • Predictive Models

16. Choosing Participants • Steps for choosing participants: • Developing user profiles • Selecting subgroups for a test • Defining and quantifying characteristics for each subgroup • Deciding how many people to include in a test

17. Developing User Profiles • Collaborative work with: • Marketing, usability specialists, product designers • Use • Surveys, focus groups sessions, contextual interviews, current users of previous version • Characteristics: • Those that all users will share • Those that might make a difference among the users

18. Profile factors • Work experience • General computer experience • Specific computer experience • Experience with this product • Experience with similar products

19. Selecting Subgroups for a Test • Dividing users into subgroups • Adding more characteristics • Selecting the most critical characteristics • Gathering other information

20. Defining and Quantifying Characteristics of Each Subgroup • “Intermediate” users • Setting a minimum and a maximum for a subgroup • Selecting a range of participants within each subgroup

21. Deciding How Many People to Include in a Test • Less than half of usability problems found with 3 participants (Nielson and Molich, 1990) • 80% of usability problems detected with 4-5 participants (Virzi, 1992) • 90% of usability problems detected with 10 participants (Virzi, 1992)

22. Deciding How Many People to Include in a Test (Cont.) • The number of participants to include in a test depends on: • Number of subgroups needed • Budget and time • Importance of computing statistical significance of results

23. Usability Testing • Planning a usability test • Defining goals and concerns • Choosing participants • Design • Predictive Models

24. Design • Techniques • Video, interaction logging, questionnaire, interviews • Tasks • Time to complete • Number/Type of errors per task/unit time • Number of help requests • Number of users per error/success • Testing conditions

25. Design (Cont.) • Variables and Conditions • Allocation of participants • Different participants • Same participants • Matched pairs of participants • Data collection and analysis

26. Usability Testing • Planning a usability test • Defining goals and concerns • Choosing participants • Design • Predictive Models

27. Predictive Models • Measure user performance without testing the users • GOMS (Goal, Operators, Methods, Selection rules) • The Keystroke Level Model

28. Balance Time, Money, and Information Gain