Foxtrot, by Bill Amend 10/31/02

1. Foxtrot, by Bill Amend 10/31/02

2. Amy Thurston UserWorks, Inc. 20 November 2002 Usability Testing

3. What is usability? • An effective, efficient, satisfying relationship among: • a system or product • its users • what the user wants to accomplish • Considering both • initial ease of use (intuitiveness, ease of learning) • operational ease of use (productivity, ease of retention)

5. What is the goal of usability? • To improve aspects of the product with which users come into contact • To make user interfaces logical, intuitive and clear to people who use them • Includes the entire user experience • physical design (dimensions, layout, packaging) • look and feel (displays, controls) • procedures (the steps to complete a task) • environment (physical, geographic) • availability/usefulness of support (online help, training, documentation, help line support)

7. Types of usability testing • Testing with users • Heuristic / expert evaluation • Online survey • Click-path monitoring • Concept evaluations Considerations • These techniques can be complimentary • Ideally, one would use groups of these as you go through an iterative design process

8. Categories of usability testing • Formative - Evaluations based on observing representative users completing important and frequent tasks with a prototype of the product. Errors, accuracy, speed of performance, and analyze user comments to recommend design changes • Summative / Acceptance - Usability engineering specialists observe and record performance and user comments at the end of a design cycle • Benchmarking - Establishes a level of product usability for future testing and measurement

9. ______________ / / Visual Design User Testing Takes: 2days-2weeks / Surface / Look and Feel Heuristic Evaluation Lasts: 2weeks - 2mos /_____________/ ______________ / / Interface Design User Testing Takes: 2days-2weeks / Skeleton / Heuristic Evaluation Lasts: 2weeks - 2mos /_____________/ Participatory Design ______________ / / Interaction Design Card sorting Takes: 2days-2weeks / Structure / Prototype Testing Lasts: 1mo - 6mos /_____________/ ______________ / / User Modeling User Interviews Takes: 2weeks - 2months / Scope / Project Objectives Personas/Scenarios Lasts: 3mos - 1year /_____________/ Mental Models ______________ / / Deep User Research Ethnography Takes: 6weeks - 6months / Strategy / Business Goals Contextual Research Lasts: 1year - 5years /_____________/ --Peter Merholz, peterme.com, based on a structure by Jesse James Garrett, jjg.net

10. Objectives of usability testing • Plan • Prepare • Collect data • Analyze • Report the data

11. What’s not on the list? • Taking action on the findings • Many times, implementation is left to another group • Findings need to be balanced with business decisions • What’s best for the user might not be best for the product • Decision makers often have a broad array of reports which inform their actions; usability is only one of these

12. Planning • What are the goals of testing? • Is the product in development and is there a need for guidance? • Is the product about to be deployed and is there only time for “tweaking” • What if there are show-stoppers? • Note that the final results will only be as good as the procedures are appropriate • Select the best usability testing technique

13. Planning • Who are the users? • Are there multiple user groups? • How many users should be tested? • Do we need to test users from all groups? • What type of test is this? (i.e,, do we need quantitative, random sampling based data? Do we need to turn around the results in two days?) • What or who determines if the product is usable?

16. Planning • What should be tested? • What are the users typically trying to do? • Are there some basic functions that have to be right? • Are there some functions that seem confusing or problematic? • Do anticipated environmental situations affect usability?

17. Planning • Who will receive the report? • What happens to the information that will be generated? • Will changes be made to the product based on the data?

18. What if you don’t know the users?

19. Types of user needs and task analysis activities • Interviews / focus groups • Competitive / comparative analysis • Contextual inquiry • Walkthrough / talkthrough • Link analysis

20. Interviews / focus groups • Interview users / conduct focus groups to probe for user needs on specific design features or preferences • Example: • Interviewer: “This product has the capability to do a three-way conference call. Would that be sufficient?” • User: “No; sometimes we have 4 to 5 locations that need to be involved.”

21. Competitive / comparative analysis • Examine other similar products • Determine typical user scenarios and evaluate products • List features, noting their design and, if possible, their effectiveness

22. Contextual inquiry • Understand procedure • Develop training criteria • Determine information needs

23. Walkthrough / talkthrough • Walkthrough only of use where there are well-established procedures and existing products • May pick up more dynamic problems than a static task analysis • Enough time to accomplish? • Physical distance between needed information

24. Link analysis • Task analysis focusing on: • Communication links • Physical access • Emphasis on frequency of contact • Often used for: • Workplace design • Task allocation • Determining instrumentation requirements

25. Preparation • Materials needed for testing and test plan • Screener to “qualify” participants • Video consent form • Tasks for participants (enough to fill the time, and a little more) • Pre-test and post-test questionnaires • Test plan

26. Preparation: Screener • Based on a description of user group(s) • Determine which characteristics are important for choosing participants • Some important characteristics are used to determine group membership (i.e., profession, experience, location) • Other characteristics should be balanced within the group (i.e., gender and age are frequently balanced)

27. Preparation: Screener • Use the screener to verify participant is willing to be video taped, if appropriate • After the potential participant successfully completes the screener, use the opportunity to schedule the participant • Participants who are “no-shows” are costly • Give incentives • Make reminder calls

28. Preparation: Consent Form • Is it legally necessary to receive written permission from a participant to video or audio tape them? • Note that testing can still go forward if they agree but that data will be without back-up verification • You need the participant’s agreement to use their image • Some clients need written non-disclosure statements from participants

29. Preparation: Tasks • Tasks should be written • Basic parts of the product (i.e., the critical functions or paths) • Parts of particular interest to the client • Parts that • Are known to cause confusion and problems • You anticipate will cause confusion and problems

30. Preparation: Tasks • Tasks should be pilot tested with someone unfamiliar with the product • To assure that the task wording is clear • To determine how long it takes to accomplish the tasks • To determine if the tasks are “do-able” • To find if there are alternate “correct” paths

31. Preparation: Pre-test and Post-test Questionnaires • Questionnaires can be of many types • Specifically designed for the particular test • General questionnaires designed by others (e.g., QUIS from U MD) • Validated questionnaires such as SUS (by John Brooke at Redhatch for DEC, free), or SUMI and WAMMI (from Jurek K. at University College Cork, not free) • Marketing questionnaires • Benchmarking, baseline measures

32. Preparation: Post-task Questionnaires • Purpose is to determine from the participant’s point of view • Satisfaction • Efficiency • Ease of learning • Ease of use • If feel in control • If like system or not

33. Preparation: Test Plan • A script for the administrator • Gives place for checklist of what to say • Can give precise wording if study needs to adhere to more precision • Gives the anticipated “correct” path for ease of the administrator to follow • Introduction should be delivered in person even if test is a non-interference test

34. Testing • Testing team can be one or more persons • Participant fills out preliminary forms • Start the tape! • Use test guide to direct the “production” • Introduce test • Have user do task(s) • Note usability issues, comments (by hand or software) • Have user answer questions, questionnaire(s) • Debrief, thank, and compensate

35. Testing • Materials to have participant complete before test • Demographic, consent forms • Use the test guide during the test • Materials to have been completed by end of test • Questionnaires, forms for usability • Data log and/or other notes

36. Analysis: the Basics • Usability heuristics • Constantine and Lockwood (see http://www.foruse.com/) • Jakob Nielsen (see http://www.useit.com/papers/heuristic/heuristic_list.html and http://www.useit.com/alertbox/9710a.html) • Expert opinions (see references) • Industry best practices • Experience

37. Report • Usability tests must be documented • To have the appropriate effect • To allow the powers-that-be to make appropriate decisions • To serve the in the historical context of a project

38. Report • Helpful to know these answers before starting the report • Who is the report for? • How will it be used? • How much time after testing is allowed for analysis and reporting? • What will be the most effective (or acceptable) format?

39. Report • Basic information about participants and test • Test based data • Analysis of the data--prioritize problem severity • High--cannot complete the task or causes data loss • Medium--likely to cost user time or difficulty • Low--not substantial but worth considering • Recommendations (if appropriate for the type of test) including further testing if problems are severe or complicated

40. Report Formats • Written reports • Top line report (1 p) • Brief report (5-7pp) • Systematic scientific (academic style) report • PowerPoint reports • Common Industry Format for software analysis reports (but there is no place for recommendations) • Briefings • Formal with presentation • Telephone based • Highlights tape • Collection of clips of participants using the product with voice over comments or introductory titles

41. Sample results from a usability test • Overall, 4 of 5 users liked the product • 3 of 5 users had some difficulty completing Task 3. The average rating for completing this task was 5.3 on a 9 point scale. The three users that had difficulty selected the blue button instead of the red button because the names are ambiguous and confusing. It is recommended that one or both of these names change to reflect their opposing effects. • 5 of 5 users completed Task 6. Users commented that they liked this feature and found it easy to complete the task. Average rating was 8.7 on a 9 point scale.

42. Usability resources Online: • Usability.gov - National Cancer Institute • Useit.com - Jakob Nielsen • Usableweb.com - Keith Instone • UPAssoc.org - Usability Professionals’ Association • UIE.com - User Interface Engineering

43. Good Luck in Your Own Usability Testing Amy Thurston athurston@userworks.com UserWorks, Inc 1738 Elton Rd., Suite 138 Silver Spring, MD 20903 Phone 301-431-0500 FAX 301-431-4834