Usability and Accessibility Research Plan

1. Usability with ProjectLecture 12 – 24/10/08 Dr. Simeon Keates

2. Exercise – part 1 • Prepare your research plan for investigating the usability and accessibility of your original and revised web site designs • Your plan must clearly include: • Summary • Research Issues (Aim / hypotheses to be tested) • Research Structure (Method) • Schedule (Method) • [ Budget (Method) ] • Deliverables (Results) • Also, prepare a 10 minute presentation to the “board” for Friday

3. Exercise – part 2 – Kuniavsky’s stages • Stage 1 – Deciding what to do • Collect issues and present them as goals (Who are the stakeholders? What are there issues? Etc. • Prioritise the issues (importance * severity) • Re-write the goals as questions to be answered • Stage 2 – Setting up a schedule • You will have a minimum of 4 afternoon slots and 2 morning slots available for your user trials + data collection + analysis + writing final report - use your time wisely! • You will be expected to test both versions of your web-sites with at least 4 users - one of whom should be blind (simulated or not)

4. Exercise – part 3 – Kuniavsky’s stages • Stage 3 – Estimating budgets • Not really that relevant here • How many cups of coffee? • Stage 4 – Preparing the research plan • More on this on the next slide • Stage 5 – Maintaining the plan • You will have opportunity to revisit and revise your plan over the next few weeks

5. Suggestions • Summary • Research Issues (Aim / hypotheses to be tested) • Does the new web-site offer increased usability and accessibility over the original design? • Add more aims! • Research Structure (Method) • Heuristic evaluation • Accessibility testing plan • Usability trials, etc. • Add more detail here! • Schedule (Method) • First two parts already completed • Usability schedule – add more detail here!

6. Suggestions (continued) • Budget (Method) • Not really applicable • Deliverables (Results) • Add more detail here! • What do you think you will test? • What quantitative tests will you use? • What qualitative tests will you use? • What measures will you use? • What do you think your success/failure criteria should be? • Remember – the more specific and explicit you make the plan, the easier the usability testing and report writing will become!!!

7. Conducting sessions with “accessibility” users

8. Conducting sessions with “accessibility” users • We will consider each of the following principal stages: • Pre-session preparation • Selection and preparation of the trial location • Conduct of the sessions • Follow-up to the sessions

9. Pre-session preparation • Planning the tasks to be performed • Can the users complete the “usability” tasks? • What is the purpose of the user trial? Identify accessibility issues, usability issues or usable access issues? • Need to add specific “accessibility” questions/tasks • Example: “Make a peanut butter sandwich” becomes “Can you tell which jar is the peanut butter?”, etc. • Can the users do them?

10. Pre-session preparation • Getting ethical approval • Implications of “accessibility” users • How is approval gained? (Example: user is deaf) • And recorded? (User cannot sign a consent form) • How can you be sure that the consent is valid? (Example: user has a cognitive impairment)

11. [Aside] Good consent practice Make it clear: • That the session can be paused at any point • That the participants can withdraw from the session at any point • That the participants can withdraw their data at any point • Outline the data to be collected, how it will be handled and who will be able to access it • Outline the process by which the data will be fully anonymised • Describe the risks to the participants in clear and unequivocal language • Give this to the participant in writing and ask for them to sign

12. [Aside] Good consent practice You should also: • Provide examples of consent forms that participants will be expected to sign prior to the session • Ensure that at no point are the participants exposed to potential risks of “harm” (not just physical, but also emotional)

13. Pre-session preparation Identifying the needs of the participants: • Navigation • How is the person going to get to the venue? • How is he/she going to navigate around the venue? • Communication • What mode of communication should be used? • Written / spoken / Braille / sign language, etc.? • Assistive technologies • Does the users use AT? • Will they bring their own? Or do you have to supply it? • Will it work with your product?

14. Pre-session preparation Conducting pilot studies • As for “usability” • Also test for accessibility of the venue, instructions, etc. • Also do tests to ensure that the AT works

15. Pre-session preparation Preparing and distributing pre-session documentation • What to include? • All consent forms that the participants will have to sign • Details of any reimbursement offered • Details of the time and location of the trials • Directions to the location where the trials are to be held • The tasks to be performed and details of how the session will be structured and run (the experimental protocol) • Send these out in good time (allow extra time for reading and replying) • Check the correct format • Braille • Audio files/tapes • etc.

16. Selecting a trial location • Best for you? • Best for data collection/analysis? • Best for the user?

17. Challenges of trial locations

18. Selecting a trial location – Usability Lab Advantages: • Tightly controlled • Already set up (usually) • Sufficient space and vantage points Disadvantages: • Unfamiliar for participants • Can be intimidating • May not be accessible • Getting there • Navigating within

19. Selecting a trial location – 3rd party locations (e.g. day care centres) Advantages: • Usually known by users • Familiar • Usually already accessible Disadvantages: • Need to set up (possibly daily) • Sufficient space and vantage points? • Privacy? • Possible interruptions

20. Selecting a trial location – Remote testing Advantages: • No travel requirement for users • Should have necessary AT and optimised computer access • Cheap (time, resources and money) Disadvantages: • Reduced feedback from users • Difficulty of communication if user is (e.g.) deaf • Difficulty observing coping strategies • Often difficulties getting software installed, data mailed back, etc.

21. Conducting the sessions Need to consider 3 main components • Pre-trial briefing • Performance of the tasks • Interviews and final debrief

22. Pre-trial briefing • What mode will be used for the briefing? • Spoken / written / sign language, etc. • Find what works best for the user • Ideally, as much material as possible should be sent to the user ahead of the session • Remind users that they are in control • You may need to reassure them

23. Performance of the tasks • Allow more time for task completion • It may simply take more time • Do not coerce users into doing tasks that are beyond them • Be prepared for data “incompleteness” • Watch out for tiredness • It is not good for the user or your data collection • Keep reassuring the participants • “No – the computer is not about to blow up!” • Keep obersvers to a miminum • Do not crowd the user • Look out for coping strategies

24. Interviews and final debrief • As for pre-trial briefing • What mode will be used for the briefing? • Spoken / written / sign language, etc. • Find what works best for the user • Ideally, as much material as possible should be sent to the user ahead of the session

25. Working with “accessibility” users - Summary • Do not make assumptions about the needs of the participants • If in doubt – ask • Make sure that the most relevant users are selected to participate in the trials • Plan ahead and be prepared for unexpected results

26. How many users???

27. Choosing how many users • For this discussion we will assume: • (a) you have accurately identified the right types of users • (b) there is one major user demographic • (c) the users are failry homogeneous (i.e. similar) within that demographic • We will relax these constraints later

28. Jakob Nielsen and 5 to 8 users • The most famous answer to this question is from Jakob Nielsen • He suggested 5 to 8 users Logic: • Landauer and Nielsen showed that: Number of usability problems found = N(1-(1-L)n) where: N = total number of usability problems in the design L = proportion of usability problems found by a single user • A typical value for L = 31%

29. Jakob Nielsen and 5 to 8 users • Plotting for L = 31%

30. Jakob Nielsen and 5 to 8 users Important observations: • 0 users find 0 problems! • 1 users finds approximately 1/3rd of the problems • 15 users find approximately all of the problems Notes: • These numbers are based on L=31%, that may not always be the true value • Depends on the size and complexity of the product • Nielsen recommends 5 users (not 10 or 15) to save costs • They’ll find approximately 85% of the problems • He prefers 3 trials with 5 people to 1 trial with 15 • More robust…

31. Nielsen and card sorting • Nielsen does not recommend 5 users for everything • He recommends 15 to 20 users for card sorting • He reasons that there’s only ever a 75% correlation between the results from 5 users and the ultimate results • 15 users give 90% correlation • 20 users give 93% correlation • 30 users give 95% correlation • i.e. an additional 5% for twice the number of people! • 60 people reach 98% correlation

32. Nielsen and card sorting • Why the difference from 5-8 to 15-20? • User trials are for evaluations • i.e. trying to find out how good something that exists is • Card sorting is a generative method • i.e. trying to establish the parameters for a new product • Greater variability in the data to be collected and analysed • e.g. different perspectives of the fundamental problem • different mental models to be captured and synthesised • etc. • Can (and should) combine card sorting and user trials

33. Other perspectives • Nielsen and Landauer recommend 5 to 8 users • Laurie Faulkner recommends 15 users • “Beyond the 5 user assumption: Benefits of increased sample sizes in usability testing” • Still more people recommend other numbers • So what should it be? • Let’s look at what we are trying to achieve…

34. Why don’t we do large numbers in usability? (source: Frank Spillers “Demystifying usability” • We are looking for behavioural insight (what people do) • Statistics tell you how many people failed, but not necessarily why they did so • The objective is usually to apply findings to fix design problems in a corporate context (not academic analysis) • Research shows that even with low numbers, you can gain valid data • Note: this is especially true for “accessibility” testing • Usability testing has been used for 25 years • Experts, authors and academics put their reputations behind the methodology

35. Behaviour vs. opinion • Market research is largely opinion-driven • You need to ask lots of people what they think • Needs very large samples – often 00s or 000s • Usability research is behaviour-driven • You observe what people do in a restricted context, not what they say • If 10/15 users are confused, you can assume many more will be • Behaviour is governed by cognitive factors (biological and otherwise) not opinion which is more emotional (and thus variable)

36. Faulkner’s approach 60 users were involved in usability trials • Assumed that 100% of problems were found Taking random collections of 5 people out of the 60 • Some sets found 99% of the problems • Some found only 55% Taking random collections of 10 people out of the 60 • Lowest success was 80% Taking 20 people out of 60 • Lowest success was 95% Note – these results depend on the quality of the 60 users selected

37. Suggested sample sizes Spillers suggests the following for corporate user research • Surveys (phone and web) = ~240 to ~1000+ • Focus groups = 15 to 20 (depending on goals of study) • Usability trials = 10 to 15 participants • Field studies = 15 to 40 participants • Card sorting = 15 to 30 (higher is better because of the analyses methods needed) • Accessibility trials – typically as many as you can get hold of

38. Exercise

39. Exercise – part 1 • Prepare the testing protocol for evaluating the accessibility and usability of your web-site • Also, address any additional research aims identified in your research plan from Wednesday

40. Exercise – part 2 You need to consider the following: • Pre-session briefing • Prepare your welcome statement • What you are doing and why • Privacy issues and right to withdraw • Any initial questions you wish to ask • Prepare a consent form • Tasks • Identify at least 5 tasks for each user on each site • Ensure you do not introduce systematic errors • Prepare any likely questions you may wish to ask • Remember to add/amend tasks for the “blind” test

41. Exercise – part 3 You need to consider (continued) • Post-session de-brief • Ask any remaining questions needed to address your research issues • Thank the user for their time • E-mail your protocol to Anne and me • Remember – you will be putting this to the test next week!

42. Exercise – suggestions for tasks • Exploring the site / describe each page • Great for getting users used to what is where • Completing a guided product selection task • Find “this” product • Completing an unguided product selection task • Find “any” product of your choice • Changing your mind • You decide you do not want this • How many types of [x] (example: tea)?

43. Exercise – additional points • Decide whether all users do the same tasks in the same order or not • Be on the lookout for “order” effects • You should randomise the presentation of the sites • ½ do site 1 first • ½ do site 2 first