Usability Research: Unexpected Results

1. Usability Research:Unexpected Results

2. Overview • User feedback and user performance • Unexpected results in research/usability • Small scale research accuracy

3. Product: College Writing Handbook How it is tested • Textbook publishers – experts • Computer industry – usability tests

4. Product: College Writing Handbook Research Goals • Convince publishers of value of usability testing • Compare visual against traditional version • Help find/fix problems in new handbook • Usability research: new thinking about old products

5. Delivery of Material Old way: prose instruction Traits Traditional Vocabulary Information listed by bullets Emphasis – minimal visuals “^” Italics

6. Delivery of Material New Way: artwork Traits Minimal grammar terminology Color coding Simpler language “adding a medial modifier to an independent clause”

7. User Profiles: 12 Participants • All 18 or 19 years old • First semester, first composition course • Engineering, Business, etc • 6 student from 2 year college • 4 male, 2 female • 6 students from 4 year college • 3 male, 3 female

8. Scenarios 1. “Put complex source into correct MLA style” 2. “Identifying non-trivial comma errors” 3. Evaluate source acceptability for assignment/audience • Talk out loud while using textbook • Point at text while reading • Prompted after 5 seconds of silence • Videotaped

9. Scenario 1: Citation • Two sources provided • Use both handbooks – alternate handbook used first • Ratings: • Very useful • Useful • Rarely Useful • Not useful • Explain the rating

10. Scenario 2: Punctuation • Evaluate a paragraph “pregnant with comma errors” • Comma required? • Comma optional? • Ease of use • Explain the rating

11. Scenario 3: Acceptability of Source • Users given a research topic and audience • Users given possible sources • Acceptable? • Unacceptable? • More information required? • Ease of use

12. “Print Quality Bias” • Prototype vs finished product • Color copies of prototype vs color excerpts • Texts plastic comb bound • Both texts referred to as prototypes

13. Findings • Visual product preferred by users • Verbal product rated slightly more difficult

14. “Users failed at tasks, but didn’t realize it” • Ease of use does not equal usability • Works cited - both prototypes failed users • 12 unsatisfactory work cited entries • Minor omissions: “Press” or “Inc” • Critical omissions: authors, title, edition number, pages • Punctuation – 11 of 12 students misuse comma • Source acceptability Howard 10

15. Creating a citation • Positive responses • 12/12 user failure • Problem areas • Large font, highlighting, underlining • Users misled – other info required for citation

16. “In America* it is quite possible to live a cocoon.” • Correct Response • Comma optional at * • Visual prototype - page 433 • Verbal prototype - page 236 • User results • 11/12 gave “required” as response – both books • Visual prototype – 12/12 cited the correct page • Verbal – 12/12 cited incorrect page

17. Explanation of failures • “Readers scanned pages for examples that matched mental models” • “They thought the problem was simple and didn’t look beyond the first solution…” • Relied on bold headings, skipping paragraphs Howard 11

18. Explanation of failures cont’d • “Visual manual tried to combine too much information in one graphic.” • “Authors of the manuals didn’t understand their users’ mental models.” • One text failed: possible delivery problem? Howard 11

19. First Simple Solution • Users appeared to focus on bold headings • Scanned examples • Looked for examples to match pattern of task • Rarely read prose paragraphs

20. Additional Issues • Visual is too complex • User comments • “’tangled up’” or “’messy’” • “’Too busy’” • “Too much effort” • Skipped it

21. Preferred page – why? • Users scan for syntax patterns • “Does not…combine elements into one visual”

22. Acceptability of Source • Step-by-step instructions: too simple • Provide context or “’If, Then’” scenarios • Visual book used “stories” • Pedro, Aaliyah with respective assignments • Both students evaluate the same source • Story shows decision making process • User-centered design? User-experience!

23. Context • Both texts made assumptions • Knowledge of corporate authors, reference books, etc • How to determine the type of source • Fixes • Task Environment • “’How do quote or paraphrase in my text?’” • “’How do I format entry for works cited, reference list, etc” Howard 14

24. Other Results • Fixes for handbooks – visuals, complexity indicators • Total client focus can be bad • “I like/want this!” • Focus: task completion AND decision making • Make users aware of complexity – context • Usability Test/final product • Model problem solving behavior in usability test

25. Small Scale Research Accuracy

26. Small Numbers? • Revised handbook • Usability results vs actual user results? • Task success • Oversimplified results • Extreme results

27. Confidence Interval • 95% - by convention • 95% of the time - results fall within planned range • Based on sample size and success rate • 5 users – large margin for error • 100 users – smaller margin for error

28. Confidence Interval • 5 users - 95% of the time, completion of tasks will be between 48% - 100%

29. Adjustment – Maximum Likelihood Estimate (MLE) 4/5 = .80 or 80% • Successful attempts (x)/total attempts(n) • x/n = probability of success (p) • x/n = p

30. Adjustment – Jeffreys Method 4/5 = .80 or 80% 4.5/6 = .75 or 75% • Successful attempts (x)/total attempts(n) • x/n = probability of success (p) • x/n = p • (x+.5)/(n+1)

31. Adjustment – Laplace Method 4/5 = .80 or 80% 5/7= .714 or 71.4% • Successful attempts (x)/total attempts(n) • x/n = probability of success (p) • x/n = p • (x+1)/(n+2)

32. Adjustment – Wilson Method 4/5 = .80 or 80% 6/9= .667 or 66.7% • Successful attempts (x)/total attempts(n) • x/n = probability of success (p) • x/n = p • (x+2)/(n+4)

33. Adjustment Review 4/5 = .80 80% MLE 4.5/6 = .75 75% Jeffreys 5/7= .714 74.1% Laplace 6/9= .667 66.7% Wilson • 5/5 = 100% - Really?

34. Sample less than 20, use adjustment method www.measuringusability.com/wald 4 of 5 users succeed: 71.4% Keep it Simple Lewis and Sauro 2-15

35. Summary • Writing Handbook • Unexpected results – usable, but not useful • Adjusting for small samples • http://www.upassoc.org/upa_publications/jus/ • www.measuringusability.com/wald