1 / 41

Gathering Usability Data

Gathering Usability Data. Observing users & subjective data. Directing Sessions. Issues: Are you in same room or not? Single person session or pairs of people Objective data -- stay detached. Collecting Data. Data gathering Note-taking Audio and video tape Instrumented user interface

preston-nunez
Download Presentation

Gathering Usability Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Gathering Usability Data Observing users & subjective data

  2. Directing Sessions • Issues: • Are you in same room or not? • Single person session or pairs of people • Objective data -- stay detached PSYCH / CS 6750

  3. Collecting Data • Data gathering • Note-taking • Audio and video tape • Instrumented user interface • Post-experiment questions and interviews PSYCH / CS 6750

  4. Collecting Data • Identifying errors can be difficult • Qualitative techniques • Think-aloud - can be very helpful • Post-hoc verbal protocol - review video • Critical incident logging - positive & negative • Structured interviews - good questions • “What did you like best/least?” • “How would you change..?” PSYCH / CS 6750

  5. Observing Users • Not as easy as you think • One of the best ways to gather feedback about your interface • Watch, listen and learn as a person interacts with your system PSYCH / CS 6750

  6. Observation • Direct • In same room • Can be intrusive • Users aware of your presence • Only see it one time • May use 1-way mirror to reduce intrusiveness • Indirect • Video recording • Reduces intrusiveness, but doesn’t eliminate it • Cameras focused on screen, face & keyboard • Gives archival record, but can spend a lot of time reviewing it PSYCH / CS 6750

  7. Location • Observations may be • In lab - Maybe a specially built usability lab • Easier to control • Can have user complete set of tasks • In field • Watch their everyday actions • More realistic • Harder to control other factors PSYCH / CS 6750

  8. Challenge • In simple observation, you observe actions but don’t know what’s going on in their head • Often utilize some form of verbal protocol where users describe their thoughts PSYCH / CS 6750

  9. Verbal Protocol • One technique: Think-aloud • User describes verbally what s/he is thinking and doing • What they believe is happening • Why they take an action • What they are trying to do PSYCH / CS 6750

  10. Think Aloud • Very widely used, useful technique • Allows you to understand user’s thought processes better • Potential problems: • Can be awkward for participant • Thinking aloud can modify way user performs task PSYCH / CS 6750

  11. Teams • Another technique: Co-discovery learning (Constructive interation) • Join pairs of participants to work together • Use think aloud • Perhaps have one person be semi-expert (coach) and one be novice • More natural (like conversation) so removes some awkwardness of individual think aloud PSYCH / CS 6750

  12. Alternative • What if thinking aloud during session will be too disruptive? • Can use post-event protocol • User performs session, then watches video afterwards and describes what s/he was thinking • Sometimes difficult to recall • Opens up door of interpretation PSYCH / CS 6750

  13. Historical Record • In observing users, how do you capture events in the session for later analysis? • ? PSYCH / CS 6750

  14. Capturing a Session • 1. Paper & pencil • Can be slow • May miss things • Is definitely cheap and easy Task 1 Task 2 Task 3 … Time 10:00 10:03 10:08 10:22 S e S e PSYCH / CS 6750

  15. Capturing a Session • 2. Recording (audio and/or video) • Good for talk-aloud • Hard to tie to interface • Multiple cameras probably needed • Good, rich record of session • Can be intrusive • Can be painful to transcribe and analyze PSYCH / CS 6750

  16. Capturing a Session • 3. Software logging • Modify software to log user actions • Can give time-stamped key press or mouse event • Two problems: • Too low-level, want higher level events • Massive amount of data, need analysis tools PSYCH / CS 6750

  17. Issues • What if user gets stuck on a task? • You can ask • “What are you trying to do..?” • “What made you think..?” • “How would you like to perform..?” • “What would make this easier to accomplish..?” • Maybe offer hints • Can provide design ideas PSYCH / CS 6750

  18. Subjective Data • Satisfaction is an important factor in performance over time • Learning what people prefer is valuable data to gather PSYCH / CS 6750

  19. Methods • Ways of gathering subjective data • Questionnaires • Interviews • Booths (eg, trade show) • Call-in product hot-line • Field support workers • (Focus on first two) PSYCH / CS 6750

  20. Questionnaires • Preparation is expensive, but administration is cheap • Oral vs. written • Oral advs: Can ask follow-up questions • Oral disadvs: Costly, time-consuming • Forms can provide better quantitative data PSYCH / CS 6750

  21. Questionnaires • Issues • Only as good as questions you ask • Establish purpose of questionnaire • Don’t ask things that you will not use • Who is your audience? • How do you deliver and collect questionnaire? PSYCH / CS 6750

  22. Questionnaire Topic • Can gather demographic data and data about the interface being studied • Demographic data: • Age, gender • Task expertise • Motivation • Frequency of use • Education/literacy PSYCH / CS 6750

  23. Interface Data • Can gather data about • screen • graphic design • terminology • capabilities • learning • overall impression • ... PSYCH / CS 6750

  24. Question Format • Closed format • Answer restricted to a set of choices • Typically very quantifiable • Variety of styles PSYCH / CS 6750

  25. Closed Format • Likert Scale • Typical scale uses 5, 7 or 9 choices • Above that is hard to discern • Doing an odd number gives the neutral choice in the middle Characters on screen hard to read easy to read 1 2 3 4 5 6 7 PSYCH / CS 6750

  26. Other Styles Rank from 1 - Very helpful 2 - Ambivalent 3 - Not helpful 0 - Unused Which word processingsystems do you use? LaTeX Word ___ Tutorial ___ On-line help ___ Documentation FrameMaker WordPerfect PSYCH / CS 6750

  27. Closed Format • Advantages • Clarify alternatives • Easily quantifiable • Eliminate useless answer • Disadvantages • Must cover whole range • All should be equally likely • Don’t get interesting, “different” reactions PSYCH / CS 6750

  28. Open Format • Asks for unprompted opinions • Good for general, subjective information, but difficult to analyze rigorously • May help with design ideas • “Can you suggest improvements to this interface?” PSYCH / CS 6750

  29. Questionnaire Issues • Question specificity • “Do you have a computer?” • Language • Beware of terminology, jargon • Clarity • Leading questions • Can be phrased either positive or negative PSYCH / CS 6750

  30. Questionnaire Issues • Prestige bias • People answer a certain way because they want you to think that way about them • Embarrassing questions • Hypothetical questions • “Halo effect” • When estimate of one feature affects estimate of another (eg, intelligence/looks) PSYCH / CS 6750

  31. Deployment • Steps • Discuss questions among team • Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions • Administer final test PSYCH / CS 6750

  32. Interviews • Get user’s viewpoint directly, but certainly a subjective view • Advantages: • Can vary level of detail as issue arises • Good for more exploratory type questions which may lead to helpful, constructive suggestions PSYCH / CS 6750

  33. Interviews • Disadvantages • Subjective view • Interviewer can bias the interview • User may not appropriately characterize usage • Time-consuming PSYCH / CS 6750

  34. Interview Process • How to • Plan a set of questions (provides for some consistency) • Don’t ask leading questions • “Did you think the use of an icon there was really good?” • Can be done in groups • Get consensus, get lively discussion going PSYCH / CS 6750

  35. Data Analysis • Simple analysis • Determine the means (time, # of errors, etc.) and compare with goal values (coming up…) • Determine • Why did the problems occur? • What were their causes? PSYCH / CS 6750

  36. Experimental Results • How does one know if an experiment’s results mean anything or confirm any beliefs? • Example: 20 people participated, 11 preferred interface A, 9 preferred interface B • What do you conclude? PSYCH / CS 6750

  37. Hypothesis Testing • In experiment, we set up a “null hypothesis” to check • Basically, it says that what occurred was simply because of chance • For example, any participant has an equal chance of preferring interface A over interface B PSYCH / CS 6750

  38. Hypothesis Testing • If probability result happened by chance is low, then your results are said to be “significant” • Statistical measures of significance levels • 0.05 often used • Less than 5% possibility it occurred by chance PSYCH / CS 6750

  39. Presentation Techniques Middle 50% Age low high Mean 0 20 Time in secs. PSYCH / CS 6750

  40. Upcoming • Audio • Web PSYCH / CS 6750

  41. Using the Results • How do you use the results of your evaluation? • How can you make your design better with this knowledge? PSYCH / CS 6750

More Related