HCI460: Week 5 Lecture

1. HCI460: Week 5 Lecture • October 7, 2009

2. Outline • A few more words about project 1b (Expert Evaluation Report) • Plan for next Wednesday • Test Plan: Common feedback themes • Moderating usability sessions • Observing sessions and note taking • Guest lecture: • Speaker: Ed Israelski, Human Factors Manager, Abbott • Title: “Usability Methods in Healthcare” • Time for questions, feedback on your Moderator’s Guide and Screener, group work

3. A Few More Words About Project 1b

4. A Few More Words About Project 1b Points (out of 15) in-class DL

5. A Few More Words About Project 1b Can I Resubmit? • Everyone can resubmit their Project 1b. • No grade change • No deadline • Feedback will be provided verbally during office hours • We will look at how you have improved from the first report to the second and third.

6. Plan for Next Wednesday

7. Plan for Next Wednesday Formative Usability Testing of: Samsung TV remote control TRUEtrack glucose meter Policy declarations page for an auto insurance company Flip Video Mino HD Franklin Spelling Ace Tenori-On digital musical instrument (synthesizer) Livescribe Pulse Smartpen Corner Bakery printable pdf menu Slacker Radio application on the BlackBerry Storm Google Maps Brookstone AM/FM radio alarm clock

8. Plan for Next Wednesday Location • Address: • 500 North Michigan Avenue • Floor 16 • Suite 1610 – User Centric, Inc. • Directions: • http://www.usercentric.com/directions_chicago.html

9. Plan for Next Wednesday Schedule

10. Plan for Next Wednesday Groups Group 1: Dietz, Dulski, Dzienisowicz, Remington Group 2: Birdseye, Dash, Gamboa, Welense Group 3: Elliott, Petlick Group4: Berberian, Devlin, Shah, Sienkiewicz Group 5: Albarracin, Ginez, Schwarz Group 6: Epps, Jones, Lund Group 7: Barbera, Schulte, Schwantes, Young Group 8: Canady, Doshi, Roberts Group 9: Garcia, Goldberg, Haines, Wickenkamp Group 10: Diemer, Komosa, Ranguelov, Young Group 11: Cheng, Freeman, Taylor

11. Plan for Next Wednesday Arrival Times • Groups 3, 5, 6, 8, 9 and 10 should arrive before 5:45pm. • If not all team members can arrive early, whoever can come in early should bring the product you are testing. • Groups 1, 2, 4, 7 and 11 should arrive at 7:15pm. • Your early arrival will help us make sure that the lab setup works with your product.

12. Plan for Next Wednesday Other Important Details • There will be only one test session per group. • You should conduct the rest of your sessions (min 5 total) on your own. • Limit your session to 20-25 minutes. • Typical session lengths: 30, 45, 60, 90, 120 minutes but class time is limited, so we will have shorter sessions. • You can conduct longer sessions on your own if you’d like. • Participants will be provided. • The moderator will be selected randomly from each group, so you all should be prepared to moderate. • Do a dry run before Wed session to make sure you can fit the session into a 20-25-minute time slot. • Don’t forget to bring what you are testing.

13. Plan for Next Wednesday Feedback • You will receive feedback from: • Gavin or Aga • Your participant • Your observers (other groups observing your test session) • Feedback will be provided on: • The moderation • The procedure • Each observer will fill out simple feedback sheets: • 1 – 3 things that you think the moderator did well • 1 – 3 things that the moderator should improve on • 1 – 3 good things about the procedure • 1 – 3 ways in which the procedure could be improved • All feedback will be compiled and posted on COL by EOD 17th.

14. Test Plan: Common Feedback Themes

15. Test Plans: Common Themes Scope • Narrow down your scope. • Keep in mind session length • Too many objectives / tasks • Comparing products instead of testing a single product • Good idea but not for formative testing with a few very short sessions.

16. Test Plans: Common Themes Business Objective • Provide a business objective. • Not critical but very helpful • In real life, you should get business objectives from stakeholders • Why are we testing this product? • To lower the return rate? • To decrease the volume of calls to the call center?

17. Test Plans: Common Themes Objectives and Tasks • Objectives and tasks/questions must be in perfect alignment. • Objectives without tasks/questions can lead to disappointed stakeholders. • Tasks/questions not addressing any objectives are a waste of time. • Did you forget to list an important objective?

18. Test Plans: Common Themes Stimuli and Measures • What are stimuli? • Stimuli = artifacts = stuff that will be tested • E.g., XYZ alarm clock, XYZ glucose meter, XYZ website • Use of quantitative measures (e.g., time on task, ratings) • Appropriate in summative testing • Why and how should we use quantitative measures in formative testing?

19. Moderating Usability Sessions

20. Moderating Usability Sessions Experience • Who has moderated? • Explain insights/lessons learned • What makes a good moderator?

21. Moderating Usability Sessions Need for Objectivity One basic tenet of usability testing is that it is virtually impossible to remain objective when usability testing your own product Separate design and test teams Reality is that some testing is better than no testing at all

22. Moderating Usability Sessions Characteristics of a Good Moderator (Rubin) Knowledgeable in UCD Quick learner Builds a rapport with participants / good communicator Excellent memory Good listener Comfortable with ambiguity Flexibility Long attention span Empathic “people person” “Big picture” thinker

23. Moderating Usability Sessions Characteristics of a Good Moderator (Lew/Bojko) Reading the guide vs. just reading the guide Biggest fear (study) Biggest fear (formative session)

24. Moderating Usability Sessions Depends on Technique • State task and participant does task • Discussions are limited • Think aloud protocol • Pros / cons • Retrospective review • Pros / cons

25. Moderating Usability Sessions Moderators Should Avoid… (Rubin) Leading…instead, enable Being overly consumed with data collection Acting too knowledgeable Being rigidly adhering to moderator’s guide Not relating well to every participant Jumping to conclusions

26. Moderating Usability Sessions Moderators Should Remember… (Lew & Bojko) • When in doubt, read • The power of silence (simply pause) • What it is like to cut video clips • When trying to nail a point, avoid questions with yes/no, instead ask for elaboration • Manage time, but also take your time (yeah, I know) • Don’t jumping to conclusions, BUT be aware of trends • Especially when you are the single moderator

27. Moderating Usability Sessions Simultaneous Testing Sessions • How many sessions can be run in a day? • 60 min sessions? • 90 min sessions? • 120 min sessions? • Consider sample size of N=12 • N=20 • N=20, but with a between groups design (N=40) • Implications • Parallel sessions • Ramp up

28. Moderating Usability Sessions Communication During Session • Pros and cons • Notes handed back • Walk outside • Ear buds • IM • Do not get flustered • Okay to wait a moment

29. Moderating Usability Sessions What if the Participant is… • Confused • Bad • Define bad • Lying • Drunk

30. Moderating Usability Sessions What if Your Mobile Usability Lab… • Somehow it looks worse after the luggage handlers touched it… • It is 7:45am. The first session starts at 9 am.

31. Moderating Usability Sessions What if the Prototype is Not Ready? • Cancel study? • Cost / timeline • Ready the day before? • Technique

32. Observing Sessions and Note Taking

33. Observing Sessions and Note Taking Team Size • Two-person test teams vs. one-person test teams • Single session • One moderator and one note taker • Pros/Cons • Switching considerations • One and only one moderator?

34. Observing Sessions and Note Taking Managing the Observation Room • Manage observation room • Redirect • Trends vs. outliers • Note trends on a whiteboard • Assess politics • Achievable recommendations

35. Screener and Moderator’s Guide

36. Screener and Moderator’s Guide Questions About the Mod Guide & Screener • Both the Screener and Moderator’s Guide are due on Friday 11:59 pm Central Time for in-class students (Tuesday for DL students). • Again, we will provide you with feedback. • Grade? • Project 2 will be graded after the final report is turned in. • As appendices, you should turn in: • Test Plan • Moderator’s Guide improved based on our feedback • Screener • Questions: • How will the screener be used? • ….?