Pervasive Computing User Studies

1. Pervasive Computing User Studies A.J. Bernheim Brush

2. Who am I? • Ph.D. in Computer Science • Researcher at Microsoft Research • Technology for families, workgroups (HCI/CSCW/Ubicomp) • “Love” studies

3. Why do a user study? • Bad Reasons • You don’t have anything else to do… • You think it’s a requirement to get your paper accepted • It might be fun to see how people use your stuff • Good Reasons • You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it? • There is a new domain or behavior you want to observe

4. The Reality…. • User Studies are a lot of work • Really, more work than you ever expected • No, really I’m not kidding • Understanding your GOALis critical • Studying current behavior: What are people doing now? • Proof of concept: Does my novel technology work for people? • Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things WARNING: INTERACTIVE EVENT AT THE ENDSTART THINKING ABOUT ONE OF YOUR OWN PROJECTS

5. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

6. Many types of studies Formative/Current Behavior Prototyping/Proof of Concept Summative/ Usage Experience Ethnography Survey Interviews Focus Groups Logging “Discount Usability” Heuristic Evaluations Lab studies Lab studies Field Studies Logging Types Learn about a domain Design inspiration Does it work? Can people use it? How does it compare to other designs/prototypes? Iterative Testing Fix & understand your prototype Goal

7. Many types of studies Formative/Current Behavior Prototyping/Proof of Concept Summative/ Usage Experience Ethnography Survey Interviews Focus Groups Logging “Discount Usability” Heuristic Evaluations Lab studies Lab studies Field Studies Logging Types Learn about a domain Design inspiration Does it work? Can people use it? How does it compare to other designs/prototypes? Iterative Testing Fix & understand your prototype Goal

8. Surveys • Easy to get large number of people • Design guidance • Evaluation of deployed system • Surprisingly hard to do well…. • Phrasing of questions • Biased responses • Pilot your survey!

9. “Discount Usability” (Jakob Nielsen) If you are building prototype very useful to get feedback from users early and often • Low-cost, Quick, Iterative, Small N, Identify big problems • Lo-fi prototypes • Paper version can be very helpful • People feel ok telling you to change stuff • No feedback on responsiveness etc. • Heuristic evaluation • Experts review the interface based on list of heuristics • Cognitive walk-through • Determine tasks, review and ask questions for each task WARNING: Not typically a research contribution

10. Lab Studies • Bring participants into a lab • Minimize variability • Hypothesis testing • Independent variables between conditions • Interface A vs. Interface B • Control condition? • Measure dependent variable • Speed of use, … (Quantitative) • Preference, …. (Qualitative) http://www2.sta.uwi.edu/usability/facilities.htm

11. Between vs. Within Subjects Designs • Between subjects • Participants each in different condition • E.g., everyone randomly assigned to a group • ANOVA, t-test • Within subjects • Each participants experiences all conditions • E.g., expertise, practice or individual differences accounted for • RM ANOVA, paired t-test • Have to worry about repetition and carryover effects • Counterbalancing, Latin square, etc. I PREFER DUE TO PARTICIPANT VARIABILITY

12. Statistics • Descriptive Statistics (count, mean, …) • Nominal – categories: frequency only • Ordinal – Ranked preference: frequency, median • Interval – numbers: frequency, sum, median, mean.. • Watch out for outliers • Inferential statistics • Are results statistically significant between groups? • T-test, ANOVA, paired t-test, etc. • Significance values • If p < 0.01, there is a 99% chance that the data collected represents a real difference in the population rather than a sampling error

13. Field study • In-situ (“not on your turf”) • Trading “control” for realism • Think carefully if this is important • All types • current behavior, • proof-of-concept • prototype WARNING: Often good idea to do lab study before field study

14. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

15. How do I choose? • You might not… • What is your goal/research contribution? LINC: An Inkable Digital Calendar

16. Research Question/Goal Your Research Question is critical Bad: How will families use SPARCS? Better: Do sharing suggestions promote sharing? There are very few right decisions, instead decisions you need to justify

17. Study Design • What type of study? • What will your participants do during the study? • Give them hardware? Give them tasks? • Why type of participants you should recruit? • What data will you collect? • How long will the study be? • Where can you skimp during the study…. • What absolutely has to work (if it’s a prototype)

18. Human Subjects • Ethical treatment of people in the study • Respect—remember that they are doing YOU a favor • Participants can stop at any time • Consent Forms • Privacy Statements • Compensation • Your organization should have some review process • THIS IS IMPORTANT! • What will they will be doing during the study • How will you report on what they did • http://www.hhs.gov/ohrp/irb/irb_chapter3.htm

19. PILOT!!!! • Allow time to a pilot study, • Run through the entire methodology with volunteer participants. • Uncover system problems • Uncover experimental design problems • Uncover problems with materials IMPORTANT IMPORTANTIMPORTANT

20. Participant Profile • What types people do you want to participate? • All the same?

21. Participant Profile • What types people do you want to participate? • All the same? • All different?

22. Participant Profile • What types people do you want to participate? • All the same? • All different? • People with extreme characteristics vs. “normal” people

23. Participant Profile • What types people do you want to participate? • All the same? • All different? • People with extreme characteristics vs. “normal” people • Consider • Age • Gender • Technology experience • ….

24. How many participants? • This can be a difficult question • Between or within subjects • What claims you are making • What is feasible • Some will drop out!

25. Length • How long should the study be? • Another difficult question…. • Novelty • What are you asking participant to do? • Long term use vs. feasibility

26. Data Collection • How will you understand if you answered your research question? • Quantitative Data (logs, timing, # errors, …) • Qualitative Data (interviews, surveys, …) • TRIANGULATE between multiple sources

27. Logging • You must have a plan going in about how you will use the log data • Risk of forgetting to log something important • Logging too much can create an analysis nightmare • Make a list of questions you expect to answer with log data • How many times did they upload a photo? • How many days did they use your prototype?

28. Logging

29. Qualitative Data • Surveys • Pre-survey • Post condition • Post-survey • Experience Sampling Methodology • Small set of question • Event triggered, random…. • Diaries • Interviews (semi-structured, structured) • Observation

30. Analyzing Qualitative Data • Affinity Diagramming • Coding of Comments • Inter-rater reliability

31. What if it doesn’t work? • There are many ways a study can fail • Technical problems • They don’t like it • Nobody but you cares about usability problems • Brace yourself for this • Figure out what has to work and skimp other places • Comparison between prototypes • Pilot studies

32. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

33. Example CareNetConsolvo, Roessler, SeltonIntel Research Seattle

34. CareNet Methodology • 4 “care networks” (3F, 1M) • 13 people (4 elders and 9 care givers) • 2-3 people had CareNet displays • 3 weeks • Compensation • With CareNet - $150 • Elders $75- $300 (depend on data provided) • Sept. – Dec. 2003

35. CareNet Data Collection • Interviews, before and after (semi-structured, recorded) • Questionnaires: half-way, end • Photos • 3-6 times per day by phone to collect data for display • Glowing display • People will take your technology apart, even when you tell them not too.

36. Lots of other examples… ButterflyNetYeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger, PaepckeStanford, Maryland UbiFitConsolvo, McDonald, Toscos, Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand, Libby, Smith, LandayIntel Research Seattle SuperBreakMorris, Brush, Meyers Microsoft Research Wearable Jersey DisplayPage and MoereUniversity of Sydney

37. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

38. #10 Not enough people involved • It takes a village • Huge time commitment, 24 hour support for field studies • Do • Include multiple people

39. #9 Not being prepared • Don’t want to realize at the end of the study that you forgot to do something important • Do • Study design document • Research Question • Participant Profile • Methodology (within/between etc) • Timeline • Data collection • Pilot studies WARNING: If you do not plan, you plan to fail!

40. #8 Not enough time for logistics • Everything takes time…. (more than you think) • Recruiting • Installation (e.g. 16 people X 2 hours = ) • Support • Do • Allow plenty of time for the study • Have enough people

41. #7 Seeing what you want to see • We all want our prototypes to be popular • Do: • Think carefully about how you discuss the technology with participants • Avoid leading questions • Stay close to the data and find multiple support for conclusions • Use neutral language, “Tell me more,” “ummm” • Don’t get defensive

42. #6 Being judgmental • Users are always “right” • They may say things that are offensive, objectionable, etc. • Do: • Leave your opinions at home • Collect feedback • Use neutral language

43. #5 Not monitoring usage • Don’t want to find out at the end of the study that people were not using the prototype • Do: • “phone home” messages • Server logs • Have a plan about when and how you might intervene

44. #4 Not collecting a variety of data • Hard to understand logs without interviews • Hard to know if what someone says in an interview matches their use without logs • Do • Interviews • Surveys • Logs • Support for your findings in multiple ways gives you (and reviewers) more confidence

45. #3 Prototype is not robust enough • Technical challenges are a distraction • You are making a “product” • Do • Usability studies • Pilots with friendly folks

46. #2 Making inappropriate claims • One of the biggest and most common mistakes • Your study is one data point • Do: • Being careful with your language • Include a “limitations” section • Tell a clear story (don’t have to tell every finding)

47. #1 Not having a clear research question • Hard to explain the choices you made • Hard to explain your findings • May end up focusing on usability problems • Do • Have a clear research question

48. Your Turn…. • Get into groups of 2-3 • Choose an example prototype • Determine a research question • Come up with a study design • Participants • Length • Data to collect • Timeline for study …

49. Your thoughts

50. Acknowledgments Ed Cutrell MSR India Sunny Consolvo Intel Research Seattle Beverly Harrison Intel Research Seattle