Mobile and Pervasive Computing - 3 Pervasive Computing User Studies

1. A.J. Bernheim Brush, ajbrush@microsoft.com Mobile and Pervasive Computing - 3Pervasive Computing User Studies Presented by: Dr. Adeel Akram University of Engineering and Technology, Taxila,Pakistan http://web.uettaxila.edu.pk/CMS/SP2014/teMPCms

2. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

3. Dr. A.J.Bernheim Brush ? http://research.microsoft.com/en-us/groups/CSCW/ Ph.D. in Computer Science Researcher at Microsoft Research Technology for families, workgroups (HCI/CSCW/Ubicomp) Researching on various “studies”

4. Why do a user study? • Bad Reasons • You don’t have anything else to do… • You think it’s a requirement to get your paper accepted • It might be fun to see how people use your stuff • Good Reasons • You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it? • There is a new domain or Scenario and you want to observe user behavior

5. The Reality…. • User Studies are a lot of work • Really, more work than you ever expected • Understanding your GOALis critical • Studying current behavior: What are people doing now? • Proof of concept: Does my novel technology work for people? • Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things

6. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

7. Many types of studies Formative/Current Behavior Summative/ Usage Experience Prototyping/Proof of Concept Ethnography Survey Interviews Focus Groups Logging “Discount Usability” Heuristic Evaluations Lab studies Lab studies Field Studies Logging Types Learn about a domain Design inspiration Does it work? Can people use it? How does it compare to other designs/prototypes? Iterative Testing Fix & understand your prototype Goal

8. Many types of studies Formative/Current Behavior Summative/ Usage Experience Prototyping/Proof of Concept Ethnography Survey Interviews Focus Groups Logging “Discount Usability” Heuristic Evaluations Lab studies Lab studies Field Studies Logging Types Learn about a domain Design inspiration Does it work? Can people use it? How does it compare to other designs/prototypes? Iterative Testing Fix & understand your prototype Goal

9. Surveys • Easy to get large number of people • Design guidance • Evaluation of deployed system • Surprisingly hard to do well…. • Phrasing of questions • Biased responses • Pilot your survey!

10. “Discount Usability” (Jakob Nielsen) http://www.nngroup.com/people/jakob-nielsen/ If you are building prototype, very useful to get feedback from users early and often • Low-cost, Quick, Iterative, Small N, Identify big problems • Lo-fi prototypes • Paper version can be very helpful • People feel ok telling you to change stuff • No feedback on responsiveness etc. • Heuristic evaluation • Experts review the interface based on list of heuristics • Cognitive walk-through • Determine tasks, review and ask questions for each task WARNING: Not typically a research contribution

11. Lab Studies http://www2.sta.uwi.edu/usability/facilities.htm • Bring participants into a lab • Minimize variability • Hypothesis testing • Independent variables between conditions • Interface A vs. Interface B • Control condition? • Measure dependent variable • Speed of use, … (Quantitative) • Preference, …. (Qualitative)

12. Field study • In-situ (“not on your turf”) • Trading “control” for realism • Think carefully if this is important • All types • current behavior, • proof-of-concept • prototype WARNING: Often good idea to do lab study before field study

13. How do I choose? LINC: An Inkable Digital Calendar http://research.microsoft.com/en-us/um/people/ajbrush/LINCCSCW2006Video.wmv http://research.microsoft.com/pubs/69395/cscw2006lincvideo.pdf You might not… What is your goal/research contribution?

14. Research Question/Goal Bad: How will families use SPARCS? Better: Do sharing suggestions promote sharing? There are very few right decisions, instead decisions you need to justify http://research.microsoft.com/apps/pubs/default.aspx?id=73932 Your Research Question is critical

15. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

16. Study Design • What type of study? • What will your participants do during the study? • Give them hardware? Give them tasks? • What type of participants you should recruit? • What data will you collect? • How long will the study be? • Where can you skimp during the study…. • What absolutely has to work (if it’s a prototype)

17. Human Subjects • Ethical treatment of people in the study • Respect—remember that they are doing YOU a favor • Participants can stop at any time • Consent Forms • Privacy Statements • Compensation • Your organization should have some review process • THIS IS IMPORTANT! (U.S. Department of Health & Human Services) • What will they be doing during the study • How will you report on what they did http://www.hhs.gov/ohrp/assurances/index.html

18. PILOT!!!! IMPORTANT IMPORTANT IMPORTANT • Allow time to a pilot study, • Run through the entire methodology with volunteer participants. • Uncover system problems • Uncover experimental design problems • Uncover problems with materials

19. Participant Profile • What types people do you want to participate? • All the same?

20. Participant Profile • What types people do you want to participate? • All the same? • All different?

21. Participant Profile • What types people do you want to participate? • All the same? • All different? • People with extreme characteristics vs. “normal” people

22. Participant Profile • What types people do you want to participate? • All the same? • All different? • People with extreme characteristics vs. “normal” people • Consider • Age • Gender • Technology experience • ….

23. How many participants? • This can be a difficult question • What claims you are making • What is feasible • Some will drop out!

24. Length • How long should the study be? • Another difficult question…. • Novelty • What are you asking participant to do? • Long term use vs. feasibility

25. Data Collection • How will you understand if you answered your research question? • Quantitative Data (logs, timing, # errors, …) • Qualitative Data (interviews, surveys, …) • TRIANGULATE between multiple sources

26. Logging • You must have a plan going in about how you will use the log data • Risk of forgetting to log something important • Logging too much can create an analysis nightmare • Make a list of questions you expect to answer with log data • How many times did they upload a photo? • How many days did they use your prototype?

27. Logging

28. Qualitative Data • Surveys • Pre-survey • Post condition • Post-survey • Experience Sampling Methodology • Small set of question • Event triggered, random…. • Diaries • Interviews (semi-structured, structured) • Observation

29. Analyzing Qualitative Data Affinity Diagramming Coding of Comments Inter-rater reliability

30. What if it doesn’t work? • There are many ways a study can fail • Technical problems • They don’t like it • Nobody but you cares about usability problems • Brace yourself for this • Figure out what has to work and skimp other places • Comparison between prototypes • Pilot studies

31. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

32. Example CareNet Consolvo, Roessler, Selton Intel Research Seattle

33. The CareNet Display: Lessons Learned from an In Home Evaluation of an Ambient Display Sunny Consolvo, Peter Roessler, & Brett E. Shelton Intel Research Seattle November ’04

34. Meet Rita… • Rita is one of the elders we worked with • 83 years old • Lives alone • Conditions: Mild dementia & Type 2 Diabetes (takes insulin) • She is struggling to maintain her independence, but needs help… The CareNet Display * Intel Research Seattle

35. Rita’s Care Network Significant contributor: son Zack Hannah’s boyfriend Drastic life changer: daughter Hannah Son Daughter-in-law Significant contributor: son Simon Neighbor Part-time professional caregiver Daughter-in-law The CareNet Display * Intel Research Seattle

36. The focus of this work • Improve the quality of life for ALL care network members, including the elder • Help members coordinate care activities • Ensure elder gets care she needs • Give time back to overburdened members The CareNet Display * Intel Research Seattle

37. The CareNet Display • Interactive digital picture frame • always-on Internet access from wireless GPRS card • Augments photo with updates • meds, meals, outings, activities, mood, falls, & shared calendar • Goal: help local care network members provide daily care The CareNet Display * Intel Research Seattle

38. Participants & Methodology • 13 participants from 4 care networks • In home Wizard of Oz deployments • i.e., phone calls instead of sensors • Interviews & surveys • No special instructions on: • how or when to use it, or where to place it The CareNet Display * Intel Research Seattle

39. Some Results • Drastic Life Changers: • time back, less stress, “meaningful” conversations • Significant Contributors: • increased awareness of what others contribute • Peripherally Involved Members: • gave them something to talk about • Elders: • changed minds about how open they would be The CareNet Display * Intel Research Seattle

40. Challenges: Sensor data with a “human touch” • Participants are afraid sensor data will be too impersonal • How can we provide this “human touch” without adding to the responsibilities of already overburdened network members? The CareNet Display * Intel Research Seattle

41. Early & in situ evaluations • Early evaluations help us learn where to concentrate our technology development • In situ evaluations provide us with valuable insight into how new tools are used and what effects they have on the intended users The CareNet Display * Intel Research Seattle

42. For more information… • Publications: • S. Consolvo, P. Roessler, & B.E. Shelton, "The CareNet Display: Lessons Learned from an In Home Evaluation of an Ambient Display," Proceedings of the 6th Int'l Conference on Ubiquitous Computing: UbiComp '04 (Sep 2004), pp.1-17. • S. Consolvo, P. Roessler, B.E. Shelton, A. LaMarca, B. Schilit, & S. Bly, "Technology for Care Networks of Elders," IEEE Pervasive Computing Mobile and Ubiquitous Systems: Successful Aging, Vol. 3, No. 2, (Apr-Jun 2004), pp.22-29. The CareNet Display * Intel Research Seattle

43. Lots of other examples… ButterflyNet Yeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger, Paepcke Stanford, Maryland SuperBreakMorris, Brush, Meyers Microsoft Research UbiFitConsolvo, McDonald, Toscos, Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand, Libby, Smith, LandayIntel Research Seattle Wearable Jersey DisplayPage and Moere University of Sydney

44. Outline • Introduction • Types of Studies • Study Design • Example • Ten Mistakes to Avoid • Your Turn

45. #10 Not enough people involved • It takes a village • Huge time commitment, 24 hour support for field studies • Do • Include multiple people

46. #9 Not being prepared WARNING: If you do not plan, you plan to fail! • Don’t want to realize at the end of the study that you forgot to do something important • Do • Study design document • Research Question • Participant Profile • Methodology (within/between etc) • Timeline • Data collection • Pilot studies

47. #8 Not enough time for logistics • Everything takes time…. (more than you think) • Recruiting • Installation (e.g. 16 people X 2 hours = ) • Support • Do • Allow plenty of time for the study • Have enough people

48. #7 Seeing what you want to see • We all want our prototypes to be popular • Do: • Think carefully about how you discuss the technology with participants • Avoid leading questions • Stay close to the data and find multiple support for conclusions • Use neutral language, “Tell me more,” “ummm” • Don’t get defensive

49. #6 Being judgmental • Users are always “right” • They may say things that are offensive, objectionable, etc. • Do: • Leave your opinions at home • Collect feedback • Use neutral language

50. #5 Not monitoring usage • Don’t want to find out at the end of the study that people were not using the prototype • Do: • “phone home” messages • Server logs • Have a plan about when and how you might intervene