experience sampling method: preliminary findings

1. experience sampling method: preliminary findings sunny consolvo & miriam walker intel research seattle sunny@intel-research.net user-centered evaluations of ubicomp workshop 4th int’l conference on ubiquitous computing september 29, 2002

2. before we get started… • this work ispreliminary • wrapped up data collection this month • expect future publication experience sampling method -- ubicomp 2002

3. outline of talk • intro to experience sampling method (esm) • case study: the personal server • issues • wrap-up experience sampling method -- ubicomp 2002

4. esm: a brief introduction • also called: • beeper studies • time sampling • field study technique from psychology • used to understand areas such as… • mood • time use • social interactions • we’re exploring its uses for ubicomp experience sampling method -- ubicomp 2002

5. esm: a brief introduction • fill out questionnaire when alerted • researcher is not present • alerted several times per day • business as usual until alerted • duration is often 1-3 weeks experience sampling method -- ubicomp 2002

6. running esm: tools • need to choose tools for: • alerting participant • random alerts • timed alerts • event-based/user-driven • delivering questionnaire • capturing participant responses experience sampling method -- ubicomp 2002

7. choosing tools: alerting • tools for alerting the participant include: • cell phones • pagers • watches • PDAs • custom devices • some desirable abilities: • specify time period during which alerts may go off • specify # of alerts per time period & overall experience sampling method -- ubicomp 2002

8. choosing tools: delivering • tools for delivering questionnaires include: • paper booklets • PDAs • phones (traditional or mobile) • live researcher • recorded message • audio recordings (e.g., cassette player) experience sampling method -- ubicomp 2002

9. choosing tools: capturing • tools for capturing responses include: • paper booklets • PDAs • phones (traditional or mobile) • live researcher • recording service (e.g., Tellme) • audio recordings (e.g., cassette player) • cameras experience sampling method -- ubicomp 2002

10. esm: some advantages • reduces effects of intrusive observers • larger sample size & time interval • reduces memory biases by asking about current activity, feelings, etc. • it’s all about now experience sampling method -- ubicomp 2002

11. esm: more advantages • less time consuming than observations, yet… • study activities as they occurin the field • study activities and emotions often not accessible to observers • not as disruptive to others experience sampling method -- ubicomp 2002

12. esm: some disadvantages • questionnaires must be brief • participant must carry tools at all times • power consumption and/or service availability • certain events may only show up in very large samples sizes/durations • more intrusive than interviews & surveys experience sampling method -- ubicomp 2002

13. outline of talk • intro to experience sampling method (esm) • case study: personal server • issues • wrap-up experience sampling method -- ubicomp 2002

14. case study: personal server • personal server team was interested in: • how people use information • what they want • when they want it • where they want it • used esm to help collect this data experience sampling method -- ubicomp 2002

15. pilot study #1 • 4 participants • lab researchers & administrative staff • palm m100s for • alerting • delivering • capturing • original boston college ESP software • for running ESM on palms • duration: 1½ - 4 days experience sampling method -- ubicomp 2002

16. pilot study #1: problems • palm dropped  batteries fell out  data lost • only method of alert was beep (m100) • participants wouldn’t carry it in several situations • once volume of beep was set, it couldn’t be changed • only response option: multi-choice, single answer • difficult for participants to answer some questions • e.g., “where are you?” • some participants wanted a bag to carry the palm • participants wanted to know about missed questionnaires experience sampling method -- ubicomp 2002

17. pilot study #2 • 4 participants • lab researchers, administrative staff, family • modified iESP software • palm m500s for alerting and delivering • for capturing: • 2 participants carried fold-out keyboards (for palm) • 2 participants carried paper booklets • all 4 carried digital cameras • for describing where & what they were doing • duration: 1½ - 4 days experience sampling method -- ubicomp 2002

18. pilot study #2: justifications • upgrading to m500s: • ability to alert via beep and/or vibrate • almost impossible for batteries to fall out • iESP software added functionality, including: • freeform text entry • multi-choice, multi-answer • new formats for multi-choice, single answer • slider (e.g., “on a scale of 1 to 5…”) • ability to change volume, vibrate, etc. during study • ability to check how many questionnaires have been completed vs. asked during study experience sampling method -- ubicomp 2002

19. our implementation: the study • 31 participants • recruited from local community • pre/post-study interviews + 1 week with equipment • 10 questionnaires/day • 70 questionnaires total • incentive (in USD) • $50 for participating and returning equipment • $1 per completed questionnaire • $120 total possible experience sampling method -- ubicomp 2002

20. the study (continued) • palm m500 to alert, deliver, & capture • modified iESP software • more about capturing: • also used camera • free-form data entry: • paper booklet for all 31 participants • graffiti / soft keyboard for all 31 participants • full-sized fold-out keyboard for 9 of 31 participants experience sampling method -- ubicomp 2002

21. what the participants carried experience sampling method -- ubicomp 2002

22. type of data collected • responses to following types of questions: • yes / no • multiple choice, single answer • multiple choice, multiple answers • freeform text entry • looks a lot like survey data except • multiple responses from same participant in variety of circumstances over time experience sampling method -- ubicomp 2002

23. other data collected • qualitative data from interviews • participants’ photos • some photos were of interest to the participant • other photos documented where s/he was or what s/he was doing experience sampling method -- ubicomp 2002

24. questions on the palm experience sampling method -- ubicomp 2002

25. more questions experience sampling method -- ubicomp 2002

26. participant photos: in transit experience sampling method -- ubicomp 2002

27. participant photos: at home experience sampling method -- ubicomp 2002

28. participant photos: at work experience sampling method -- ubicomp 2002

29. participant photos: retail experience sampling method -- ubicomp 2002

30. participant photos: pets experience sampling method -- ubicomp 2002

31. outline of talk • intro to experience sampling method (esm) • case study: personal server • issues • wrap-up experience sampling method -- ubicomp 2002

32. our tool decisions… • chose palm because: • no service problems—cost or coverage • power consumption not an issue for duration • know when participant fills out questionnaire • probabilities and contingencies can be used • if a, then b • if b, then not c • d 50% of time • don’t have to transcribe data experience sampling method -- ubicomp 2002

33. tool decisions • used camera to see if it would help participants describe… • where they were • what they were doing • photos used to jog memories during post-study interviews experience sampling method -- ubicomp 2002

34. according to the participants… • 1/31 would not participate again • 2/31 would only do it for more money • 28/31 would participate again • “it was cool” • “I felt like a spy” • “friends were jealous of the [fold-out] keyboard” • a lot to carry • mixed feelings about carry bag • several claimed they would be purchasing palms • only one participant wanted voice recording option • when asked, most did not respond favorably experience sampling method -- ubicomp 2002

35. from our perspective… • woo-hoo! we got all of the equipment back • only 1/31 palms was broken • of the 31 participants and 70 questionnaires: • 28 participants completed 50+ questionnaires • 13 of whom completed 60+ questionnaires • 2 participants completed 25- questionnaires • 1 participant performed a hard reset and returned the palm with no data or software experience sampling method -- ubicomp 2002

36. outline of talk • intro to experience sampling method (esm) • case study: personal server • issues • wrap-up experience sampling method -- ubicomp 2002

37. wrap up: we’ll use it again • pleased with results so far • intend to use esm again experience sampling method -- ubicomp 2002

38. wrap-up: pda software • original ESP software available at: • http://www2.bc.edu/~barretli/esp/ • iESP software (includes upgrades): • launching on intel research seattle web site in Oct or Nov 2002 • if you want to be notified… • business card with “iESP notification” written on it • legibly fill out notification sign-up sheet experience sampling method -- ubicomp 2002

39. acknowledgements • drs. lisa & daniel barrett • justin silves • gaetano borriello • ken anderson • rahul shah • trevor paring • roy want • john light • our pilot testers • and our colleagues at intel research seattle experience sampling method -- ubicomp 2002

40. questions? experience sampling method -- ubicomp 2002

41. examples of study questions • where is your cell phone? in my… • hand • pocket/belt clip • bag • car • home • office • somewhere else and with me • somewhere else and not with me experience sampling method -- ubicomp 2002

42. examples of study questions • in the last 30 minutes, which of the following would you have liked to have access to but didn’t have available? • web pages • movie times • restaurant reviews • personal calendar • telephone number(s) • computer files • music files • shopping list • “to do” list experience sampling method -- ubicomp 2002