Measuring wireless network usage with the Experience Sampling Method

1. Measuring wireless network usage with the Experience Sampling Method • Tristan Henderson, Denise Anthony, David Kotz • Dartmouth College

2. Motivation • Measuring wireless networks is useful • deployment, protocol development, application development, mobility modeling, etc... • So lots of people have done it • Network-side measurements of lots of WLANs: IBM, UNC, SIGCOMM, Georgia Tech, Stanford, UCSD, Dartmouth, ... • Network-side measurement only gets you so far • Tells you what happens, but not why it happens • To know why, need to ask wireless users themselves • But reliable polling can be difficult

3. The Experience Sampling Method • A psychological ‘diary method’ • subjects self-report their own experiences • potentially-fewer biases since no interviewer involved • signal subjects at random times and ask them to report • timely reports means users don’t forget their experiences • interval-contingent: subjects report at regular intervals • signal-contingent: subjects report when an event occurs • ESM reports: subjects fill out questionnaires • questionnaires ask about current activities, conditions, feelings • finite time period, e.g., seven signals per day over seven days

4. Experimental setup - users • Dartmouth WLAN • ≥560 Cisco/Aruba 802.11a/b/g APs • complete campus coverage • ongoing monitoring since 2001 (syslog, SNMP,tcpdump, CDR) • 30 subjects • all undergraduates, 15 male, 15 female • all owned wireless laptops • 23 Windows, 7 Macs • recruited via website • offered $100 each • subjects provided ‘conflict times’: times not to be signaled

5. Experiment setup - technical • 30 participants, 30 alphanumeric pagers, 30 notebooks • Motorola Bravo 406-512MHz pagers • considered PDAs or 802.11 devices • but wouldn’t work off-campus or when network down • Notebook: signal- and event-contingent questionnaires • different questionnaires depending on type of alert • questions incl: devices/applications in use, location, current task • 7 day study, 7 signal-, ≤3 event-contingent alerts/day • signal-contingent alerts at random times (excluding conflicts) • minimum 45-minute interval between alerts • monitored subjects’ wireless MAC addresses and sent event-contingent alerts if certain events occurred • high throughput, high errors, ping-ponging, association with busy AP

6. Wireless vs overall mobility • Users were paged at up to 6 locations in a day, but were only on WLAN at ≤ 2 • Do we need better coverage?

7. Usage locations • Wireless used more in homes, libraries % of alerts at location

8. Applications in use • Unsurprisingly, P2P was big bandwidth hog - our sample is representative of whole campus

9. Wireless vs wired behaviour • Wireless behaviour is multi-modal • when using wireless laptop, more than one device in use • More likely to be using cellphone and/or TV when on wireless • Fewer high-bandwidth activities (filesharing, streaming media) on wireless than on wired “ ” [Wireless] tends to be a little bit slower than wired... I download a lot of music so that can be frustrating.

10. Home vs non-home behaviour • Both home and non-home: • e-mail • work-related web-browsing • When at home: • P2P, IM • non-work-related web-browsing “ ” [Wireless] lets me move my study space to wherever I want it to be.

11. Communication patterns • Locality • 83% of email/IM is local • 86% of phone (analogue/cell/VoIP) is non-local • VoIP • 65% of subjects completely unaware of VoIP • VoIP users did not use cell phones, and vice versa • (but we have very little cellphone usage in general) “ ” If the quality and price of VoIP phones were the same as the cell phone, I think I would definitely use them.

12. Lessons learned • Dealing with subjects is hard work! • choose time carefully • avoid student examinations, vacations • allow time to collect info (MAC addresses etc.) • expect schedules to change • students change conflict times with 5-minutes notice... • one laptop broke and was replaced during the week • explain parameters clearly • one user went home (to a different state) but thought she could still be monitored on home WLAN • automatic participant pre-screening • we used face-to-face interviews • could monitor wireless activity prior to participant-selection

13. More lessons learned • Consider all event-contingent alerts • we thought it would be less intrusive to limit to 3/day • ended up with very few event-contingent alerts • ignore 45-minute alert interval • interval meant we ignored some interesting wireless events • Use a two-way signalling channel • lots of pages failed, but we didn’t know • no real-time feedback • users would complain, but only at the end of the day (when they hadn’t received sufficient pages) • two-way pagers? • 802.11 devices inappropriate

14. Related work • Lots of ESM studies in psychological literature • teacher job satisfaction • youth happiness • experiences of new parents • Communications-based ESM studies • Ubicomp - Intel Personal Server (Consolvo & Walker 2003) • Virtual Environments (Gaggioli et al. 2003) • Television-viewing habits (Kubey et al. 1986/1987/1990) • Most studies are signal- or event-contingent • Our study attempts to combine both

15. Summary • As well as measuring wireless networks, need to measure wireless network users • ESM provides a technique for measuring user experiences • allows us to see when users aren’t on the WLAN • allows us to see more detail about what users are doing • allows us to see usage of non-802.11 devices • ESM studies are difficult to set up • difficult to deal with users • Future work: larger study that benefits from the lessons learned in this study