Exploring Mobile Usability Testing Methods for Enterprise Users: A Case Study

1. Exploring Mobile Usability Testing Methods for Enterprise Users: A Case Study Minmin Yang & Judy Keeley Boston Mini UPA Conference May 26, 2009

2. Outline Background Testing approach In-person Remote Testing equipment In-person Remote Findings about our mobile testing method 2 2

3. Background Usability testing on Verizon Enterprise Center (VEC) mobile portal before release to customers Portal for enterprise customers to manage services from a mobile phone Research & preparation Reviewed mobile testing literature for selection of test approach Researched test equipment Acquired test equipment Tested selected method (approach & equipment) 3 3

4. Testing Approach 4

5. What the research says… Lab testing vs. field testing on mobile devices Kjeldskov et al. (2004) & Kaikkonen et al. (2005) Field testing added little to usability evaluations Lab testing seems to be sufficient when searching user interface flaws to improve user interaction Nielsen et al. (2005) Field testing revealed significantly more usability problems than lab testing overall Field testing revealed problems with interaction style and cognitive load that were not identified in the laboratory Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. (2004). Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Proceedings of the 6th International Mobile HCI 2004 conference. LNCS, Springer-Verlag. Kaikkonen, A., Kekalainen, A., Cankar M., Kallio, T., & Kankainen A. (2005). Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies, 1(1), 4-16. Nielsen, C. M., Overgaard M., Pederson M. B., Stage J., & Stenield S. (2006). It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. In Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. Oslo, Norway. 5

6. What the research says… Field testing vs. lab testing on mobile phone vs. lab testing with emulator Betiol & Cybis (2005) All approaches revealed more similarities than differences in terms of results Lab testing with emulator could reveal a large percentage of usability problems Lab Testing on Mobile Phone Lab Testing with Emulator 3 2 0 32 4 0 2 Field Testing with Wireless Camera Betiol, A. H., & Cybis, W. A. (2005), Usability Testing of Mobile Devices: A Comparison of Three Approaches. International Federation for Information Processing, 470 – 478.

7. Testing Approach In-Person Testing Remote Testing 7

8. In-Person: Field Testing? Pros: Provides highest validity Reveals more usability problems compared to lab testing (Nielsen et al., 2006) Cons: Equipment is expensive Hard to find participants for field testing Difficult to find situations where following participants is appropriate Field Testing 8

9. In-Person: Emulators? Pros: Emulators are free Testing with emulators can reveal a large percentage of the usability problems (Betiol & Cybis, 2005) Cons: Emulators don’t duplicate experience of using actual phone Lab Testing with Emulator 9

10. In-Person: Lab Testing on Mobile Phone? Lab testing on mobile phone is sufficient in identifying user interface issues (Kaikkonen et al. 2005) Compared to field testing Cheaper to set up equipment Easier to conduct testing Compared to using emulators: Can provide a more authentic experience Decision: use lab testing on mobile phone for in-person testing Lab Testing on Mobile Phone

11. Remote: Which Approach? Remote field testing: not practical Remote testing with participants’ own mobile phone: Expensive: webcam or sled need to be purchased and shipped to participants Troublesome: participants need to set up equipment and install software themselves; Remote testing with emulators: No cost: emulators are free Much less troublesome: no installation on participants’ end via web conferencing Decision: use emulators for remote testing Remote Testing with Emulator 11

12. In-Person Test Equipment

13. In-Person Test Equipment Fixed lab Could use existing lab in nearby office, or set up a lab in local office Participants must come to lab for testing Portable lab More convenient for participants - Lab can travel to them Decision: Use portable lab Fixed lab Portable lab

14. In-Person Test Equipment: Recording How to record: Phone screen Screen capture software or video camera? User interaction with phone Typing on keyboard Holding phone Audio

15. In-Person Test Equipment: Recording Screen capture software (short list)

16. In-Person Test Equipment: Recording Blackberry Screen Capture Software BBScreenStream http://www.judykeeley.com/portfolio/presentations/UPA-2009/screenstream.avi 16

17. In-Person Test Equipment: Recording Screen capture software Advantages Records screen clearly Cheap software available Disadvantages Skips typed characters on Blackberry Doesn’t work with password-protected Blackberry phones Doesn’t capture user’s interaction with phone (requires camera)

18. In-Person Test Equipment: Recording Video Camera Advantages Records users interaction with phone Records audio along with video Disadvantages Need to purchase Harder to clearly capture phone screen Decision Use video camera instead of screen capture software

19. In-Person Test Equipment: Recording How to use the video camera for recording? Video camera fixed to desk Video camera mounted to phone Video camera mounted to participant’s head Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, Schusteritsch R., Wei C., & LaRosa M.(2007).CHI '07 extended abstracts on Human factors in computing systems, pp. 1839-1844. Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03

20. In-Person Test Equipment: Recording Device-mounted Camera mounted on mobile device or sled Pro: Camera moves with device, so device stays in video frame Cons: Makes phone heavier; may feel cumbersome to participant Buying sled is expensive Takes time to make custom sled Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03

21. In-Person Test Equipment: Recording Subject-mounted Camera mounted on participant’s head or eyeglasses Records participant’s field of view Pro: Participants can walk around test area while using device Cons: Camera on head might feel awkward or annoying Device lost from video frame when participant turns head Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03 21

22. In-Person Test Equipment: Recording Environment-mounted Camera attached to fixed spot (e.g. desk) Pro: Simplest and easiest to set up Con: Must keep phone in fixed location Decision: Use environment-mounted camera Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03

23. Logitech webcam Model: QuickCam Pro for Notebooks HD Video: 960 x 720 pixels 30 frames per second Built-in microphone to record sound Webcam attached to lamp neck and base Laptop with Logitech webcam software Cost: $90 for webcam In-Person Equipment: Our Portable Lab

24. In-Person Equipment: Our Portable Lab

25. In-Person Testing: Webcam Demo http://www.judykeeley.com/portfolio/presentations/UPA-2009/webcam.wmv

26. Remote Test Equipment

27. Remote Test Equipment: Emulators All remote participants had Blackberry phones Two had BB Curve, two had 8830, and one had Storm Some emulator choices: Blackberry Storm (touchscreen) Blackberry Curve Blackberry 8830

28. Remote Test Equipment: Emulators http://www.judykeeley.com/portfolio/presentations/UPA-2009/emulators.avi

29. Remote Test Equipment: Emulators This is a test; click the phone below http://www.judykeeley.com/portfolio/presentations/UPA-2009/emulators.avi

30. Remote Test Equipment: Emulators Goal: select emulators similar to participants’ phones but also easy for participant to control Performed dry run testing on 3 emulators Blackberry Curve Cursor moves pixel by pixel - painfully slow and awkward Blackberry 8830 On older models, cursor jumps from link/field to link/field - Faster and easier to control Selected for participants already using Curve & 8830 Blackberry Storm Touchscreen - Easy to control Selected for participant already using Storm Blackberry 8830 Blackberry Storm

31. Remote Test Equipment: Record/Connect Recording/Connecting software (short list) 31

32. Remote Test Final Equipment Blackberry Emulators 8830 and Storm Webex net conference to share emulator with participant Phone conference (webex dials in) Webex records both audio and video Cost: $0

33. Findings about Our Usability Testing Method

34. Findings - 1 In-Person testing Environment-mounted recording approach is good overall Video quality excellent - phone screen can be seen clearly Equipment simple and easy to carry around Issue Moderator needed to remind the participant whenever the phone was far away from the camera, which was not very frequent, but when it happened, it did interfere the performance a little bit

35. Findings - 2 Remote Testing Webex + emulator approach for remote testing worked out very well The video quality was good and the interface can be seen clearly The video size was pretty small vs. webcam Issues Video conversion quality – Webex arf format was converted to wmv; quality degraded with conversion and further saves Participants needed quick training session before using emulator

36. Findings - 3 Remote: 31 Problems In-Person: 48 Problems Mean = 12 In-person testing (M=16) could reveal more usability problems than remote testing (M=9), t(7) = 5.52, p = .0009 36

37. Findings - 3 In-person Remote 28% 14% 58% Based on our testing (sample size: 9) • Remote testing revealed 42% of usability problems • Remote testing revealed problems not found in in-person testing 37

38. Thank You! • Questions? • Contact us: • yangminmin22@gmail.com • judy@judykeeley.com