High-fidelity or Low-fidelity, Paper or Computer? Choosing attributes when testing web prototypes

1. Group for User Interface Research University of California Berkeley High-fidelity or Low-fidelity, Paper orComputer?Choosing attributes when testing web prototypes Miriam Walker Leila Takayama Professor James Landay

2. Outline • Motivation • Experimental testing of prototypes • Results and conclusions “You can fix it now on the drafting board with an eraser, or you can fix it later with a sledgehammer” Frank Lloyd Wright HFES 46th Annual Meeting

3. Practical prototyping • Prototypes as tools for design • “Track changes” is easier on computer than paper • Designers invest less time and ego in low-fidelity prototypes • Prototypes as tools for usability testing • Computer prototypes allow remote testing • Computer prototypes have more realistic interactions HFES 46th Annual Meeting

4. Website prototyping tools • Prototype functionality depends on tools • Prototyping tools can be • Low-fidelity or high-fidelity • Paper medium or computer medium Research and practical considerations should drive the selection of prototyping tools HFES 46th Annual Meeting

5. Dreamweaver: Familiar high-fidelity, computer prototyping tool HFES 46th Annual Meeting

6. Paper: Familiar low-fidelity, paper prototyping tool Post-its Scissors Pens Tape Transparencies Rulers Cardboard Foam-core (e.g. Rettig, 1994) HFES 46th Annual Meeting

7. DENIM: A low-fidelity, computer prototyping tool (Lin, Newman, Hong, & Landay, 2000) HFES 46th Annual Meeting

8. Why would fidelity and medium affect user testing? • Fidelity and medium change interaction • Example: text-entry is handwritten or typed • Colors in high-fidelity direct attention • Fidelity and medium may alter the users’ views on: • Functionality of prototype • Causes of and solutions for usability problems • Ability of users to have an impact on design (Hong et al, 2001) HFES 46th Annual Meeting

9. Outline • Motivation • Experimental testing of prototypes • Results and conclusions HFES 46th Annual Meeting

10. Making early stage prototypes Sketched with paper and pens Scanned in paper pages Printed screens Coded in HTML HFES 46th Annual Meeting

11. Low-Fidelity Prototype HFES 46th Annual Meeting

12. High-Fidelity Prototype HFES 46th Annual Meeting

13. Experimental design • Participants were unaware of the experimental hypotheses • Each participant saw either low-fidelity or high-fidelity websites on both paper and computer HFES 46th Annual Meeting

14. Testing Methods • Faked prototype functionality by constraining tasks • Sign-up for online banking services • Email a checking account statement • Calculate value of foreign currency • … • Asked participants to think aloud • Recorded their comments, and took copious notes • Gave participants minimal assistance • Followed up user tests with more questions HFES 46th Annual Meeting

15. Outline • Motivation • Experimental testing of prototypes • Results and conclusions HFES 46th Annual Meeting

16. Analysis method:effective usability testing • More problems - six comments on one issue vs. one comment on each of six issues • Most severe problems • All types of problems e.g. consistency, feedback • Level of detail - information architecture problems, widget problem HFES 46th Annual Meeting

17. comment comment comment comment comment comment Quantitative analysis process • Ratings: • Issue severity • Issue heuristic category (Nielsen, 1994) • Comment scope • (widget, page, website) • Counts: • Issue • Comments issue issue issue Quantitative statistical analysis HFES 46th Annual Meeting

18. Issues Issue: confusion between scheduled, single, and recurring on bill payer Comments: “I would like recurrent payments…no scheduled. I don’t see the point, I don’t see the difference between these two but um..” “Payment. Oh, actually, it would be recurring. I’m trying to decide if it’s a scheduled payment or recurring payment.” “Oops. Recurring. Then the single payment would be like a scheduled? How would… I’m just trying to figure out what the difference would be between the two.” HFES 46th Annual Meeting

19. Quantitative Analysis of Results • 1270 comments and 169 issues • Low-fidelity vs High-fidelity • No significant differences in number of comments or issues • Paper vs. computer • Average of 5 more comments about computer prototype (Wilcoxon signed ranks test, p = 0.015) • Issues – no significant difference HFES 46th Annual Meeting

20. Results • No differences in severity of issues found • No differences in scope of issues • Differences between fidelities but not media categorizing issues by Nielsen’s heuristics (Chi Squared, p<0.01) • Only 10% of comments mentioned aesthetics • Classifying issues using Nielsen’s Heuristics is difficult HFES 46th Annual Meeting

21. Conclusion Fidelity and medium do not seem to affect quantity of problems found by user testing Prototyping techniques should be chosen by considering: • Need for remote testing • Importance of recording design process • Keeping designs at a level of detail appropriate to the stage of design HFES 46th Annual Meeting

22. University of California Berkeley Group forUser Interface Research Special thanks to: Corey Chandler Jason Hong, James Lin, and Francis Li Sarah Waterson Professor Rashmi Sinha Ten anonymous expert raters Miriam Walker - mwalker@cs.berkeley.edu Leila Takayama - leila@cs.berkeley.edu http://guir.berkeley.edu/projects/fidelity HFES 46th Annual Meeting

23. Question Time! HFES 46th Annual Meeting