1 / 42

Take a web-site Take N professional usability teams Let each team usability test the web-site

This presentation discusses the results of large-scale web usability testing and compares the findings from different professional teams. It also provides insights into improving user experience and highlights common problems found during testing.

lisk
Download Presentation

Take a web-site Take N professional usability teams Let each team usability test the web-site

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Take a web-site Take N professional usability teams Let each team usability test the web-site Are the results similar?

  2. Practical Results from Large-Scale Web Usability Testing Rolf Molich DialogDesign

  3. http://www.dialogdesign.dk/cue2.htmSlides in Microsoft PowerPoint 97 and Adobe Acrobat formatThis slide will reappear at the end of this presentation Download Test Reports and Slides

  4. How It All Started... A recent survey shows that • 80% of all Danish drivers think that their driving skills are above average.

  5. How It All Started... A recent survey shows that • 80% of all Danish drivers think that their driving skills are above average. How about usability testers?

  6. How It All Started... • Too much emphasis on one-way mirrors and scan converters • Little knowledge of REAL usability testing procedures - mainly beautified descriptions • ”Who checks the checker?”

  7. ”Who Checks the Checker?” • When did YOU last have an objective check of your usability testing skills? • Who would you trust as an evaluator of your usability testing skills?

  8. Comparative Evaluations Test End Test object Student Professional teams teams 1 Oct 97 9 Danish web-sites 50 0 2 Oct 98 9 Danish web-sites 50 0 3 Dec 97 CUE-1: Win Calendar Progr. 0 4 4 Dec 98 CUE-2: www.hotmail.com 2 7 All results point in the same direction.

  9. Student Tests • Introductory couse in Human-Computer Interaction at the Technical University of Copenhagen • Two courses, 120 students per course • Fifty teams of one to three students • 2 x 9 Danish web-sites tested by four to nine teams with at least four test participants • Quality of Usability tests and reports is acceptable considering that most teams used 20-50 hours

  10. www.bokus.com - Bookstore Buttons in lower right corner: • Empty shopping basket • Change order • Continue shopping • Go on with your purchase

  11. Conclusions • Inhuman treatment of users on many e-commerce web-sites • On-site searching seldom works. Users are better off without on-site searching • Many web-sites focus on the company, not the user -

  12. Conclusions • Nice layout and graphics • Good response time • Give correct results +

  13. Problem Example User task: • You want to take your business to BG Bank. Make an appointment with the bank • Hard to find in menu structure • Users entered ”appointment” as keyword for Search

  14. How to Improve Search • Provide human error messages (constructive) • Recommend index, site-map • Special handling of frequent keywords • Show user search keywords in context

  15. CUE-1Comparative Usability Evaluation 1 • Four professional teams usability tested the same Windows calendar program • Two US teams (Sun, Rockwell), one English (NPL) and one Irish (HFRG, Univ. Cork) • Results published in a panel and a paper at UPA98 • Main conclusions similar to CUE-2

  16. CUE-2Comparative Usability Evaluation 2 • Nine teams have usability tested the same web-site • Seven professional teams • Two student teams • Four European, five US teams • Test web-site: www.hotmail.com

  17. Purposes of CUE-2 • Investigate the reproducibility of usability test results • Survey the state-of-the art within professional usability testing of web-sites.

  18. NON Purposes of CUE-2 • To pick a winner • To make a profit

  19. Usability Test Procedure • Web-site address (www.hotmail.com)disclosed at start of three week test period. • Client scenario (written by Erika Kindlund and Meeta Arcuri) • Access to client through intermediary (Erika Kindlund) • Three weeks to carry out test using standard approach • Deliver anonymized usability test report

  20. Problems Found CUE-1 CUE-2 • Total number of problems 141 300 • Found by seven teams - 1 • six teams - 1 • five teams - 4 • four teams 1 4 • three teams 1 15 • two teams 11 49 • Found only by one team 128 (91%) 226 (75%)

  21. CUE-2 Credits • Barbara Karyukina, SGI (USA) • Klaus Kaasgaard & Ann D. Thomsen, KMD (Denmark) • Lars Schmidt and others, Networkers (Denmark) • Meghan Ede and others, Sun Microsystems, Inc., (USA) • Wilma van Oel, P5 (The Netherlands) • Meeta Arcuri, Hotmail, Microsoft Corp. (USA) (Customer) • Rolf Molich, DialogDesign (Denmark) (Coordinator)

  22. CUE-2 Credits • Joseph Seeley, NovaNET Learning Inc. (USA) • Kent Norman, University of Maryland (USA) • Torben Norgaard Rasmussen and others, Technical University of Denmark • Marji Schumann and others, Southern Polytechnic State University (USA)

  23. Resources Team A B C D E F G H J Person hours used for test 136 123 84 (16) 130 50 107 45 218 # Usability professionals 2 1 1 1 3 1 1 3 6 Number of tests 7 6 6 50 9 5 11 4 6

  24. Usability Test Reports Team A B C D E F G H J # Pages 16 36 10 5 36 19 18 11 22 Exec summary Y Y N N N Y N Y Y # Screen shots 10 0 8 0 1 2 1 2 0 Severity scale 2 3 2 1 2 1 1 3 4

  25. Usability Results Team A B C D E F G H J # Positive findings 0 8 4 7 24 25 14 4 6 # Problems 26 150 17 10 58 75 30 18 20 % Exclusive 42 71 24 10 57 51 33 56 60

  26. Usability Results Team A B C D E F G H J # Problems 26 150 17 10 58 75 30 18 20 % Core problems (100%=26) 38 73 35 8 58 54 50 27 31 Person hours used for test 136 123 84 NA 130 50 107 45 218

  27. Results from All Four Studies • There are overwhelmingly many usability problems. • Many of them are serious. • Limited overlap between team findings.

  28. Conclusions • In most cases, no form of cost-effective testing will find all or most of the problems - or even most of the serious ones • Claims like ”Method x finds at least 80% of all serious usability problems” are not in accordance with the results of this study

  29. Problems Found in CUE-2 • Total number of different usability problems found 300 • Found by seven teams 1 • six teams 1 • five teams 4 • four teams 4 • three teams 15 • two teams 49 • Found only by one team 226

  30. Problem Found by Seven Teams During the registration process Hotmail users are asked to provide a password hint question. The corresponding text box must be filled. Most users did not understand the meaning of the password hint question. Some entered their Hotmail password in the Hint Question text box. Clever but unusual mechanisms like the password hint question must be explained carefully to users.

  31. Language Related Problems Examples of language related problems that were found by European teams

  32. Language Related Problems Examples of language related problems that were detected by European teams • Send Mail: Term "Compose" difficult to understand. Use "Create new message" or "Write Mail” (5/9) • Create new account: "State/Province" textbox is required but does not make sense in many countries (2/9)

  33. Language Related False Problems Some language related problems suggested by an US team were not confirmed by European test teams • Change "last name" to "family name” • Meaning of "U.S. Residents only" and "Non-U.S. Residents Only" is unclear

  34. Advice for a Usable Usability Report • List problems with severity, #users • Provide short executive summary • Keep it short • Distinguish clearly between • Personal opinions, • Expert opinions, • User opinions, • User findings

  35. Some State-of-the-Art Boundaries • No power user test, although four teams also recruited power users • Few tests that require complicated setup. Examples: Attachments; boundary testing, e.g. large number of e-mails in in-box • Teams completed their usability tests within schedule, but they hesitated to compared their results to those from the other teams

  36. Conclusions • The total number of usability problems for each tested web-site is huge,much larger than you can hope to find in one series of usability tests • Usability testing techniques can be improved • We need more awareness of the usability of usability work

  37. http://www.dialogdesign.dk/cue2.htmSlides in Microsoft PowerPoint 97 formatCUE-2 Panel:Tuesday at 4.30 p.m. Download Test Reports and Slides

More Related