400 likes | 411 Views
Explore the process and benefits of usability testing in library planning and assessment. Learn how to analyze study results and make recommendations for design improvements.
E N D
Got Data, Now What?Analyzing Usability Study Results Lynn Silipigni Connaway June 26, 2005 Presented at the ALA 2005 Annual Conference Chicago, IL LAMA/MAES Using Measurement Data for Library Planning and Assessment Committee
Usability Testing: Why? “Probably the best reason to test for usability is to eliminate those interminable arguments about the right way to do something. With human-factors input and testing, however, you can replace opinion with data. Real data tend to make arguments evaporate and meeting schedules shrink.” (Fowler, 1998, Appendix, p. 283)
Usability Testing: Definition • Degree to which a user can successfully learn and use a product to achieve a goal • Research methodology • Evaluation • Experimental design • Observation and analysis of user behavior while users use a product or product prototype to achieve a goal (Dumas and Reddish, 1993, p.22) • “User-centered design” process involving user from initial design to product upgrade (Norlin and Winters, 2002) • Approach is to be a servant to the users of a system NOT to be subservient to technology (Gluck, 1998) • Goal is to identify usability problems and make recommendations for fixing and improving the design (Rubin, 1994)
Usability Testing: Background • Relatively new methodology (Norlin and Winters, 2002) • Origins in aircraft design • Traced back to marketing • Development of a product • Popular in 1980s with widespread access to computers • Initiation of human computer interface usability studies • Evolved from human ethnographic observation, ergonomics, and cognitive psychology • Qualitative and quantitative data
Usability Testing: Purpose • Evaluation tool • Identify problem areas • “Determine the fit of the design to the intended users” (Norlin and Winters, 2002, p. 5)
Usability Testing: Suitable Questions • What is the best layout for a web page? • How can you optimize reading from PDAs and small screen interfaces? • Which online fonts are the best? • What makes an e-commerce site difficult to use? • Can individual personality or cognitive skills predict Internet use behavior? • How can library collection holdings and library data be represented geographically?
Usability Testing: Principles • Keep the end user in mind • Achieve superiority through simplicity • Improve performance through design • Refine and iterate (Norlin and Winters, 2002, p.10)
Usability Testing: Web Design Criteria • Links must be consistent and predictable • Group like things on the same page • Be consistent with language • Most important information should be on the first screen • Provide keywords for quick reading/scanning • Do not use animation or sounds • Make links look like links • Distinguish text from graphics • Avoid jargon (Spool, 1999)
Usability Testing: Web Design Criteria • Ten Usability Heuristics (Nielsen) • Visibility of system status • Match between system and the real world • User control and freedom • Consistency and standards • Error prevention • Recognition rather than recall • Flexibility and efficiency of use • Aesthetic and minimalist design • Help users recognize, diagnose, and recover from errors • Help and documentation
Usability Testing: Web Design Criteria • Goals for user-centered design • Enable users to • Achieve their particular goals and meet their needs • Move quickly and with few errors • Create a site that users like • More likely to perform well on a product that provides satisfaction
Usability Testing: Methodology • Artificial environment (laboratory) • Maintain more control • May provide more specific data on a particular feature • Natural environment • Better holistic representation of real people doing real work
Usability Testing: Methodology • Four types of usability tests (Rubin, 1994, p. 31-46) • Exploratory test – early product development • Assessment test – most typical, either early or midway in the product development • Validation test – verification of product’s usability • Comparison test – compare two or more designs; can be used with other three types of tests
Usability Testing: Methodology • Develop problem statements, objectives, and/or hypotheses • Identify and select participants who represent target population • May or may not be randomly selected • Select test monitor/administrator • Empathetic • Impartial • Good communicator • Good memory • Able to follow test structure • Able to react spontaneously to situations that cannot be anticipated • Allow user time for task • Don’t rescue the user • Continue with the plan if mistakes occur
Usability Testing: Methodology • Design test materials • Screening questionnaire • Provides user profile • Ascertains pretest attitudes and background information • Provides information about participants’ previous knowledge and experience • Orientation script • Describes the test to participants • Aids in understanding the participants’ performance • Data logger materials • Data collection instrument for categorizing participants’ actions • Can note time to match with videotape recording
Usability Testing: Methodology • Design test materials • Non-disclosure and tape consent forms for legal protection • Task list • List of actions participants will execute • Desired end results • Motives for performing task • Actual observations monitor will record • State of system
Usability Testing: Methodology • Design test materials • Posttest questionnaire • All participants asked the same questions • Gather qualitative information and precision measurements • Debriefing guide • Structure and protocols for ending the session • Participants explain things not apparent in actions • Motive • Rationale • Points of confusion
Usability Testing: Methodology • Test materials and equipment • Conduct the test • Represent the actual work environment • Users are asked to think aloud • Observe users while using or reviewing the product • Probe • Controlled and extensive questioning • Collect quantitative and qualitative data and measures • Record comments or questions about the product • Observe and document users’ behaviors
Usability Testing: Methodology • Debrief • Analyze the data • Diagnose and recommend corrections • Categorize and identify problems with the product • Identify solutions • Qualitative analysis • Textual notes from debriefing • Read responses • Summarize findings
Usability Testing: Methodology • Analyze the data • Quantitative analysis • Questionnaires • Screening • Posttest • Triangulation to validate findings • Data from questionnaires, observations, screen tracking software, comments, and open-ended questions
Usability Testing: Interpret Data • Interpret the data • Five factors for benchmarking the usability of an interface (Shneiderman and Plaisant, 2004) • Time to learn • Speed of performance • Rate of errors • Retention over time • Subjective satisfaction
Usability Testing: Interpret Data • Interpret the data • Prioritize severity of problems • Severity ratings (Zimmerman and Akerelrea, 2004) • Time required to complete task • Number of users who encountered problem • Negative impact on users’ perception of the product • Difficult if 70% of users cannot perform task • Error criticality = Severity + Probability of Occurrence (Rubin, 1994)
Usability Testing: Interpret Data • Usable Web site: (Rubin, 1994) • Usefulness • Establish whether it does what the user needs it to do • Effectiveness • Ease of use to achieve the desired task • Learnability • Ease of learning application and moving from being a novice to a skilled user • User satisfaction • User’s attitude about the site—how enjoyable it is to use
Usability Testing: Report Results • Executive summary • Report • Describe methodology • Who, what, when, where, and how • Describe how tests were conducted • Profile users and describe sampling • Detail data collection methods • Succinctly explain the analysis • Provide screen captures • Include tables and graphs • Provide examples • Identify strengths and weaknesses • Recommend improvements
Usability Testing: Making the Data Work • Read report • Determine what worked and what did not work • Redesign product/system based upon findings • May be necessary to conduct another usability test
Usability Testing: Limitations • Two major limitations (Wheat) • Reliability • Testing of users who are not atypical users • Individual variation within the test population • Validity • Test tasks, scenarios of the search processes, and testing environment are not accurate • Results not generalizable to the entire user population • Testing is always artificial (Rubin, 1994, p.27)
OCLC WorldMapTM • Research prototype • Test geographical representation of WorldCat holdings • By country and date of publication • For library collection assessment and comparison • Complement the AAU/ARL Global Resources Network project • Geographically represent library statistical data from UNESCO, ARL, Bowker, and others • Number of libraries by type • Expenditures by library type • Number of volumes and titles • Number of librarians • Number of users
Usability Testing: OCLC WorldMapTM • Review sample handouts • Screening questionnaire • Task list • Posttest questionnaire • Executive summary
Usability Testing: OCLC WorldMapTM • Conducted informal usability tests • Currently redesigning the interface • Conduct second group of formal usability tests • Make revisions prior to making publicly available
Questions and Discussion connawal@oclc.org