Download
1 / 107
1.07k likes | 1.19k Views
An Introduction to Usability Testing. Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com. Background. Origin of the Species.
E N D
An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design www.user-centereddesign.com
Background User-Centered Design www.user-centereddesign.com
Origin of the Species • “Usability testing” is the common name for multiple forms both user and non-user based system evaluation focused on a specific aspect of the design • Done for many, many years prior, but popularized in the media by Jakob Neilson in the 1990’s User-Centered Design www.user-centereddesign.com
What does “usability” mean? ISO 9126 “A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users” ISO 9241 “Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” User-Centered Design www.user-centereddesign.com
The Ontology of “Usability” Accessibility A precursor to usability: if users cannot gain access to the product, its usability is a moot point Functional Suitability Does the product contain the functionality required by the user? Functional Discoverability Can the user “discover” the functions of a product? Ease-of-learning Can the user figure out how to exercise the functionality provided? Ease-of-use Can the user exercise the functionality accurately and efficiently once its learned (includes accessibility issues)? Can users use it safely? Ease-of-recall Can the knowledge of operation be easily maintained over time? Safety Can the user operate the system in relative safety, and recover from errors? Subjective Preference Do user’s like using it? User-Centered Design www.user-centereddesign.com
Usability, Organizations, and Process User-Centered Design www.user-centereddesign.com
Thought From CHI ‘92 • The 1970s, when Hardware is King • 1950s – its an art • 1960s – there are degrees • 1970s – they’re in management • The 1980s, when Software is King • 1960s – its an art • 1970s – there are degrees • 1980s – they’re in management • 1990s, when Interaction will be King • 1970s – its an art • 1980s – there are degrees • 1990s – they’re in management User-Centered Design www.user-centereddesign.com
Processes • System Development Models • Waterfall • Spiral • V-Model • Software Development Models • Dynamic System Development Process (DSDP) • Joint Application Development Process (JAD) (circa 1970) • Structured Systems Analysis and Design Methodology (SSADM) (circa 1980) • Information Requirement Analysis/Soft System (circa 1980) • Object Oriented Programming (origins in 1960, but a common methodology in the 1990s) • Rapid Application Development (circa 1991)* • Agile* • Extreme Programming (circa 1990) • SCRUM User-Centered Design www.user-centereddesign.com
Processes (concluded) • Interface Design Models • User-Centered Design (the common term) • Star (Hartson & Hix, 1989) • LUCID (Cognetics, 2008) • ISO 13407/ISO 9241 • Design Thinking (aka Human Centered Design) (IDEO) • Characteristics of a User-Centered Design Process • Design is a separate activity, distinct from development • Design should occur, completely, before development begins • Feedback is needed at many steps in the design process to… • Confirm the direction of design • Evaluate alternatives • User-Centered Design techniques can also be used to test the outcome (the final product) under the correct conditions User-Centered Design www.user-centereddesign.com
Corporate Organization Structure C-Level Management (CEO, CFO, CIO, CTO, CPO) Marketing Sales Product Design & Development Training Field Services R&D User-Centered Design www.user-centereddesign.com
Product Design & Development Systems Engineer (Management) R&D Design Team Development Team Industrial Design Visual Design Interaction Design Technical Writers Test & Evaluation Mechanical Engineering Electrical Engineering Software Engineering & Web Development User-Centered Design www.user-centereddesign.com
Testing Methods Part 1: Non-User Based Testing User-Centered Design www.user-centereddesign.com 12
Compliance Testing • The Spelling and Grammar checker of usability testing • Possible (within limits) to be performed by anyone • Can remove the low level usability issues that often mask more significant usability issues User-Centered Design www.user-centereddesign.com
Compliance Testing (concluded) • Style Guide-based Testing • Checklists • Interpretation Issues • Scope Limitations • Available Standards • Commercially GUI & Web Standards and Style Guides • Domain Specific GUI & Web Standards and Style Guides • Internal Standards and Style Guides • Interface Specification Testing* *Special Case of QC Testing that assumes a usable design to start with User-Centered Design www.user-centereddesign.com
Expert Review • Aka: Heuristic Evaluation • One or more usability experts review a product, application, etc. • Free format review or structured review • Subjective but based on sound usability and design principles • Highly dependent on the qualifications of the reviewer(s) User-Centered Design www.user-centereddesign.com
Expert Review (Concluded) • Nielson’s 10 Most Common Mistakes Made by Web Developers (three versions) • Shneiderman’s 8 Golden Rules • Constantine & Lockwood Heuristics • Forrester Group Heuristics • Norman’s 4 Principles of Usability User-Centered Design www.user-centereddesign.com
1st Heuristic Functional discoverability through obvious interactive elements and adequate feedback User-Centered Design www.user-centereddesign.com 17
2nd Heuristic A good, complete, and unambiguous cognitive (or conceptual) model to predict the effects of our actions User-Centered Design www.user-centereddesign.com 20
Cognitive Models • We all develop cognitive models • They may not be complete • They may be inconsistent • They ay be self contradicting • They are not always correct • We don’t necessarily invest in maintaining our cognitive models User-Centered Design www.user-centereddesign.com
3rd Heuristic Design for the intended users (and not yourself) User-Centered Design www.user-centereddesign.com
4th Heuristic Design for Errors (Slips) User-Centered Design www.user-centereddesign.com 35
Error versus Slip • Errors are generated by a lack of understanding or a lack of sufficient or correct information • Lack of sufficient or correct information is the responsibility of the designer in the presentation layer of an interface • Lack of understanding is the responsibility of the designer in interaction and in conceptual model of an interface • Errors are often undetectable by the end user • Slips are common users issues • Hand/eye coordination or basic control of our psychomotor systems • Exacerbated by distraction, speed, attention overload • Unavoidable by design but need to be anticipated and addressed by the designer User-Centered Design www.user-centereddesign.com
Others • Cognitive Walkthrough • Specific review to ensure the correct information is available for the task being performed • Also low cost usability testing • Highly dependent on the qualifications of the reviewer(s) • Pluralistic Walkthrough • Team Approach • Best if a diverse population of reviewers • Issues related to cognition (understanding) more than presentation • Also low cost usability testing • Highly dependent on the qualifications of the reviewer(s) User-Centered Design www.user-centereddesign.com
Testing Methods Part 2: User Based Testing User-Centered Design www.user-centereddesign.com 38
Statistics: A Primer User-Centered Design www.user-centereddesign.com 39
Some Principles • Research to used to test a hypothesis based on a theory • Smoking increases the likelihood of developing cancer • Testing is used to support a decision • For example, “this design change is going to be better for users”, or “design A is better than design B” • Statistics are used to provide a way relate the small sample tested to the larger population, but small is a relative term • 25-30 is considered minimal before you see regression to the mean • Statistical analysis assumes the data obtained is valid and reliable User-Centered Design www.user-centereddesign.com 40
Validity • Validity is the degree to which the results of a research study provide trustworthy information about the truth or falsity of the hypothesis* • Internal validity refers to the situation where the “experimental treatments make a difference in this specific experimental instance” (from Cambell, D.T. & Stanley, J.C. (1963) Experimental and Quasi-experimental Designs for Research • External validity asks the question of “generalizability” *Cherulnik, P.D. 2001. Methods for Behavioural Research: A Systematic Approach User-Centered Design www.user-centereddesign.com 41
Reliability • Reliability is the ability of a test to show the same results if conducted multiple times • Test-retest reliability • Repeatability • Reproducibility User-Centered Design www.user-centereddesign.com 42
Use of Confidence Intervals • When working with small samples, confidence interval provide a way to represent uncertainty in test results • Since each sample and each test is different, the confidence level tells the informed reader the likelihood that another sample will provide the same results. (In other words, if you ran the test again, what value are you likely to get next time?) • Typical confidence intervals in research include the 90% or 95% confidence interval. Behavioural research often uses a 80% confidence interval. User-Centered Design www.user-centereddesign.com 43
Use of Confidence Intervals (continued) • “You just finished a usability test. You had 5 participants attempt a task in a new version of your software. All 5 out of 5 participants completed the task. You rush excitedly to tell your manager the results so you can communicate this success to the development team. Your manager asks, ‘OK, this is great with 5 users, but what are the chances that 50 or 1000 will have a 100% completion rate?’ ”- Jeff Sauro • The confidence level tells the informed reader the likelihood that another sample will provide the same results. In other words, if you ran the test again, what value are you likely to get next time? User-Centered Design www.user-centereddesign.com 44
Use of Confidence Intervals (continued) • Usability is typically done with very few people per round • Neilson says 5 (but not for the right reason) • Krug says 2 or 3 (also not for the right reason) • 3 per user group, profile, or persona is considered a minimum by convention and ISO standard, a day consisting of about 8-9 people • You could do statistical analysis on the results of a typical usability if… • Your test as valid and reliable • You had truly random sampling • You did not interfere with performance during testing User-Centered Design www.user-centereddesign.com 45
Use of Confidence Intervals (concluded) • Confidence intervals when testing with, say, 8 people range from 37% (0 out of 8 or 8 out of 8) to between 50%-70% (all other values) • For example, if 6 out of 8 people successfully completed a task in your test, you can only predict that somewhere between 20% and 97% of all people would complete the task (assuming all conditions for validity and reliability have been met) • If you want to confidently state, based on your testing, that 9 out of 10 people will be able to successfully complete a task, and all conditions needed for validity and reliability have been met, you need to test 430 people and 400 of them have to successfully complete the task User-Centered Design www.user-centereddesign.com 46
The Psychology of Usability User-Centered Design www.user-centereddesign.com 47
Attention • Highly Limited • Attenuator Model • Switching Model • But attention is conscious attention, we have non-conscious attention 48
