Usability…and Beyond!

1. Usability…and Beyond! Understanding Usefulness, Usability & Use CHI 2004 Tutorial April 2004 Diane J. Schiano <Abbreviated for PARC> CHI2004 Tutorial

2. Table of Contents • Page • Introduction 15 • A Primer on User Exp Research Design 30 • ---Core UE Research Design Principles 37 • ---Key Pragmatic Issues 56 • Methods, Measures & More 67 • ---Key Research Design Decisions 68 • ---Methods: Self-Report & Observational 69 • ---Measures & More: Quantitative, Qualitative 128 • More On Communicating Results 148 • Appendix 155 CHI2004 Tutorial

3. Course Learning Objectives • This tutorial will provide a general understanding of: • Usefulness, usability and use studies: what they are, how and why they are done, and the kinds of information they yield, with extended examples and resources available in the tutorial notes. • User research design principles, and the pragmatic challenges of attaining validity, reliability, generalizability and robustness in user studies. • The major self-report and observational methods available for user studies, and how to choose and apply them appropriately. • Approaches to dealing with qualitative, quantitative and hybrid data, and to summarizing results and making design recommendations from them • Pragmatic considerations in applying procedures, discussed from logistical, organizational, and personal/privacy perspectives • Also provided: • Guidelines, readings, other resources for conducting studies and evaluating findings, both in class and in the tutorial notes • Opportunities for expert and peer feedback on practical problems participants are invited to bring in for consideration during the User Research Design Clinic. CHI2004 Tutorial

4. Abstract Digital products are growing increasingly complex, encompassing interactions among humans as well as between humans and technology. Careful methods are required to understand user experience of these products, and to use this understanding to inform iterative design. The goal of this tutorial is to provide a practical understanding of the principles and procedures used to assess product usefulness, usability and use. Participants are given guidance and grounding in general user research design principles and procedures to aid them in choosing methods, conducting studies, evaluating results, and making recommendations effectively, even under constrained conditions. A principled yet pragmatic approach is advocated, consolidating the best of classic usability engineering and ethnographic methods—and applied appropriately for the research question and context at hand. Useful exercises, extended examples and extensive references and other resources are provided. Finally, attendees are invited to bring actual problems to discuss in the ‘User Research Design Clinic’ for peer and expert feedback. CHI2004 Tutorial

5. Usability…and Beyond! Understanding… Usefulness, Usability & Use CHI2004 Tutorial

6. Usability…and Beyond! Understanding… User Experience! Focusing on the Principles Underlying the Procedures CHI2004 Tutorial

7. Introduction CHI2004 Tutorial

8. Creating Useful, Usable & Used Products… • Depends critically on effective User Research. • “If we build it they will come… NOT!!!” • What’s the use… of designing products that aren’t… • Useful, • Usable, • Used???!!! CHI2004 Tutorial

9. Core UserExperience Research Concepts • Usefulness • Why--and how--could the product be useful to people? Design (& marketing) implications from current practice? • Usability • How easily--and well--can the product be learned and used? Implications for re-design? • Use • How do people actually use the product? Implications for re-design? CHI2004 Tutorial

10. User-Centered Product Research& Design Usability Usefulness --> Use <----------------------> From UsabilityNet CHI2004 Tutorial

11. Usefulness, Usability, Use in Product Cycle Usefulness Usability Requirements Implementation Test + Measure Planning + Feasibility Design Post-Release Use CHI2004 Tutorial

12. Usefulness, Usability, Use in Product Cycle Usability Use Usefulness From UsabilityNet CHI2004 Tutorial

13. We’ll Focus on Research Design Principles • Primarily the WHYs behind choosing and implementing user research methods. • For more on HOW-TOs, see… • UsabilityNet (http://www.usabilitynet.org) • Appendix references & recommendations CHI2004 Tutorial

14. UsabilityNet: An Excellent Resource! • High Quality, Free • How-Tos, Mini-Tutorials • http://www.usabilitynet.org CHI2004 Tutorial

15. Core UserExperience Research Concepts • Usefulness • Why--and how--could the product be useful to people? Design (& marketing) implications from current practice? • Usability • How easily--and well--can the product be learned and used? Implications for re-design? • Use • How do people actually use the product? Implications for re-design? CHI2004 Tutorial

16. Two Classic Approaches to UE Research • Ethnography (Usefulness & Use) • Human Factors Engineering (Usability) CHI2004 Tutorial

17. The Ethnographic Approach Traditional Emphases: • Usefulness & Use(motivations, practice) • Self-report w/ contextualized observation • Naturalisticcontext, no (or low) control • “Why? How?” questions • Qualitative data & deliverables CHI2004 Tutorial

18. The Human Factors Engineering Approach Traditional Emphases: • Usability • Observation (task performance) • Lab context, high control • “How often/fast/much?” questions • Quantitative data & deliverables CHI2004 Tutorial

19. These Approaches are Now Converging… • Self-report & observation are complementary • “Naturalistic” observations are becoming increasingly common (esp. on the Internet) • Using converging methods is more informative and cost-effective. CHI2004 Tutorial

20. These Approaches are Now Converging… • So it is becoming increasingly important to understand the core principles underlying ALL user experience research. • And that’s why we’re here! CHI2004 Tutorial

21. A Primer on User Research Design CHI2004 Tutorial

22. A Primer on User Research Design ‘Science is the elucidation of common sense.’ Francis Bacon (attrib) ‘There are the hard sciences, and then there are the difficult sciences.’ • Gregory Bateson CHI2004 Tutorial

23. The Art & Science of UE Research Design The creative use of …. research principles & pragmatics …to construct, conduct & communicate research to effectively inform product design. CHI2004 Tutorial

24. Overview of the Research Process • Prioritize. Focus on what you want to learn. • Design your research project using appropriate methods based on: - Research principles & pragmatic considerations taken together. • Conduct the research appropriately. • Analyze & interpret findings responsibly. - Use caution; qualify as needed. • Communicate your findings effectively. CHI2004 Tutorial

25. Your Key Research Design Decisions • Methods (What you can do): Ask < ----------- > Observe (Self-Report) (Behavior) • Context (How--& where--you do it): Naturalistic < ----------- > Controlled • Data analyses & deliverables: Qualitative < ----------- > Quantitative CHI2004 Tutorial

26. My Advice… KISS:Keep it simple, s’il vous plait! >> Focus on what you really need to learn. • Your questions, goals, deliverables >> Prioritize! • Design principles & pragmatic considerations >> What evidence would convince YOU? • Use common sense • Be your own best critic. Challenge yourself! CHI2004 Tutorial

27. Core Research Design Principles • Validity • Reliability • Generalizability CHI2004 Tutorial

28. Core Research Design Principles • Validity(Am I really studying what I think I am?) • aka “internal validity” • Reliability (Will my findings be repeatable?) • aka “statistical significance” • Generalizability (Do my findings apply appropriately?) • aka “external validity” CHI2004 Tutorial

29. Validity & Reliability CHI2004 Tutorial

30. Validity & Reliability CHI2004 Tutorial

31. Validity & Reliability High Reliability, High Validity Consistent & ON-Target CHI2004 Tutorial

32. Validity & Reliability CHI2004 Tutorial

33. Validity & Reliability High Reliability, Low Validity Consistent but OFF-Target CHI2004 Tutorial

34. Validity & Reliability CHI2004 Tutorial

35. Validity & Reliability High Reliability, Low Validity NOT Consistent & OFF-Target CHI2004 Tutorial

36. Generalizability CHI2004 Tutorial

37. Generalizability, aka “External Validity” External Applicability “Throughput” to Related Target(s) CHI2004 Tutorial

38. Core Research Design Principles • Validity(Am I really studying what I think I am?) • Confounds & controls, errors & biases • Look for disconfirming evidence • Reliability (Will my findings be repeatable?) • Statistical significance v chance • Sample size (# participants, observations) • Generalizability (Will my findings apply appropriately?) • Representativeness (of sample, context) CHI2004 Tutorial

39. Core Research Design Principles • Validity • Reliability • Generalizability • These 3 principles may seem simple at first glance, but they are profound…and the issues involved can become quite complex. • They provide a foundation for evaluating ALL research—qualitative or quantitative, naturalistic • or controlled, market research or usability testing. CHI2004 Tutorial

40. Example:IRC’sLambdaMOO Project CHI2004 Tutorial

41. LambdaMOO Project Overview Goals: Assess “Hype-otheses”, Characterize Community • Ask (Self-Report) < ----------- > Observe Behavior Survey, Interviews < ----------- > Logfile Analysis • Naturalistic < ----------- > Controlled Context LambdaMOO < ------> Lab? • Qualitative < ----------- > Quantitative Data E.g., Transcripts < ----------- > # Hours Logged On Convergent Methods Approach CHI2004 Tutorial

42. LambdaMOO Project Methods • Survey (Self-Report) • 1 Week Call upon Login; 581 Respondents • ~ 30 Questions, Various Formats, Online • Interviews (Self-Report) • 12+ Real-Life, Long-Term Participants (Many IVR,etc) • 1.5-2 hr In-Depth, Semi-Structured Interviews & Maps & Follow-ups • Logging Studies(Naturalistic Observed Behavior) • Who/Where/When @ ~ 1min Intervals, 24 hr/day, ~2wks • Privacy Respected • Data on ~ 4,000 Users Obtained (Twice, ~ 6 Mo. Interval) • And Much More... • Participant observations, attending BayMOO mtgs, comparison studies, etc. CHI2004 Tutorial

43. Q: Addiction to LambdaMOO? • Self-Report Use Estimates Very High • Previous research papers, popular books & press: • 80 hrs/wk“not uncommon” • Our interview findings not inconsistent • Our Logfile Observations Differed Greatly • Mean ~ 8 hrs/wk (with multi-tasking & idle time!) • Less than 5% users on for 20 or more hrs/wk CHI2004 Tutorial

44. LambdaMOO Use Data from Logfiles Mean=1.13 hrs/day; ~ 8 hrs/wk CHI2004 Tutorial

45. Q: Addiction to LambdaMOO? • Q: Are you convinced by the logfile data? Why or • why not? Can you explain divergent results? • >>Validity • (Am I really studying what I think I am?) • >>Reliability • (Will my findings be repeatable?) • >>Generalizability • (Will my findings apply appropriately?) CHI2004 Tutorial

46. Key Pragmatic Issues • Robustness (How strong is this effect?) • Impact (How important is this finding?) • Convergence (More is better!) CHI2004 Tutorial

47. Key Pragmatic Issues • Robustness (How strong is this effect?) • Large in magnitude (effect size) • Impact (How important is this finding?) • High in priority, severity • Convergence (More is better!) • Multiple modest methods can be most informative • and cost effective. CHI2004 Tutorial

48. Q: Addiction to LambdaMOO? • Robustness (How strong is this effect?) • Huge effect size (difference): 80 v 8 hrs/wk • Impact (How important is this finding?) • High potential social import, impact • Convergence (More is better!) • Self Report & Observation; Qualitative & Quantitative • Triangulating “why”, “how”, “how often/fast/much” • questions from various perspectives. CHI2004 Tutorial

49. Other Pragmatic Considerations… • Complexity of Product/Design/System • Your Deliverables • Design recommendations? Presentation? Paper? • Time Frame • Product deadlines, readiness, design cycle • Cost • Time, money, personnel • Other Resources & Constraints • Availability of participants, prototypes, tools • Your skills, expertise & interests • Organizational & political priorities • Etc. CHI2004 Tutorial

50. Example: LambdaMOO Project • Extremely Complex System, User Community • Our Deliverables • Business & design recommendations • Time-Frame • System complete but evolving; cohort effects • Cost • High • Other Resources & Constraints • Research ideal: Many resources, few constraints CHI2004 Tutorial