Interpretive Evaluation

1. Interpretive Evaluation • Agenda • Simple user modeling • Interpretive Evaluation • - Motivation • - Methods • Ethnography • Fitt’s Law

2. Simpler User Modeling • How do attributes of users influence the design of user interfaces? • Are there some design guidelines that we can derive from different attributes? CS/PSY 6750

3. User Profiles • Attributes: • attitude, motivation, reading level, typing skill, education, system experience, task experience, computer literacy, frequency of use, training, color-blindness, handedness, gender,… • Novice, intermediate, expert CS/PSY 6750

4. User Low motivation, discretionary use Low motivation, mandatory High motivation, due to fear High motivation, due to interest Design goal Ease of learning Control, power Ease of learning, robustness, control Power, ease of use Motivation CS/PSY 6750

5. Experience task system low low high high low high high low Design goals Many syntactic and semantic prompts Efficient commands, concise syntax Semantic help facilities Lots of syntactic prompting Knowledge & Experience CS/PSY 6750

6. Job & Task Implications • Frequency of use • High - Ease of use • Low - Ease of learning & remembering • Task implications • High - Ease of use • Low - Ease of learning • System use • Mandatory - Ease of using • Discretionary - Ease of learning CS/PSY 6750

7. Evaluation • Predictive • Discount Usability (Mon) • Cognitive Modeling-GOMS family (Wed) • Cognitive Modeling-contextual (next Mon) • Interpretive (today) • User testing (later) CS/PSY 6750

8. Approaches • Experimental (Lab studies, quantitative) • Typically in a closed, lab setting Manipulate independent variables to see effect on dependent variables • Naturalistic (Field studies, qualitative) • Observation occurs in “real life” setting Watch process over time “Ecologically valid” CS/PSY 6750

9. Interpretive Evaluation • Experimental: Formal and objective • Interpretive: More subjective • Concerned with humans, so no objective reality • Sociological, anthropological approach • Users involved, as opposed to predictive approaches CS/PSY 6750

10. Beliefs • Sees limitations in scientific hypothesis testing in closed environment • Lab is not real world • Can’t control all variables • Context is neglected • Artificial, short tasks CS/PSY 6750

11. IE Methods • A number of different methods or techniques in this area exist • Ethnography • Our main focus • Contextual inquiry • More specific form of ethnography with a focus on asking questions • Field study • Common notion, often equated with ethnography • Observational study • Much more informal, just watching users • … CS/PSY 6750

12. Ethnography • Deeply contextual study • Immerse oneself in situation you want to learn about (anthropological and sociological roots) • Observing people in their cultural context • Interpretation of data is primary • Behavior is meaningful only in context CS/PSY 6750

13. Philosophy • Argues that formal environment of controlled study is artificial --- Experimenter wields “power” over subject • So … get into working environment of user • -> Interpretation is primary, rather than data CS/PSY 6750

14. Objectives • Understanding the user • Understand goals and values • Understand individual’s or group’s interactions within a culture • Try to make tacit domain knowledge explicit • Do this in an unbiased fashion • For UI designers: Improve system by finding problems in way it is currently being used CS/PSY 6750

15. Techniques • In person observation • Audio/video recording • Interviews • “Wallow in the data” CS/PSY 6750

16. Observation is Key • Carefully observe everything about users and their environment • Think of describing it to someone who has never seen this activity before • What users say is important, but also non-verbal details CS/PSY 6750

17. Observations • Things of interest to evaluator • Structure and language used in work • Individual and group actions • Culture affecting work • Explicit and implicit aspects of work • Example: Office work environment • Business practices, rooms, artifacts, work standards, relationships between workers, managers, … CS/PSY 6750

18. Interviews Important • Have a question plan, but keep interview open to different directions • Be specific • Create interpretations together with users • At end, query “What should I have asked?” • Record interviews CS/PSY 6750

19. Steps • 1. Preparation • Understand organization policies and work culture • Familiarize yourself with system and its history • Set initial goals and prepare questions • Gain access and permission to observe & interview • 2. Field study • Establish rapport with users • Observe/interview users in workplace and collect all different forms of data • Follow any leads that emerge from visits • Record the visits Rose et al ‘95 CS/PSY 6750

20. Steps • 3. Analysis • Compile collected data in numerical, textual and multimedia databases • Quantify data and compile statistics • Reduce and interpret data • Refine goals and process used • 4. Reporting • Consider multiple audiences and goals • Prepare a report and present findings CS/PSY 6750

21. One Technique • Affinity Diagram • Process • Write down each quote/observation on a slip of paper • Put up on board • Coalesce items that have affinity • If they are saying similar things about an issue • Give names to different groups (colors too) • Continue grouping subgroups • A hierarchy will be formed CS/PSY 6750

22. Why Useful? • Can help designer gain a rich and true assessment of user needs • Help to define requirements • Uncovers true nature of user’s job • Discovers things that are outside of job description or documentation • Allows you to play role of end-user • Can sit in when real users not available • Open-ended and unbiased nature promotes discovery • Empirical study and task analysis are more formal – ethnography may yield more unexpected revelations CS/PSY 6750

23. Types of Findings • Can be both • Qualitative • Observe trends, habits, patterns, … • Quantitative • How often was something done, what per cent of the time did something occur, how many different … CS/PSY 6750

24. Drawbacks • Time required • Can take weeks or months • Scale • Most use small numbers of participants just to keep somewhat manageable • Type of results • Highly qualitative, may be difficult to present and use • Acquired skill • Identifying and extracting “interesting” things is challenging CS/PSY 6750

25. Ethnomethodology • Concurrent/informed ethnography • Study is being done in conjunction with a system being developed • + Helps keep focus on user throughout design • - Requires lots of time and coordination CS/PSY 6750

26. Cooperative Evaluation • User is viewed as collaborator in evaluation, not a subject • “Friendly approach” • Relaxed version of think-aloud • Evaluator and participant can ask each other questions CS/PSY 6750

27. CE Methods • Seeks to detect errors early in a prototype • Experimenter uses tasks, also talks to participant throughout, asks questions CS/PSY 6750

28. Low Level Models • Fitt’s Law • Models movement times for selection tasks • Basic idea: Movement time for a well-rehearsed selection task • Increases as the distance to the target increases • Decreases as the size of the target increases CS/PSY 6750

29. Components • ID - Index of difficulty ID = log2 ( 2A/W ) distance to move width tolerance of target bits result Both quantities are distances so unitless result CS/PSY 6750

30. Components • MT - Movement time MT = b * ID or MT = a + b*ID MT is a linear function of ID CS/PSY 6750

31. Original Experiment • 1-D A W CS/PSY 6750

32. Exact Equation • Run empirical tests to determine a and b in MT = a + b*ID • Will get different ones for different input devices and ways the device is used MT ID CS/PSY 6750

33. Common Equation • MT = a + b log2 (A/W + 1) • Provides useful numbers CS/PSY 6750

34. Questions • What do you do in 2D? • Where can this be applied in user interface design? CS/PSY 6750