Lecture 8: An evaluation framework and Observing users Ref: Ch. 11-12

1. Lecture 8:An evaluation frameworkand Observing users Ref: Ch. 11-12

2. The aims • Explain key evaluation concepts & terms. • Describe the evaluation paradigms & techniques used in interaction design. • Discuss the conceptual, practical and ethical issues that must be considered when planning evaluations. • Introduce the DECIDE framework.

3. Evaluation paradigm • Any kind of evaluation is guided explicitly or implicitly by a set of beliefs, which are often underpined by theory. These beliefs and the methods associated with them are known as an ‘evaluationparadigm’

4. User studies • User studies involve looking at how people behave in their natural environments, or in the laboratory, both with old technologies and with new ones.

5. Four evaluation paradigms • ‘quick and dirty’ • usability testing • field studies • predictive evaluation

6. Quick and dirty • ‘quick & dirty’ evaluation describes the common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in-line with users’ needs and are liked. • Quick & dirty evaluations are done any time. • The emphasis is on fast input to the design process rather than carefully documented findings.

7. Usability testing • Usability testing involves recording typical users’ performance on typical tasks in controlled settings. Field observations may also be used. • As the users perform these tasks they are watched & recorded on video & their key presses are logged. • This data is used to calculate performance times, identify errors & help explain why the users did what they did. • User satisfaction questionnaires & interviews are used to elicit users’ opinions.

8. Field studies • Field studies are done in natural settings • The aim is to understand what users do naturally and how technology impacts them. • In product design field studies can be used to:- identify opportunities for new technology- determine design requirements - decide how best to introduce new technology- evaluate technology in use.

9. Predictive evaluation • Experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems. • Another approach involves theoretically based models. • A key feature of predictive evaluation is that users need not be present • Relatively quick & inexpensive

12. Overview of techniques • observing users, • asking users’ their opinions, • asking experts’ their opinions, • testing users’ performance • modeling users’ task performance

14. Asking experts User Testing Modeling user’s task performance

15. DECIDE: A framework to guide evaluation • Determine the goals the evaluation addresses. • Explore the specific questions to be answered. • Choose the evaluationparadigm and techniques to answer the questions. • Identify the practical issues. • Decide how to deal with the ethical issues. • Evaluate, interpret and present the data.

16. Determine the goals • What are the high-level goals of the evaluation? • Who wants it and why? • The goals influence the paradigm for the study • Some examples of goals: • Identify the best metaphor on which to base the design. • Check to ensure that the final interface is consistent. • Investigate how technology affects working practices. • Improve the usability of an existing product .

17. Explore the questions • All evaluations need goals & questions to guide them so time is not wasted on ill-defined studies. • For example, the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:- What are customers’ attitudes to these new tickets?- Are they concerned about security?- Is the interface for obtaining them poor? • What questions might you ask about the design of a cell phone?

18. Choose the evaluation paradigm & techniques • The evaluation paradigm strongly influences the techniques used, how data is analyzed and presented. • E.g. field studies do not involve testing or modeling

19. Identify practical issues For example, how to: • select users • stay on budget • staying on schedule • find evaluators • select equipment

20. Decide on ethical issues • Develop an informed consent form • Participants have a right to:- know the goals of the study- what will happen to the findings- privacy of personal information- not to be quoted without their agreement - leave when they wish - be treated politely

21. Evaluate, interpret & presentdata • How data is analyzed & presented depends on the paradigm and techniques used. • The following also need to be considered:- Reliability: can the study be replicated?- Validity: is it measuring what you thought?- Biases: is the process creating biases?- Scope: can the findings be generalized?- Ecological validity: is the environment of the study influencing it - e.g. Hawthorn effect

22. Pilot studies • A small trial run of the main study. • The aim is to make sure your plan is viable. • Pilot studies check:- that you can conduct the procedure- that interview scripts, questionnaires, experiments, etc. work appropriately • It’s worth doing several to iron out problems before doing the main study. • Ask colleagues if you can’t spare real users.

23. Key points • An evaluation paradigm is an approach that is influenced by particular theories and philosophies. • Five categories of techniques were identified: observing users, asking users, asking experts, user testing, modeling users. • The DECIDE framework has six parts: - Determine the overall goals - Explore the questions that satisfy the goals - Choose the paradigm and techniques - Identify the practical issues - Decide on the ethical issues - Evaluate ways to analyze & present data • Do a pilot study

24. Observing users

25. The aims • Discuss the benefits & challenges of different types of observation. • Describe how to observe as an on-looker, a participant, & an ethnographer. • Discuss how to collect, analyze & present observational data. • Examine think-aloud, diary studies & logging. • Provide you with experience in doing observation and critiquing observation studies.

26. What and when to observe • What: • Goals & questions determine the paradigms and techniques used. • Observation is valuable any time during design. • Observers: • outsiders looking on • participants, i.e., participant observers • ethnographers • When: • Any time during product development

27. Approaches to observation • Quick & dirty observations early in design • Observation in the field (i.e., field studies) • Observation in controlled environments (i.e., usability studies)

28. Types of observation

29. Frameworks to guide observation in the field • - The person. Who? - The place. Where?- The thing. What? • The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?

30. The Robinson (1993) framework • Space. What is the physical space like? • Actors. Who is involved? • Activities. What are they doing? • Objects. What objects are present? • Acts. What are individuals doing? • Events. What kind of event is it? • Goals. What do they to accomplish? • Feelings. What is the mood of the group and of individuals?

31. Before going into the field, you need to consider • Goals & questions • Which framework & techniques • How to collect data and record events • Which equipment to use • How to gain acceptance and trust • How to handle sensitive issues • Whether and how to involve informants • How to analyze the data, your notes • Consider working as a team • Plan to look at from different perspectives

32. Observing as an outsider • As in usability testing • More objective than participant observation • In usability lab equipment is in place • Recording is continuous • Analysis & observation almost simultaneous • Care needed to avoid drowning in data • Analysis can be coarse or fine grained • Video clips can be powerful for telling story • (Pang2 Guan1 Zhe3 Qing1)

33. Participant observation & ethnography • Debate about differences • Participant observation is key component of ethnography • Must get co-operation of people observed • Informants are useful • Data analysis is continuous • Interpretivist technique • Questions get refined as understanding grows • Reports usually contain examples

34. Data collection techniques • Notes & still camera • flexible • Difficult to observe and write at the same time • Audio & still camera • Less intrusive than video • Mobile • Flexible • Lack of visual record • Onerous to transcribe the data • Video • Both visual and audio data • Intrusive • Attention focus on the screened site • Data analysis is time-consuming (100 times)

37. Tracking users • Diaries • When • What • What they thought • Advantage • When users are not easy to reach • Disadvantage • Rely on the participants • Only remember the events better or worse than the real • Interaction logging • Key presses • Mouse • Other device movements • Time-stamped • Advantages: • Unobtrusive • Large volumes of data • Disadvantages: • May still raise ethical concerns • Need powerful tools

38. Data analysis • Qualitativedata - interpreted & used to tell the ‘story’ about what was observed. • Qualitative data - categorized using techniques such as content analysis. • Quantitative data - collected from interaction & video logs. Presented as values, tables, charts, graphs and treated statistically.

39. Main activities of quali-analysis to tell a story • Review the data • Record the themes in a coherent yet flexible form. • Record the date and time of each data analysis session • Check your understanding with the people you observe • Iterate the above processes • Report your findings

40. Interpretive data analysis • Look for key events that drive the group’s activity • Look for patterns of behavior • Test data sources against each other • Report findings in a convincing and honest way • Produce ‘rich’ or ‘thick descriptions’ • Include quotes, pictures, and anecdotes • Software tools can be useful e.g., NUDIST

41. Quali-analysis for categorization • Look for incidents or patterns • Obviously stuck • Comment • Silence • puzzlement • Analyze data into categories • Content analysis: a fine grain way of video analysis • Key point: do not overlap each other in any way • Analyzing discourse • Conversation analysis • Focus on dialogues • Pay more on contexts • Fine-grain

42. Key points • Observe from outside or as a participant • Analyzing video and data logs can be time-consuming. • In participant observation collections of comments, incidents, and artifacts are made. Ethnography is a philosophy with a set of techniques that include participant observation and interviews. • Ethnographers immerse themselves in the culture that they study.