Observing users

1. Observing users Presenter : JungMin shin & JaeYoung Lee

2. Contents • Goals, questions, paradigms • How to observe • Data Collection • Indirect Observation: tracking user’s activities • Analyzing, interpreting, and presenting data • Field studies

3. Goals, questions, paradigms • What and when to observe • Approached to observation

4. Goals, questions, paradigms Goal and question provide a focus for observation, DECIDE framework. Having goal helps to guide the observation because there is always so much going on. • What and when to observe On developing product starting time : helps understand user’s needs. ending time : examine to satisfy user’sneeds.

5. Goals, questions, paradigms • Role of evaluator Onlookers, participant observers, ethnographers Table 12.1 Type of observation

6. Goals, questions, paradigms • Approaches to observation 1. “Quick and dirty” observation Watching and talking to users in a casual way to get immediate feedback. 2. Observation in usability testing Video and interacting logs capture everything. Can watch through a one-way mirror or via TV screen. Observational data is used to see and analyze for user’s action.

7. Goals, questions, paradigms • Observation in usability testing System

8. Goals, questions, paradigms 3. Observation in field studies Observers may be anywhere along the outsider – insider spectrum. Colin Robson(1993) – participating level complete participants more marginal participants observers who also participate people who observefrom the outside and do not participate. The goal is to cause as little disruption as possible.

9. Goals, questions, paradigms 3. Observation in field studies (Continue) ex) A study of the time spent by boy and girl using technology in the classroom. -> observe and note at the back of room. A study of understanding how the computer integrates with other artifacts and social interacting. -> might take of insider perspective.

10. Goals, questions, paradigms 3. Observation in field studies (Continue) participant observation evaluator participate with users in order to learn what they do and how and why they do it. fully participant observer a member of group. being present to share experience and learning the social conventions of the group.

11. Goals, questions, paradigms Determining goals , exploring questions Choosing techniques are necessary steps in the DECIDE framework.

12. How to observe • In controlled environments • In the field environments • Participant observation and ethnography

13. How to observe • In controlled environments Collect and then Make sense of thedata while watching users in controlled environment. Practical issues in controlled environment - where users will be located - equipment test - An informed consent form

14. How to observe • In controlled environments the observer doesn’t know what users are thinking, and can only guess from what she sees. Think-aloud technique by Erikson and Simon Require people to say out loud everything that they are thinking and trying to do. ex) I’m typing in http://ics.uci.edu (type) Now I press the enter key, right? (presses enter key) But if users meet some errors, they could be silent again... Sol) Evaluator reminds him think out loud, but that would be intrusive. -> two people work together so that they talk to each other.

15. How to observe • In the field environments Events in the field can be complex and rapidly changing. Framework help observers to keep their goals and questions in sight. Collin Robson(1993) Space, Actors, Activities, Objects, Acts, Events, Goals, Feelings

16. How to observe • In the field environments • Checklist of things to plan before going into field 1. State the initial study goal and question 2. Select the framework 3. Decide how to record events 4. Be prepared to go through your notes and other records 5. Make and review your notes 6. Be prepared refocus your study 7. Think about how you will get the acceptance and trust of those you observe 8. Think about how to handle sensitive issues (privacy) 9. Consider working as a team 10. Consider checking your notes with an informant or members of the group 11. Plan to look at the situation from different perspectives

17. How to observe • Participant observation and ethnography Checklist for doing ethnography • Identifying a problems or goal and then ask good questions to be answered by the study • The most important part of fieldwork just being there to observe, ask questions, and record. • Collect a variety of data (Interview, retrospectives interview) • Be prepared to move backwards and forwards between the broad picture and specific questions • Analyze the data using a holistic approach -> contextualized, usually iterative

18. How to observe • Participant observation and ethnography <Dilemma> When have I observed enough? schedules often dictate when your study end. stop when you stop learning new things when you start to see similar pattern of behavior being repeated. when you have listened to all the main stake holder group and understood their perspectives.

19. How to observe • Participant observation and ethnography How can I adapt ethnography to fit the development process? Ann Rosa Procedure Preparing (Understand/Familiarize/Set Goal /Gain permission) Field study (Establish/Observe and interview/ Follow any lead/ Record) Analysis (Compile/Quantify/Reduce and interpret/ Refine the goals) Reporting (Consider/Prepare)

20. Data collection • Notes plus still camera • Audio recording plus still camera • Video

21. Data collection • Notes plus still camera being difficult and tiring to write and observe at the same time bored, the limitation of writing speed (sol) working with another person (disadvantage) have to be transcribed • Audio recording plus still camera Useful alternative to note taking and is less intrusive than Video. Drawback: Lack of visual record, transcribing the data So, evaluators use the recording to remind them about important details and as a source of anecdotes for reports.

22. Data collection • Video Advantage Capturing both Audio and Video Data Disadvantage Intrusive. Attention becomes focused on what is seen through the lens. Time-consuming for analyzing video data. ex) 1 hour video recording : over 100 hours of analysis time

23. Indirect Observation: tracking user’s activities • Diaries • Interaction logs

24. Indirect Observation: tracking user’s activities When ? Impossible to observe directly • Diaries A record of what users did, when did it, what thought about their interactions with the Technology. (Advantage) When User scattered and unreachable person. Inexpensive - no equipment or expertise. Suitable for long term studies. Template and Enable the data to go straight into database. Ex) open-ended online questionnaires (Disadvantage) Rely on participant (reliable/remembering). Participant remember better or worse that they really were.

25. Indirect Observation: tracking user’s activities • Interaction logs key presses, mouse or other device movement are recorded usually synchronized with A/V log. Has time-stamped (to calculate how long) Explicit counter / Recording number of visitors EX) In study of interact art museum (Researchers at USC) Server log (7 months/ analysis tool : webtrends) when/what /how long/what browser/what country… (Advantage) Unobtrusive (but this also raises ethical concerns) Large volumes of data.

26. Analyzing, interpreting, and presenting data The first things to do is to “eyeball”. Proceed to analyze it according to the goals and questions. • Qualitative analysis to tell a story • Qualitative analysis for categorization • Quantitative data analysis • Feeding the findings back into design

27. Analyzing, interpreting, and presenting data • Qualitative analysis to tell a story Power of analyzing descriptive data -> being able to tell a convincing story • Interview with Sara Bly (User-Centered Design Consultant) At the end of each observation period Review their data Discuss what they observed Construct a story from their data

28. Analyzing, interpreting, and presenting data • Qualitative analysis to tell a story To summarize about Interview with Sara Bly. Review the data. Record the themes. Record the date and time of each data analysis Session. As themes emerge. Iterate this process (until your story faithfully represents). Report your findings to the development team.

29. Analyzing, interpreting, and presenting data • Qualitative analysis to tell a story • Analyzing and reporting ethnographic data - ethnographers reconstruct knowledge to produce detailed description known as rich or thick description. Main activities (Fetterman-1998) • Look for key events • Look for patterns of behavior in various situations and among different players • Compare sources • Report your findings in convincing and honest way • S/W tools : NUDIST and Ethnograph

30. Analyzing, interpreting, and presenting data • Qualitative analysis for categorization Looking for incident or patterns Common strategy is to look for critical incidents ex) Jurgen Koenemann-Belliveau el al(1994) used this form of critical incident analysis to examine breakdown or problems in achieving a programming task and also to identify possible threats of incidents. Using the theory helped the evaluators to focus in relevant Incidents.

31. Analyzing, interpreting, and presenting data • Qualitative analysis for categorization analyzing data into categories contents analysis provides another fine grain way of analyzing video data. determined by the evaluation question. must also be reliable so that the analysis can be replicated. EX) training second person and then analyze the same data sample. if there are large discrepancy between each other. -> what? Inter-research reliability rating -> the percentage of agreement between the two researchers.

32. Analyzing, interpreting, and presenting data • Analyzing discourse Dialog Strongly interpretive, pays great attention to context. Language is viewed as a constructive tool and discourse analysis provides a way of focusing upon how people use language to construct versions of their worlds (Fiske,1994) Small changes in wording can change meaning, as the following excerpts indicate (Coyle, 1995) (According to Coyle,) discourse analysis is what you do when you are saying that you are doing discourse analysis…

33. Analyzing, interpreting, andpresenting data • Quantitative data analysis In case of errors or unusual behavior -> marks the video and records a brief remark. To use this annotated recording 1. Calculate performance time 2. further analyzed using simple statistics such as means, standard deviations, T-tests, etc.

34. Feeding the findings back into design The results can be reported to the design team. Written reports with a overview and detailed content list. Quantitative data -> its value depends on the type of study and its goals. Often both quantitative data analysis are useful because of they provide alternative perspectives. Analyzing, interpreting, and presenting data

35. Field Study

36. Field Study • A broad range of data gathering techniques at the user’s location • There is no one best way to conduct a field study. It depends on the goals of your study and your access to users.

37. The Goal of a Field Study • Field studies are excellent for collecting rich, detailed data and obtaining a holistic view of the process or domain. • Field studies are not a good technique if you want to collect quantitative data from large samples. • The goal of a field study is to make the implicit aspects and processes of the user environment explicit.

38. Field Study Methods • Techniques are divided into three categories: • Observation only • Interacting with the user • Method supplements • The Goal of the Field Study Methods • To observe users and collect information about their tasks and the context in which they are done.

39. Field Study Methods • Observation Only • Pure Observation • Deep Hanging-Out • Interacting with the user • Contextual Inquiry • Process analysis • Condensed ethnographic interview • Discount User Observation (DUO) • Method supplements • Artifact walkthroughs • Incident diaries • Observing while you are not present

40. Field Study Methods • Observation Only • Pure Observation • Deep Hanging-Out

41. Observation only - Pure Observation • Synopsis • When you are unable or don’t wish to interact with the user, you simply observe from a distance. • Advantages • Flexible • Low resources • Level of effort • Minimal, Place your self in a good vantage point and observe as many user/sites/tasks as you feel appropriate • You continue to conduct observations until you feel you have a good understanding of the domain of areas of focus.

42. Observation only - Pure Observation • Valuable in situations where you cannot interact with the end users. • Users may or may not know they are being studied. • Do not interact with the participant • Do not distribute surveys, interview the user, or ask for artifacts from the user. • It is essential to have a good sampling plan. The sampling plan should include specific information such as days/times you anticipate key events (e.g., the day before Thanksgiving, or bad weather at an airport), as well as “normal” days.

43. Field Study Methods • Observation Only • Pure Observation • Deep Hanging-Out

44. Observation only – Deep Hanging out • Synopsis • This method is similar to pure observation but provides more structure by suggesting focus areas and things to observe. • Advantages • It has more structured than pure observation so you can do a more detailed level of data analysis and compare data collected across multiple sites. • Level of effort • Because there is more structure, it takes more effort than pure observation. • You are “on” at all times, which can be tiring. • It is also valuable to become a user yourself (if possible) and collect artifacts along the way.

45. Observation only – Deep Hanging out • A more structured form of pure observation. • Involves significant amounts of observation along with involving yourself in the process • Method of Deep Hanging-Out includes • Structured observation • Collection of artifacts • Becoming a user yourself • However, you do notinterview participants, distribute surveys, or present design ideas for feed back.

46. Field Study Methods • Observation Only • Pure Observation • Deep Hanging-Out • Interacting with the user • Contextual Inquiry • Process analysis • Condensed ethnographic interview • Discount User Observation (DUO) • Method supplements • Artifact walkthroughs • Incident diaries • Observing while you are not present

47. Field Study Methods • Interacting with the user • Contextual Inquiry • Process analysis • Condensed ethnographic interview • Discount User Observation (DUO)

48. Interacting with the user - Contextual Inquiry • Synopsis • Interview, apprentice with, and interpret the resulting data with users • Advantages • Contextual Inquiry is more focused and context dependent than the other methods • At the end, you walk away with actionable items • Level of effort • The effort level is higher than for observation-only techniques. You must develop an observation guide, observe users, apprentice with them, and discuss your observations with them

49. Interacting with the user - Contextual Inquiry • Four main parts to Contextual Inquiry: • Context • Partnership • Interpretation • Focus

50. Interacting with the user - Contextual Inquiry • Four main parts to Contextual Inquiry: • Context : You must go to the user’s environment in order to understand the context of his/her actions. Contextual Inquiry assumes that observation alone or out-of-context interviews are insufficient