Seeing is believing: Using systematic observations in user experience research.

1. Seeing is believing: Using systematic observations in user experience research. March 11th, 2011 Bram van Mil b.van.mil@noldus.nl Sales consultant

3. Noldus Information Technology Systematic Observations Measuring Behavior with The Observer XT Observation labs and other research tools Agenda

4. Who we are Noldus Information Technology, since 1989… International developer of professional software for behavioral research From a single software license up to a complete observation lab Complete with installation, training and support HQ located in Wageningen, The Netherlands Founded in 1989, currently ≈ 115 employees Customers in 75 countries Installed base Observer software ≈ 4400 organizations

6. Develop new drugs to cure diseases Support medical diagnosis and therapy Support human resource management Improve educational systems Improve animal welfare Control insect pests Create more usable products Create safer work environments Optimize team performance Etc. Why study behavior?

7. Questionnaires Observations Ways to study behavior

8. Problem of meaning (interpretation of a question) Problem of omission (inadvertently leaving out our relevant information) Problem of memory Social desirability effect Question threat (questions may appear threatening) Interviewer characteristics (influence on participant) Gap between stated and actual behavior Disadvantages of questionnaires

9. Record who does what, where,when and with whom The alternative: Systematic observations

10. Objective Accurate Standardized Quantify qualitative data Advantages of systematic observations

11. Measuring Behavior the pioneers way… A human observer watches the behavior of thesubjects and writes down the observations on paper and uses a clock for time-information. Disadvantages: • Subjective measurements • When writing, the observer cannot observe • Very labor intensive • No integration with other modalities

12. Solution: The Observer XT for structured observations Create a coding scheme before or during the observations and annotate behavior by a simple key press. This will result in an event-log which contains time-stamped events that can be synchronized with multimodal data: Videos Physiological data Eye tracking data Etc.

13. How to execute systematic observations • Three steps • 1. Setup • 2. Observe • 3. Analyze

14. Step 1 – Setup Observation sources How to execute systematic observations

15. How to execute systematic observations • Step 1 – Setup • Coding scheme • Subjects • Behaviors • - Point events • - State events • Modifiers

16. How to execute systematic observations • Step 1 – Setup • Coding scheme : Example • TimeSubject, behavior, modifiers • 09:35:12 Joe, run, fast, North • 09:40:23 Mary, talk, Rose • 09:40:24 Rose, smile • 09:40:24Rose, affect,5 • 09:40:31 Mary, smile

17. How to execute systematic observations • Step 1 – Setup • Coding scheme : Mutually exclusive behaviors • Mutually exclusive: one behavior ends the previous

18. How to execute systematic observations • Step 2 – Observe • Live scoring: • With or without videos • Offline scoring: • From digital video files

19. How to execute systematic observations • Step 3 – Analyze • Data selection • Choose the data you want to analyze by filtering and nesting your data • This is done by data profiles: • In order to save different criteria for you data selection and data analysis

20. How to execute systematic observations • Step 3 – Analyze • Visualization of your data

21. Step 3 – Analyze • Descriptive statistics • Advanced analysis - Lag-sequential analyses - Reliability analysis • Statistical analysis - Export to Excel, SPSS or as ASCI file • Export highlight videos How to execute systematic observations

22. How to execute systematic observations • Step 3 – Analyze: Numerical analysis on external data

23. Other tools That can be used in behavioral research

24. FaceReader Detect facial expressions • Neutral • Happy • Sad • Angry • Surprised • Scare • Disgusted • Also: gender, age, ethnicity, glasses and facial hair

25. Eye Tracking Systems See what your participants are looking at • Computer monitor / Standalone / Head mounted • Gaze tracks, heat maps, fixations and areas of interest

26. Physiological data Detect mental load, physical load and emotional states • ECG – Electro Cardio Gram • EMG – Electro Myo Gram • EEG – Electroencephalogram • GSR – Galvanic Skin Response

27. Labs Complete setups for behavioral research

28. Portable Usability Lab

29. Portable Observation Lab

30. Custom Mobile Observation Labs

31. Stationary Observation Lab

32. Game Experience Lab Ghent University Two room Usability Lab for User Experience research in gaming on consoles and pc’s

33. Game Experience Lab Ghent University

34. Game Experience Lab Ghent University

35. Different facilities under 1 roof Observational-, sensory- and mood- labs Observe eating behavior Introduce better suited products Restaurant of the Future Wageningen University

36. Restaurant of the Future Wageningen University

37. Service, training and support We are there when you need us On-site installation and training • Behavioral scientists • Hardware engineers Excellent technical support • Helpdesk • Online knowledge base Other • Service contracts • Measuring Behavior conference

38. Bram van Mil b.van.mil@noldus.nl Sales consultant