Lecture #7 COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS

1. Lecture #7COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS Y39TUR Spring 2011 Testování uživatelského rozhraní

2. Today • Sběr a analýza dat při uživatelských testech v laboratoři + Zpracování výsledků kvalitativních testů • priprava testu • co je potreba pripravit, jak si muzu pomoct ruznymi nastroji • sber dat pri testu • jak muzu logovat (xls, Morae), na co se pri logovani zamerit • jaka dalsi data se daji logovat, jaky maji uplatneni/vyuziti v analyze • analyza dat po testu • co je dulezite hledat v datech z kvalitativnich testu • jak si pomoct nastroji • Materialy • hlavne z lekce 8, neco malo z lekce 5 • Casove rozvrzeni • Obecny uvod – 10 minut • Priprava testu – 10 minut • Sber dat pri testu – 20 minut • Analyza dat po testu – 20 minut TUR 2011

3. DATA COLLECTION

4. Standard Method of Usability Test User gets an assignment The observer observes Problem It is hard to remember all the user activities TUR 2010 S. Greenberg

5. Recording the Observation Video Recording Can see what the user does Typical are multi-camera setup One camera records the screen One camera records the participant face and body Problem of synchronization “Big Brother problem” Audio Recording Can hear what the user does Good for think-aloud protocol Audio recording is very important TUR 2010

6. Think-aloud Protocol The participants vocalizes their thoughts What they are trying to do Why they are taking those actions How they interpret the reactions of the system What are their suggestions Hmm, what does this do? I’ll try it… Ooops, now what happened? TUR 2010 S. Greenberg

7. Think-aloud Protocol Pros: Provides insights to the user’s thinking Most common method in the Usability Engineering Cons: Can alter the mental process of the user Unnatural It’s difficult to speak when focusing on the task TUR 2010

8. Example of Hints (Think-aloud) • “Please, speak on” • “Please tell me what you think.” • “Please tell me what you are trying to do.” • “Are you searching for something in particular?” • “What do you think that it’s going to happen now?” • “What did you mean by that?” TUR 2010 Adapted from Jake Wobbrock

9. Application Support – Morae I • Audio/Video Recording- Morae Recorder • Allows up to 2 video sources • Video recording can take some processing resources of the PC TUR 2010

10. Video Annotations - Datalogging • Processing of AV recording takes 4-8 times the length of the recording • E.g.: 6 participants, 1h each video = minimum 24h! ~ 2-4 days • Typical solution: • While capturing: Annotate video • While processing: Focus on important parts only • Problems of video annotations • It is hard to keep up with the tempo of what's going on • You may miss important interaction during annotation • Annotations contain data from different categories • e.g. user opinions, behavioral observations and demographic notes • Improvement - Markers TUR 2010

11. Basic Marker Definition TUR 2011

12. Example of Video Annotations • An actual usability test • Website • Web portal of a university • Task #8 (10 tasks total) • “Find information on life-long education program at the Faculty of Architecture.” TUR 2010

13. Example of Video Annotations TimeNoteTask IDMarker 15:06 Session Starts M [some lines omitted] 15:16 Tries to find “Info for the Students” #8 M 15:17 Can’t find in the left-hand menu X Goes systematically through all links in this menu X Wishes to use full text search, does not how C Found link “Alumni” M 15:20 Life-long Education link NOT FOUND X 15:21 Found link “For applicants” M FOUND link “Life-long education” M [some lines omitted] 15:49 Session Ends M TUR 2010

14. Video Annotations Tips • Marker categories are difficult to remember • Start with subset of the markers • Add more markers when the technique became familiar • Multiple observers: • Can increase number of found usability problems • Can put more information into annotations TUR 2010

15. Other Collected Data • Low level interaction data • Mouse clicks, mouse movements • Key presses • Can be used for location of interesting parts in video recording • Application data • Generally any information that may be interesting or valuable during evaluation • Typically requires some hooks, API or functionality inside the applications • Examples • Web - url, rendered page, Basket content • Mobile – GPS position, content of search field • Eye-tracking TUR 2010 TUR 2010 (15)

16. After-test Activities TUR 2010 It is important to find out what the participants think Do they find the test easy? Difficult? Conditions of the test: Good, bad? Other comments?

17. Interview Good for finding out specific problems “Set up” the question to match the context Can focus on the problems as they show up during the interview Good for research studies (open-ended questioning) Leads to specific suggestions Problems Statements are subjective Time-consuming The interviewer can easily affect the results TUR 2010

18. How to carry out an interview Plan the list of basic questions Several good questions can start an interview (But avoid leading questions.) These questions can focus the interview. Can be based on the results received from the observation. TUR 2010

19. Retrospective testing interviews • Post-observation interview • Observe (make the test) • Capture a video recording • User watches the recording and comments on in • Explains an unclear behavior during the test • Great for interpretation of the post-test interview • Avoids misinterpretation • Can identify particular improvements Do you know why you never tried that option? I didn’t see it. Why don’t you make it look like a button? TUR 2010 S. Greenberg

20. Critical Situations during an Interview • People speak about problems that have emerged • People vividly speak about marginal problems • Important only for them • Problem has emerged that wasnot captured during the test Tell me about the last big problem you had with Word I can never get my figures in the right place. Its really annoying. I spent hours on it and I had to… TUR 2010 S. Greenberg

21. Data Collection from Interview Same as for usability test Audio/video recording Annotations Optionally other data (low level, application) TUR 2010

22. Post Test Questionnaire Allows easier data collection compared to interview Typically uses Likert scale (1-5) Allows also Yes-No questions Allows also open ended questions (limited) TUR 2010

23. Example of Post Test Questionnaire • An actual usability test • Website • Web portal of a university TUR 2011

24. Application support – Morae III • Autopilot mode - Morae Recorder • Instructions of the tasks are provided online • Pre/Post test questionnaires are filled online • Subjective difficulty rating after each task is filled online • Data are stored into recordings for each participant • Autopilot disadvantages • Autopilot interface can hide interface of the tested application • You may loose data when recording is corrupted TUR 2010

25. DATA ANALYSIS

26. Factors that impact data analysis • Factors suggesting formal data presentation: • Summative studies • Lower level of support in company UCD processes • External audience • Novices • Factors suggesting informal data presentation: • Formative studies • Higher level of support in company UCD processes • Internal audience • Experts Source: UPA 2006 Idea Markets - Analyzing usability study results: Is it magic behind the curtain? Activator: Emma J. Rose, University of Washington TUR 2011

27. Strategies for Usability Data Analysis • Formal vs. informal approaches to data analysis • Statistical analysis • Used less frequently, requires knowledge about statistics • Calculating metrics • Calculation of success and failure rates and questionnaire data such as Likert scales • Analyzing notes for patterns • Looking for trends and patterns, across tasks and users. • Physical observations • Observing the facial and bodily expressions especially in regards to frustration or confusion • Analysis “on-the-go” • Changes to the design are made immediately based on informal notes during a study Source: UPA 2006 Idea Markets - Analyzing usability study results: Is it magic behind the curtain? Activator: Emma J. Rose, University of Washington TUR 2011

28. Statistical analysis • Time needed to carry out each task. • Frequency of errors made by the users. • Responses to questionnaire items • Etc. • See lecture #12 TUR 2010

29. Calculating metrics (examples) • Time to carry out the task • Number of tasks carried out • Number of errors • Number of used (or unused) commands and functions • Frequency of help access • Frequency of useful help access • Frequency of positive (negative) comments of the participant • Ratio of participants preferring the tested system • etc … TUR 2010

30. Application support – Morae • Graph Visualizations • Mouse Clicks Graph • Count of Markers Graph • SUS Survey Graph • Time on Task Graph • Web Page Changes Graph TUR 2010

31. Analyzing notes for patterns I • Summarize the findings from the collected data • List of all important events • Positive or negative aspects • It’s a good idea to link back to the original data • Identify why there were problems • Things you can look for in your data: • Is the UI behaving predictably? • Have the people behaved in the way you expected? • Were all necessary functions available? • Wasn’t there too many unnecessary functions? TUR 2010

32. Analyzing notes for patterns II • Read through the notes • Look for: • Repetitions • Things that could be caused by the same underlying problem • Can be done in the whole group of testers • Cluster the observations • By underlying problem • E.g.: Group all problems related to poor structure of the information • By feature • E.g.: Group all problems related to printing TUR 2010

33. Analyzing notes for patterns III • Describe the clusters • What was the problem • E.g.: “5 out of 8 participants could not locate the menu item X” • The impact of the problem • E.g.: “Function Y could not therefore be accessed.” • Place where the problem occurred TUR 2010

34. Analyzing notes for patterns IV • (continuation of the university website example) • Searching for course • Priority: 2 • The participants do not know where to look for courses. They go to "study programmes." • They do not understand the distinction between "career courses" and "retraining courses" • They do not know what the Masaryk Institute of Advanced Studes is. TUR 2010

35. Application support – Morae I End oftask Taskstart Final anchor Currentposition Editation andlogging tools Play task Timelinezoom Eventmarkers TUR 2010 • Timeline visualization • Shows annotations in the timeline • Each marker category has its own color TUR 2010 (35)

36. Application support – Morae II TUR 2010 Filtering of tasks Filtering of participants Annotation list Annotations frequency graph TUR 2010 (36)

37. Application support – Morae III TUR 2010 Filtering of tasks Filtering of participants Annotation list Annotations frequency graph TUR 2010 (37)

38. Physical observations TUR 2011

39. Physical observations TUR 2011

40. Physical observations TUR 2011

41. USAGE OF LOW LEVEL DATAA CASE STUDY R&D at CTU Maly, Mikovec, Slavik 2007

42. Use case: UI for inventory application on PDA Task scenario User comes to the room User looks around User checks hardware inventory in the room User writes down additional notes about the room to the PDA User rests User leaves the room Office TUR 2010

43. Categories of activities for mobile application TUR 2010 Activities that represent user’s interaction with a program (picking objects on the screen, selection from menus, entering data etc.) Behavior of the user more in detail (like her/his gestures, sitting, standing up, communicating with other persons etc.) Motion of the user in environment where s/he should perform interaction (e.g. in interior of a building)

44. Visualization of detailed behavior TUR 2010

45. Visualization of detailed behavior TUR 2010

46. Visualization of detailed behavior TUR 2010

47. Visualization of detailed behavior TUR 2010

48. Visualization of user motion TUR 2010

49. Visualization of user motion HighROI MiddleROI Low ROI High cognitive load TUR 2010

50. Eye tracking visualization – Heat Map http://blogs.lib.ucdavis.edu/hsl/2010/01/14/peeking-at-jakob-nielsens-eyetracking-web-usability/ TUR 2011