Data analysis and interpretation

1. Data analysis and interpretation

2. Agenda • Part 2 comments • Average score: 87 • Part 3: due in 2 weeks • Data analysis

3. Project part 3 • Please read the comments on your evaluation plans • Finish your plan • Finalize questions, tasks • Prepare scripts or tutorials, etc. • Find participants • Friends, neighbors, co-workers • Perform the evaluations • Clearly inform your users what you are doing and why. • If you are audio or video recording, I prefer you use a consent form. • Pilot at least once – know how long its going to take.

4. Part 3 write up • State exactly what you did (task list, how many, questionnaires etc.) • Summarize data collected • Summarize usability conclusions based on your data • Discuss implications for the prototype based on those conclusions

5. Quantitative and qualitative • Quantitative data – expressed as numbers • Qualitative data – difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction • Quantitative analysis – numerical methods to ascertain size, magnitude, amount • Qualitative analysis – expresses the nature of elements and is represented as themes, patterns, stories • Be careful how you manipulate data and numbers!

6. Descriptive Statistics • For all variables, get a feel for results: • Total scores, times, ratings, etc. • Minimum, maximum • Mean, median, ranges, etc. • e.g. “Twenty participants completed both sessions (10 males, 10 females; mean age 22.4, range 18-37 years).” • e.g. “The median time to complete the task in the mouse-input group was 34.5 s (min=19.2, max=305 s).”

7. Simple quantitative analysis • Averages • Mean: add up values and divide by number of data points • Median: middle value of data when ranked • Mode: figure that appears most often in the data • Percentages versus numbers • Graphical representations give overview of data

8. Subgroup Stats • Look at descriptive stats (means, medians, ranges, etc.) for any subgroups • e.g. “The mean error rate for the mouse-input group was 3.4%. The mean error rate for the keyboard group was 5.6%.” • e.g. “The median completion time (in seconds) for the three groups were: novices: 4.4, moderate users: 4.6, and experts: 2.6.”

9. Plot the Data • Look for the trends graphically

10. Other Presentation Methods Scatter plot Box plot Middle 50% Age low high Mean 0 20 Time in secs.

11. Visualizing log data Interaction profiles of players in online game Log of web page activity

12. Simple qualitative analysis • Recurring patterns or themes • Emergent from data • Categorizingdata • Categorization scheme may be emergent or pre-specified • Looking for critical incidents • Helps to focus in on key events

13. Presenting the findings • Only make claims that your data can support • The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken • Graphical representations may be appropriate for presentation • Other techniques are: • Using stories, e.g. to create scenarios based on the data • Summarizing the findings

14. Interviews • Raw data: • Audio or video recordings, interviewer notes • Initial processing • Transcribe audio, or expand upon notes • Qualitative processing • Group answers to same question (small # of questions and people) • Label interesting phrases or words • Put labels on post-its or in software and group labels • Quantitative processing • Gather quantitative responses such as age, etc. • Categorize and count responses (5 liked, 3 disliked, etc.) • Presentation • Summarize responses, tell stories and patterns • Use descriptive quotes

15. Questionnaire • Raw data: • Tables of questions and numbers or text answers • Quantitative processing • Calculate descriptive stats (means, percentages, etc.) for each question • Can break into subgroups or use statistics to look for relationships between items (does age correlate to stronger preferences?) • Qualitative processing • Group answers to same question • Presentation • Present tables & charts of means, percentages, etc. • Explain overall meaning of all the responses

16. Observation • Raw data: • Audio or video recording, log files, notes • Initial processing: • Transcribe audio, expand notes or take more based on video, synchronize logs with recordings • Quantitative processing • Record metrics such as errors, times, clicks, etc. • Produce descriptive stats and charts of those metrics • Qualitative processing • Note places where problems occurred, interesting behaviors, common behaviors • Presentation • Descriptions of common or interesting problems • Videos demonstrating issues, or descriptive quotes • Charts describing quantitative data

17. Sample Think-aloud categorization • Interface problems • Verbalizations show evidence of dissatisfaction about an aspect of the interface. • Verbalizations show evidence of confusion/uncertainty about an aspect of the interface. • Verbalizations show evidence of confusion/surprise at the outcome of an action. • Verbalizations show evidence that they are having problems achieving a goal. • Verbalizations show evidence that the user has made an error. • The participant I unable to recover from error without external help from the experimenter. • The participant makes a suggestion for redesign of the interface. See pg 380 for more complete example

18. Experimental Results • How does one know if an experiment’s results mean anything or confirm any beliefs? • Example: 40 people participated, 28 preferred interface 1, 12 preferred interface 2 • What do you conclude?

19. Goal of analysis • Get >95% confidence in significance of result • that is, null hypothesis disproved • Ho: Timecolor = Timeb/w • OR, there is an influence • ORR, only 1 in 20 chance that difference occurred due to random chance

20. Means Not Always Perfect Experiment 1 Group 1Group 2 Mean: 7 Mean: 10 1,10,10 3,6,21 Experiment 2 Group 1Group 2 Mean: 7 Mean: 10 6,7,8 8,11,11

21. Inferential Stats and the Data Are these really different? What would that mean?

22. Hypothesis Testing • Tests to determine differences • t-test to compare two means • ANOVA (Analysis of Variance) to compare several means • Need to determine “statistical significance” • “Significance level” (p): • The probability that your null hypothesis was wrong, simply by chance • p (“alpha” level) is often set at 0.05, or 5% of the time you’ll get the result you saw, just by chance

23. Errors • Errors in analysis do occur • Main Types: • Type I/False positive - You conclude there is a difference, when in fact there isn’t • Type II/False negative - You conclude there is no difference when there is • And then there’s the True Negative…

24. Drawing Conclusions • Make your conclusions based on the descriptive stats, but back them up with inferential stats • e.g., “The expert group performed faster than the novice group t(1,34) = 4.6, p > .01.” • Translate the stats into words that regular people can understand • e.g., “Thus, those who have computer experience will be able to perform better, right from the beginning…”

25. Tools to support data analysis • Spreadsheet – simple to use, basic graphs • Can even do basic statistical analysis • Statistical packages, e.g. SPSS • Qualitative data analysis tools • Categorization and theme-based analysis, e.g. N6 • Quantitative analysis of text-based data

26. Analysis and Presentation for Part 3 • List of problems from HE with severity ratings • List of problems found in CW • Basic quantitative analysis from your observation • Basic qualitative analysis from your observation • Places where problems occur, general story of what and how people did, etc. • Basic quantitative and qualitative analysis from the questionnaire or interview • Tables of responses, averages, etc. as appropriate

27. Interpreting your results • Go through each usability criteria – do results demonstrate support for meeting this criteria or not? How do they? • Discuss any other problems with aspects of the design that your results demonstrate. • Discuss how you would modify the design based on these results.