Quantitative Methods for Researchers

1. Quantitative Methods for Researchers Paul Cairns paul.cairns@york.ac.uk

2. Objectives • Statistical argument • Comparison of distributions • A fly-by of approaches

3. How are the abstracts? • Questions? • Problems? • Restarts?

4. Statistical Argument • Inference is an argument form • Prediction is essential • Alternative hypothesis • “X causes Y” • No prediction – measuring noise

5. Gold standard argument • Collect data • Data variation could be chance (null) • Predict the variations (alternative) • Statistics give probabilities • Unlikely predictions “prove” your case

6. Implications • Must have an alt hyp • No multiple testing • No post hoc analysis • Need multiple experiments

7. Silver standard argument • Collect data • Data variations could be chance (null) • Are there “real” patterns in the data? • Use statistics to suggest (unlikely) patterns • Follow up findings with gold standard work

8. Fishing: This is bad science • Collect lots of data • DVs and IVs • Data variations could be chance • Test until a significant result appears • Report the tests that were significant • Claim the result is important

9. Statistical inference • Model comparison: • Single distribution (null) • Multiple distributions (alternative) • From samples, which model is better? • From samples, is null likely?

10. What terms do you know? • The statistical zoo!

11. Choosing a test • What’s the data type? • Do you know the distribution? • Within or between • What are you looking for?

12. Distributions • Theoretical stance • Must have this! • Not inferred from samples

13. Parametric tests • Normal distribution • Two parameters • Null = one underlying normal distribution • Differences in location (mean)

14. t-test models

15. t-test • Two samples • Two means • Are means showing natural variation? • Compare difference to natural variation

16. Effect size • How interesting is the difference? • 2s difference in timings • Significance is not same as importance • Cohen’s d

17. ANOVA • Parametric • Multiple groups • Why not do pairwise comparison? • Get an F value • Follow up tests

18. ANOVA++ • Multiple IV • So more F values! • Within and between • Effect size, η2 • Amount of variance predicted by IV

19. Non-parametric tests • Unknown underlying distribution • Heterogeneity of variance • Non-interval data • Usually test location • Effect size is tricky!

20. Wilcoxon test • See sheet

21. Seeing location • Boxplots • Median, IQR, • “Range” • Outliers

23. Multivariate • Multiple DV • Multivariate normal distribution • Normal no matter how you slice • MANOVA • Null = one underlying (mv) normal distribution

25. Issues • Sample size • Assumptions • Interpretation • Communication

26. Your abstract • What sort of data will you produce? • Can you theorise about the distribution? • What sort of test do you think you will need?

27. Health warnings • Craft skill • Simpler is better • Doing it • Interpreting it • Communicating it • Experiments as evidence • Software packages are deceptively easy

28. Q & A • Any question about any aspect • Very general or very specific • Any research method!

29. Useful Reading • Cairns, Cox, Research Methods for HCI: chaps 6 • Rowntree, Statistics Without Tears • Howell, Fundamental Statistics for the Behavioural Sciences, 6thedn. • Abelson, Statistics as Principled Argument • Silver, The Signal and the Noise

30. Monte Carlo • Process but not distribution • Generate a really large sample • Compare to your sample • Still theoretically driven!

31. Example • Event = 4 heads in a row from a set of 20 flips of a coin • You have sample of 30 sets • 18 events • How likely? • Get flipping!