Quantitative Methods for Researchers

1. Quantitative Methods for Researchers Paul Cairns paul.cairns@york.ac.uk

2. Objectives • Statistical argument • Safe designs • A whizz through some stats • Time for questions

3. Statistical Argument • Inference is an argument form • Prediction is essential • Alternative hypothesis • “X causes Y” • No prediction – measuring noise

4. Gold standard argument • Collect data • Data variation could be chance (null) • Predictthe variations (alternative) • Statistics give probabilities • Unlikely predictions “prove” your case

5. Implications • Must have an alt (testable) hyp • No multiple testing • No post hoc analysis • Need multiple experiments

6. Silver standard argument • Collect data • Data variations could be chance (null) • Are there “real” patterns in the data? • Use statistics to suggest (unlikely) patterns • Follow up findings with gold standard work

7. Fishing: This is bad science • Collect lots of data • DVs and IVs • Data variations could be chance • Test until a significant result appears • Report the tests that were significant • Claim the result is important

8. Statistical pit… • … is bottomless! • Safe designs • One (or two) IV • Two (or three) conditions • One primary DV • Other stuff is not severely tested

9. Choosing a test • What’s the data type? • Do you know the distribution? • Within or between • What are you looking for?

10. Seeing location • Boxplots • Median, IQR, • “Range” • Outliers

12. Distributions • Theoretical stance • Must have this! • Not inferred from samples

13. Parametric tests • Normal distribution • Two parameters • Null = one underlying normal distribution • Differences in location (mean)

14. t-test: null vs alternate

15. t-test • Two samples • Two means • Are means showing natural variation? • Compare difference to natural variation

16. Effect size • How interesting is the difference? • 2s difference in timings • Significance is not same as importance • Cohen’s d

17. ANOVA • Parametric • Multiple groups • Why not do pairwise comparison? • Get an F value • Follow up tests

18. ANOVA++ • Multiple IV • So more F values! • Within and between • Effect size, η2 • Amount of variance predicted by IV

19. Non-parametric tests • Unknown underlying distribution • Heterogeneity of variance • Non-interval data • Usually test location • Effect size is tricky!

20. Basic tests • Mann-Whitney • Wilcoxon • Kruskal-Wallis • Friedman • No accepted two-way tests

21. Choosing a test For your fantasy abstract, what test would you choose? Why? Would you change your design?

22. Questions • Specific problems • Specific tests • Other tests?

23. Useful Reading • Cairns, Cox, Research Methods for HCI: chaps 6 • Rowntree, Statistics Without Tears • Howell, Fundamental Statistics for the Behavioural Sciences, 6thedn. • Abelson, Statistics as Principled Argument • Silver, The Signal and the Noise

24. Multivariate • Multiple DV • Multivariate normal distribution • Normal no matter how you slice • MANOVA • Null = one underlying (mv) normal distribution

26. Issues • Sample size • Assumptions • Interpretation • Communication

27. Monte Carlo • Process but not distribution • Generate a really large sample • Compare to your sample • Still theoretically driven!

28. Example • Event = 4 heads in a row from a set of 20 flips of a coin • You have sample of 30 sets • 18 events • How likely? • Get flipping!