Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement

1. Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement Lisa D. Cota, MS Statistical Analyst Office of University Assessment & Testing Oklahoma State University

2. Why not NHST? • Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003. Retrieved from http://psycnet.apa.org/journals/amp/49/12/997/ • Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1–20. Retrieved from http://www2.uni-jena.de/svw/metheval/lehre/0405-ws/evaluationuebung/haller.pdf • Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review. doi:10.3758/s13423-013-0572-3 • Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31(1), 65–71. doi:10.1016/j.newideapsych.2011.02.009 • Lykken, D. What’s wrong with psychology anyway? (1991). In Thinking Clearly About Psychology. University of Minnesota Press. • Toomela, A. (2010). Quantitative methods in psychology: Inevitable and useless. Frontiers in Psychology, 1(July), 29. doi:10.3389/fpsyg.2010.00029

3. Why not NHST? • Assessment should be understandable • NHST: Not easily understood…in fact easily misunderstood • Oakes, 1986; Gigerenzer, 2004 • Hoekstra, Morey, Rouder, & Wagonmaker, 2014 • One problem (of many) with NHST: • Variance Explained ≠ Accuracy

4. y = 0.5x + 3, R2 = 0.663

5. y = 0.5x + 3, R2= 0.663

6. y = 0.5x + 3, R2= 0.663

7. y = 0.5x + 3, R2= 0.663

8. Assumptions • Assumptions for independent samples t-test under NHST: • Null hypothesis is true • Observations are independent between and within groups • Population variances are exactly equal • Bivariate IV • Random sampling / random assignment (???) * Continuous DV * DV has a normal population distribution

9. Observation Oriented Modeling • No meaningless aggregates • Person-centered • No assumption-laden p-values

10. Observation Oriented Modeling: Ordinal Pattern Analysis

11. Results

18. Randomization Test

19. Results, with chance-value

20. VALUE Rubric Data

21. Results

24. Significant ≠ Meaningful

25. Significant ≠ Meaningful

27. Significant ≠ Meaningful

28. Transparency, Honesty, and Progress

29. Raw Numbers

30. Proportions

31. Compare w. SPSS….

32. SPSS Legacy Dialog, k Independent Samples

33. Proportions

36. Advantages of OOM • Assumption-free • Transparent • Honest

37. OOM website • “We’ve never actually looked at the data!” • www.idiogrid.com/OOM