100 likes | 215 Views
Misuses of Quantitative Research. V. Darleen Opfer. Competing Notions of the Use of Quantitative Data. You can prove anything with statistics Versus There are three kinds of lies: lies, damned lies, and statistics. Statistical Pitfalls. Sources of bias Errors in method
E N D
Misuses of Quantitative Research V. Darleen Opfer
Competing Notions of the Use of Quantitative Data You can prove anything with statistics Versus There are three kinds of lies: lies, damned lies, and statistics
Statistical Pitfalls • Sources of bias • Errors in method • Problems with interpretation
Sources of bias • Representative sampling • Statistical assumptions
Girls performance has increased more rapidly than that of boys. Speed, 1998 • Female students have improved their performance markedly, whereas male students have not shown a similar improvement. Arnot, 1996 • There are increasing gender differences in performance. Stobart, 1992
Errors in Method • Multiple comparisons • Measurement error
Problems with interpretation • Confusion over significance • Precision and accuracy • Causality • Unequal comparisons
Definitions of School Effects • School effects as the absolute effects of schooling, using naturally occurring “control” groups of students who receive no schooling • School effects as the unadjusted average achievement of all students in a school • School effects as the impact of schooling on the average achievement of all students in a school, adjusted for family background and/or prior achievement • School effects as measuring the extent of ‘between schools’ variation in the total variation of their students’ individual test scores • School effects as measuring the unique effect of each school on their students’ outcomes • School effects as measuring the impact of schools on student performance over time
What is Good Statistical Practice? • Be sure your sample is representative of the population in which you’re interested • Be sure you understand the assumptions of your statistical procedures and be sure they are satisfied • Be sure the sample size is not misleading you about the amount of significance • Be sure to use the best measurement/collection tools • Be aware of impact of multiple comparisons and the lack of an a priori analysis plan • Be clear about what you’re trying to discover • Use numerical notation in a rational way • Be sure you understand the conditions for causal inference • Be explicit about the measures you are comparing