Examination of Biased Outcome Reporting in Educational Research

1. Examination of Biased Outcome Reporting in Educational Research Terri Pigott & Ryan Williams Jeff Valentine & Dericka Canada Loyola University Chicago University of Louisville

2. Well-known tendency for studies to be published in peer-review journals when the results are statistically significant Publication bias can also work within published studies Important details about the study methods are missing Outcomes gathered in the original research are not reported in the published work Publication bias in research

3. Identified problem in clinical trials in medicine,e.g., Vedula, Bero, Scherer & Dickersin (2009), NEJM. Studies of off-label use of gabapentin Compared outcomes described in published reports with those described in internal research documents from the industry sponsor Many documents for study obtained as a result of litigation against drug companies Biased outcome reporting

4. Figure 1. Vedula et al. (2009)

5. In the example here, the published reports lead to a more favorable assessment of gabapentin’s efficacy for unapproved indications Vedula et al. (2009)

6. If published studies selectively report outcomes of an intervention, any inferences drawn about the effects of an intervention will also be biased Extent of this problem depends on the relation between the observed and the unobserved outcomes If strong (e.g., two highly correlated measures of a construct), then problem is likely minimal If weak (e.g., two measures of two different constructs) then problem will lead to biased inferences What evidence do we have in education research of selective outcome reporting? Implications for Inferences

7. No research registries in educational research Instead, we focused on dissertations completed between 2001 and 2005 at the 96 Carnegie designated Research Universities/Very High Research Activity (RU/VH) For this presentation, we randomly sampled 26 of the 96 RU/VH institutions Our study

8. Using DAI, we searched for dissertations completed between 2001 and 2005 at each sampled RU/VH institution with keyword “Education” Using the abstracts, we identified those reporting on an intervention for student outcomes in PreK – 12 We focused on interventions for ease of identification of primary outcomes Methods

9. After identifying the set of dissertations that studied an intervention with PreK-12 students, we searched for a published version of the dissertation primarily using Google Scholar In the matched dissertations, we coded each outcome and its associated p-value. In the matched publication, we coded whether each outcome was reported Methods (continued)

10. This sample contains 4,102 dissertations Of those, 199 (5%) were intervention studies with PreK-12 student outcomes Of these 16 (8%) had an identifiable and comparable published outcome that met our inclusion criteria Results

11. Our search thus far has located 16 studies with 209 different treatment outcomes Statistically significant outcomes were more likely to appear in the published version of the study than were non-statistically significant outcomes (p = .003) OR = 2.34 with CI95% (1.35, 4.14) RR = 1.48 with CI95% (1.15, 1.92) Results Continued

12. Next steps Complete coding of all 96 university dissertation sets Multilevel analysis to model bias across studies Some evidence of outcome bias in education research What are the practical implications? Next Steps

13. Few dissertations focus on interventions for PreK-12 students IES reports majority of their applications and funded studies are Goal 1 and 2 (exploration and development) and not Goal 3 or 4 (testing an intervention strategy and scaling up) Why are we seeing few intervention studies in education? Observations about Ed Research