Daniele Fanelli

1. How Many Scientists Fabricate and Falsify Research?A Systematic Review and Meta-Analysis of Survey Data Daniele Fanelli

2. “The million dollar question”…. -Only misconduct that has been discovered, and (presumably) proven to be intentional Ultimately, only scientists know about their own intentions!

3. Over the years, many surveys have asked scientists directly… …different things, in different ways… Form of misconduct Question Outcome “Since entering medical school have you…?” “Fabricated data” Yes No “Modified research or experimental results to improve the outcome” Never SometimesFrequently “Have you participated in research involving […] during the last 10 years?” “Failing to present data that contradict one's own previous research” 0 1-5 >5 “Indicate the number of […] members you have observed/experienced exhibiting […] within the last 5 years” “Seriously misleading interpretation of results” Results appeared inconclusive and dificult to compare

4. p é ù - ) 1 û p ë ( ES=Log 1 - = W= n p ( 1 p ) 2 SE “Tricks” in the analysis • How many committed or observed X at least once • Only questions on fabrication, falsification, alteration and QRP that distort scientific knowledge. No plagiarism, professional misconduct etc… • Mixed questions were excluded Question by question: effect size and weight • No other measure of study quality (it’s controversial) • Included all eligible studies that specified their methods • Entered methodological factors in inverse variance weighted regression

5. The search… "research misconduct" OR "research integrity" OR "research malpractice" OR "scientific fraud" OR "fabrication, falsification" OR "falsification, fabrication" 42 literature databases, 14 journals, 8 grey literature db, 2 internet scientific search engines, and references lists Potentially relevant studies obtained from electronic search (n=3276) Studies excluded because were not surveys on research misconduct (n=3207) Studies excluded for one of the following reasons (n=48): -Did not have any relevant or original data -Sample not exclusively composed of researchers -Misconduct not related to research (e.g. cheating on school projects) -Does not distinguish fabrication and falsification from other forms of misconduct not relevant to this review -Presents data only in format not usable in this review Studies retrieved for examination of full text (n=69) Studies included in review (n=21) Studies excluded from meta-analysis because did not meet quality criteria (n=3) Studies included in meta-analysis (n=18)

6. Characteristics of studies • Conducted between 1986-2005 • USA (15), UK (3), multinational (2), and Australia (1) • Medical/clinical (8), biomedical (6), multidisciplinary (6), economists (1) • In total 85 questions: • about fabrication, falsification, alteration, modification (meta-analysis) • Questionable research practices (systematic review only) (Full data set available soon in PLoS ONE)

7. Scientists who admit fabrication, falsification, or alteration of results Scientists who know a colleague who fabricated, falsified, or altered results b= -0.14±0.05 P=0.006 1.97% (N=7, 95%CI: 0.86-4.45) If only asked “fabrication, falsification” 1.06% (N=4, 95%CI: 0.31-3.51) 14.12% (N=12, 95% CI: 9.91-19.72) If only asked “fabrication, falsification” 12.34% (N=11, 95%CI: 8.43-17.71)

8. Questionable Research Practices (e.g. “failing to publish data that contradicts one’s previous research” “dropping data points based on a gut feeling”)

9. What influences admission rates? Inverse variance-weighted regression b±SE P - Asking about self vs colleagues: <0.0001 -4.53±0.81 82% of variance explained (N=15) - Using “fabrication” or “falsification” vs “alteration” or “modification” -1.02±0.39 0.0086 + Handed-out surveys vs mailed: 1.17±0.4 0.0032 Controlling for these factors, tested for differences between: Year USA / other Researcher / other n.s. Medical / other b=0.85±0.28 Biomedical / other P= 0.0022 Social Sc. / other

10. Leave-one-out sensitivity analysis Scientists who admit fabrication, falsification, or alteration of results Scientists who know a colleague who fabricated, falsified, or altered results Martinson 2005 is outstanding, as conservative!

11. “Repairing misconduct” Around half of recalled cases had no action whatsoever taken against them

12. Summary of key findings • Data fabrication, falsification and alteration was • admitted on average by around 2% (1% - 4%) • directly observed by 14% (10% - 20%) • Questionable Research Practices were • admitted on average by up to 34% • directly observed by up to 72% • Overall admission rates (self-/non-self) were higher in • Non-self reports, questions not using “fabrication” or “falsification”, handed out questionnaires • Medical/clinical and related research

13. How Reliable Are These Numbers? Self-reports Conservative Unlike surveys for other criminal behaviour, Scientists always lose by admitting misconduct Non-self-reports • Unclear • Risk of multiple reporting • “Muhammad Ali” effect • Unaware of all cases • Unwilling to damage their field • Regression analysis: • Medical research not robust to all sensitivity • Differences in methodology masked most effects • Non-significant effects not necessarily non-significant

14. (tip of the iceberg) (this too) Conclusions • On average, 2% of scientists admitted misconduct, and 34% QRP • Actual frequencies probably higher • Probably vary depending on field and many other factors, which meta-analysis currently cannot detect • Future surveys might benefit by • Focusing on correlates of misconduct • Common methodologies