Publication Bias in Medical Informatics evaluation research: Is it an issue or not?

1. Publication Bias in Medical Informatics evaluation research:Is it an issue or not? Mag. (FH) Christof Machan, M.Sc. Univ-Prof. Elske Ammenwerth Dr. Thomas Bodner christof.machan@umit.at

2. Overview • Background • Detecting publication bias - 2 attempts • study 1: A simplistic approach • study 2: The funnel plot • Discussion • Conclusions / Outlook

3. Background Why think about publication bias • evidence-based practice • systematic review as best evidence • should include as many publications on the question invesigated as possible • high danger of publication bias leading to wrong conclusions • Strong evidence for publication bias in other fields (social sciences, biomedicine) • What about evaluation literature in medical informatics?

4. Detecting PB Study 1: Detecting publication bias - a simplistic approach Assumption If there is a remarkable difference between publications reporting positive and publications reporting negative results in a random sample publication bias is a possible reason. (see also Dickersin 1990) Methods • random sample of 86 MI evaluation publications out of evalDB • Classify / Count / Compare Results 72 publications reported clear results (positive or negative) • 60 positive • 12 negative Positive / negative papers positive negative

5. n n effect size effect size Detecting PB Study 2: Detecting publication bias - the funnel plot - basics • graphical depiction of publication bias • easy to understand • scatter plot displaying study quality (e.g. sample size, standard error,...) and effect size • recommended by Cochrane Source: http://www.ukl.uni-freiburg.de/med/med8/seminar%2018%2011%2004.ppt#30

6. Methods study design: meta-analysis Search in Medline and Embase for controlled trials on CPOE and medication errors / ADEs Hand Search in several journals intervention: CPOE system effect size: relative risk for medication errors / ADEs between intervention and control group 26 studies included assessment of publication bias by funnel plot Detecting PB Study 2: Effects of CPOE on medication errors / ADEs Assumption Missing publications of low study quality reporting on negative and / or non-significant results may be an indication for publication bias.

7. Detecting PB Study 2: Effects of CPOE on medication errors / ADEs

8. Discussion Discussion Study 1: • 60 out of 72 studies positive seems a high number • limitations: • assumption that ratio of positive / negative studies equals 50:50 not valid • other biases (e.g. langauge bias) • chance • ... Study 1: • plot indicates sign for publication bias • limitations: • other selection biases (language, citation, etc.) • poor methodological quality of smaller studies • true heterogenity • chance

9. Conclusions / Outlook Conclusions / Outlook • Publication bias is an issue • existence hard to proof • easy to jump to wrong conclusions • awareness of authors important • quality of publications must be sufficient • Evidence-based medical informatics