1 / 29

Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford

How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in the development of reporting guidelines. Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford

hal
Download Presentation

Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in the development of reporting guidelines Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford Sally Hopewell, PhD, UK Cochrane Centre Lindsay Shepard, MSc, University of Utah The Empty Reviews Project is funded by the Cochrane Opportunities Fund

  2. Summary of Presentation • The Problem of Empty Reviews • Current State of Cochrane Empty Reviews and Apparent Direction of Travel • How this Project Aims to Proceed • Cochrane Contributor Inputs The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  3. Introduction: Why study empty reviews? • First, what do we mean by Empty Review? • An empty review is a systematic review that reports no studies eligible for inclusion. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  4. How do systematic reviews come to be empty? • Lack of high quality studies to address question posed. • Very narrow population • Very focused intervention • Outcomes that are not measured. • Many plan empty reviews to justify new trials in response to funding policies. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  5. Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions. • Should observations be made based on excluded studies to provide decision-makers with some guidance? (following Lang, Edwards & Flieszer 2007) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  6. The Cochrane Handbook (2009) does not address the reporting of empty reviews specifically, although it does address excluded studies in general. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  7. Excluded Study Guidance in Cochrane Handbook (p. 154) “A Cochrane review includes a list of excluded studies, detailing any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may on the surface appear to meet the eligibility criteria but on further inspection do not, and also those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible…” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  8. Concern:Empty reviews may leave decision-makers dependent on weaker research which may be presented without sufficient discussion of limitations. • Concern: Empty reviews may fail to provide any implications either for practice or research. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  9. Empty Reviews Project Aim: To develop guidelines, in consultation with stakeholders and the editors of the Cochrane Handbook, for authors of reviews who do not find studies eligible for inclusion. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  10. Prevalence of Empty Reviews in TheCDSR(as of 15 August 2010) • Total reviews in TheCDSR N= 4320 • Total empty reviews in TheCDSR N= 375 (8.7%) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  11. Number of Empty Reviews across CRGs The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  12. Proportion of Empty Reviews across CRGs The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  13. Empty Reviews by Year of First Publication The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  14. Age of Empty Reviews in TheCDSR • Half of all empty reviews are 3 years or older, with some dating to 1995. • Mean Age = 3.3 years (SD 3.4) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  15. Time since last update of Empty Reviews • 48.3% of all empty reviews have been updated within the last 2 years, but 5.9% (21) have not received an update in 7 years or longer. • Mean Time since Update = 2.8 years (SD 2.2) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  16. How many excluded studies are listed in these empty reviews? • 95 (25.3%) of empty reviews do not list excluded studies. • The number of excluded studies ranges from 0 to 124. • Mean number of excluded studies = 9.6 (SD 14.5) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  17. How many studies are listed as pending assessment? • 360 (96.0%) of empty reviews do not list pending studies. • The number of pending studies ranges from 0 to 9. • Mean number of pending studies = 0.1 (SD 0.7) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  18. How many studies are listed as on-going? • 338 (90.1%) of empty reviews do not list on-going studies. • The number of on-going studies ranges from 0 to 4. • Mean number of on-going studies = 0.1 (SD 0.5) The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  19. How many reviews list no studies at all? • 88 (23.5%) of empty reviews do not list any excluded, pending, or on-going studies. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  20. So what? • So, what are the implications of all these empty reviews? • The lack of guidelines for reporting studies that have no included studies may result in inconsistently reported reviews. • Policy-makers and practitioners might not rely on Cochrane reviews for decision-making. • More importantly, however, the lack of guidelines for authors of empty reviews may result in less than useful conclusions and further, may misstate results. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  21. What do we mean by “less than useful conclusions”? Less than useful conclusions. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  22. What sorts of conclusions might be less than useful? • Reference to single studies without appropriate caveats. • Unsupported statements. • No statements at all. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  23. Review A : Implications for practice Sufficient evidence based on randomised controlled trials is lacking and therefore we can draw no reliable conclusions about the comparative effectiveness of medical and surgical interventions of [deleted to de-identify]. However, the studies identified, but excluded from this review, suggest there may be a considerable response to placebo treatment in this condition. Review B: Implications for practice On the evidence currently available, [ ] cannot be recommended for the treatment of [ ]. Implications for research [ ] has shown promise in animal studies but there are no randomized double-blind placebo-controlled trials in people with [ ]. Such trials are needed. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  24. What might be better ways to handle conclusions? • Specific recommendations for design of trials. • Appeal to related reviews, drawing analogy to present question, with caveats. • Balanced discussion of weaker studies. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  25. Review C: Implications for research • General As with all similar reviews, public registration of a study before anyone is randomised would ensure that participants could be confident that people would know that the study had at least taken place. Compliance with CONSORT (Moher 2001), both on the part of authors and editors, would help to clarify methodology and many outcomes. Failure to comply with CONSORT guidelines results in loss of data and confusion in results, neither of which helps clinicians, patients or managers. • 2. Specific [ ] is a most distressing [ ] disorder that remains highly prevalent, both in the developed and developing world. This review highlights the need for well designed, conducted and reported clinical trials (Table 1) to address the claims of open studies as regards the effects of the [ ] group of drugs for [ ]. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  26. What do we hope to accomplish? • Consistency in the way that excluded studies are addressed in the authors’ conclusions. • Collaboration with other stakeholders towards development of guidelines for inclusion in The Cochrane Handbook. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  27. Our Plan • Complete our analysis of the 375 empty reviews in The CDSR as of 15 August, 2010. • Solicit input from all interested parties via cochrane.empty.reviews@gmail.com and www.empty-reviews.org (coming soon!). • Develop specific recommendations around the writing, presentation, and persistence in updating empty reviews. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  28. Present our findings and recommendations at a consensus meeting of key stakeholders in Spring, 2011, including Cochrane Handbook editors. • Use the results of the consensus meeting to refine our recommendations. • Present our findings to the Madrid Colloquium in October 2011, for further discussion. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

  29. The Cochrane Empty Reviews Project Funded by The Cochrane Opportunities Fund Paul Montgomery, PhD, University of Oxford, PI Sally Hopewell, PhD, UK Cochrane Centre, Co-PI Joanne Yaffe, PhD, University of Utah, Co-PI Lindsay Shepard, MSc, University of Utah, Research Associate cochrane.empty.reviews@gmail.com http://www.empty-reviews.org Cochrane.empty.reviews@gmail.com

More Related