Content Analysis of Implications for Practice in Empty Reviews

1. Content Analysis of Implications for Practice in Empty Reviews Joanne Yaffe, PhD, University of Utah Lindsay Shepard, MSc, University of Utah Paul Montgomery, PhD, University of Oxford Sally Hopewell, PhD, UK Cochrane Centre The Empty Reviews Project is funded by the Cochrane Opportunities Fund

2. Introduction: What We Knew • 8.7% of reviews in The CDSR did not identify any eligible studies. • 25.3% of empty reviews did not list excluded studies. • 90.2% of empty reviews did not list ongoing studies. • 96.0% of empty reviews did not list studies waiting assessment. • 23.4% of empty reviews did not list any studies other than what was discussed in their background section. • Proportions of empty reviews differed somewhat across topical areas, from 0% to 26.9%. • CRG editors differed somewhat in their perspectives on empty reviews and on editorial guidance provided to authors of these reviews. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

3. What We Wanted to Know Despite the lack of high quality, or any, evidence included in empty reviews, how do authors report implications for practice? The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

4. Guidance from Handbook Implications for Practice 12.7.2 “Authors of Cochrane reviews should not make recommendations. If authors feel compelled to lay out actions that clinicians and patients could take, they should – after describing the quality of evidence and the balance of benefits and harms – highlight different actions that might be consistent with particular patterns of values and preferences. Other factors that might influence a decision should also be highlighted, including any known factors that would be expected to modify the effects of the intervention, the baseline risk or status of the patient, costs and who bears those costs, and the availability of resources.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

5. Conceptual Definition of Recommendation We defined recommendations for practice as statements which instruct readers, including consumers, practitioners, or policy makers to dosomething. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

6. Reporting of Empty Reviews • Lang et al. (2007) • Suggest guidelines needed. • Further, reviewers could note observations from ineligible articles and abstracts. • Green et al. (2007) • Specific structure could be helpful. • Basing conclusions on studies which do not meet inclusion criteria increases the risk of bias. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

7. Objectives of the Present Analysis To explore the following: • How authors frame discussions of Implications for Practice? • Do authors incorporate information from other studies? If so, how and where? • Do authors make recommendations for practice? The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

8. One Sentence Wonders 117 (31.1%) empty reviews contained a one sentence Implications for Practice section. These: • May be extremely brief and contain almost no information; and • vary in how well they explain implications for decision making. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

9. Examples of One Sentence Wonders • Very Brief: • "None identified from this systematic review.” • Mention lack of evidence and review topic: • “No randomized controlled trials, which assessed the effectiveness or safety of oral immunoglobulin preparations for the treatment of rotavirus diarrhoea in hospitalized low birthweight infants were found.” • Absence of evidence is not evidence of absence: • "No relevant trials appear to have been published so this review concludes that there is no evidence of effect of leukotriene antagonists in bronchiectasis rather than evidence of no effect.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

10. Vague Wording 26 (6.9%) empty reviews are written in such tentative language that despite lengthy discussions between two of us, we sometimes failed to come to agreement about whether the section made recommendations and if so, for what. • “It is likely that clinicians will continue with their current practice, using clinical judgement and prescribing patterns to dictate treatment because there is no RCT-based evidence to help guide their choice of drug. It is difficult to know whether current practice is justified outside of a well designed, conducted and reported RCT. Currently policymakers have no RCT-based evidence upon which to base guidelines for HIVAN. They are likely to continue to rely on opinion and habit when making their recommendations. Funders of studies may wish to make this important subgroup of people a priority for future research.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

11. Failure to Cite Around half of implications for practice sections make reference to evidence, but only 16% of these cite its source. • “The implications for fluid management in acute respiratory infections have not been studied in any RCTs to date. There is currently no evidence for or against increased fluids in acute respiratory infections. Non-experimental (observational) data suggests that there may be a risk of symptomatic hyponatraemia due to increased antidiuretic hormone secretion in lower respiratory tract infections, particularly in children. The incidence in the primary care setting and the clinical significance of this observational data needs to be determined with further research, conducted as randomised controlled trials.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

12. Long and Detailed Discussions • At least one CRG (Schizophrenia Group) makes efforts to provide implications for multiple audiences (consumers, clinicians, and policy makers). • The Implications for Practice sections from these reviews provide rather lengthy discussions, especially given the lack of eligible studies. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

13. Example of Long and Detailed Discussion "1. For people with severe mental illness At present, the choice between dedicated supported housing schemes and outreach services is based on a combination of personal preference, professional judgement and availability of resources. Decisions of this nature should be made with the full understanding that no one intervention has been shown to be more effective than another in making a difference to symptoms, future use of services, quality of life or other measures of importance. Furthermore, the efficacy of supported housing remains untested. Participating in trials that test the effectiveness of such services should be encouraged. 2. For practitioners In the absence of evidence of their relative efficacy, decisions on the provision of alternative forms of accommodation and continued support for people with mental illness can only be based on a combination of professional judgement, patient preference and availability. This should be made clear to the patient or client who has to make this important decision. Practitioners may wish to actively support or participate in trials to test the effectiveness of supported housing schemes for people with severe mental illness. 3. For managers and policymakers Policies in favour of dedicated supported housing schemes should be viewed with some caution and should not be implemented without plans for evaluation using rigorous methods or should be delayed pending further evidence of their effectiveness. Forming alliances with researchers within this field may result in a fruitful collaboration that would not only inform local policies on this issue, but would also provide much needed evidence base on its effectiveness." The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

14. Implications for Practice with Recommendations • 59 of 376 empty reviews (15.7%) appeared to contain recommendations. • Grouped as: • generally supportive, • recommending alternatives, or • recommending either restricted use or non-use. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

15. Generally Favorable Recommendations • Represent 23 of 59 (39.0%) empty reviews which appear to make recommendations. • 14 of these appear to have been based largely on other research cited in the background, • 7 seem to not be based on evidence, but on guidelines, theory, or absence of evidence of harm. • 2 seem to be based on excluded studies. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

16. Example of Generally Favorable Recommendation • “There is a need to raise awareness of the likely risks of waterpipe tobacco smoking worldwide. Particular attention should be devoted to examining empirically the potentially false perceptions about levels of dependence, disease and mortality associated with waterpipe tobacco smoking. • Evidence-based information about waterpipe should be developed and disseminated in order to deglamourise and denormalise its use. • Young people, especially women and girls in the Eastern Mediterranean Region, may be at higher risk of waterpipe use, and may need particular attention from healthcare workers. • In the absence of policy research on waterpipe, and given what we know about its potential harmful effects, there is a need to include this tobacco use method in all tobacco control policy initiatives (e.g. bans on advertisement and sales to minors, taxation, warning labels, smokefree indoor air). • Treatments of proven efficacy should be provided to help waterpipe smokers quit.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

17. Recommendations to Use Something Else • 8 (13.6%) of 59 empty reviews making recommendations. • Appear to be based on existing guidelines, practice standards, other research, or nothing readily apparent. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

18. Example of Recommendation to Use Something Else “As there were no randomised or quasi-randomised controlled trials identified, we cannot draw any conclusions on the benefits (or otherwise) of regular administration of vitamin A in people with cystic fibrosis. Until further data are available, country or region specific guidelines (e.g. UK CF Trust Nutrition Guidelines (CF Trust 2002)) on the use and monitoring of vitamin A in people with cystic fibrosis should be followed.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

19. Recommendations for Restricted Use • 13 (22.0%) of 59 empty reviews making recommendations. • Limits on use include use in some circumstances (1), possible use in some circumstances (1), minimal use (1), or use only in the context of a controlled trial (10). • Appear to be based largely on evidence of harm, possibility of harm, or no readily apparent basis. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

20. Examples of Recommendations for Restricted Use • Restricted Use “There is no evidence so far that subjective barriers reduce wandering, and the possibility of harm (particularly psychological distress) cannot be excluded. If used, then subjective barriers should form part of a diverse approach to problem wandering, which may include the identification and definition of the problem in the individual, preventative activities such as exercise classes or occupational therapies, and improved communication between carer and wanderer.” • “Restrict to Trials” “…2. For clinicians In the absence of reliable evidence, clinicians prescribing calcium channel blockers for people with tardivedyskinesia must balance the possible benefits against the potential adverse effects of the treatment. Calcium-channel blockers lower people’s blood pressure and may even cause symptoms of tardivedyskinesia to increase. These drugs should really only be used in a situation where their effects are closely monitored, i.e. within a randomised experimental design that can be written up and disseminated in order to inform practice.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

21. Recommendations to Not Use • 15 (25.4%) of 59 empty reviews making a recommendation. • Appear to be based on lack of evidence, evidence of harm, or ongoing research where trials had been aborted. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

22. Example of Recommendation to Not Use “Given the lack of evidence comparing hydralazine versus placebo for primary (essential) hypertension, hydralazine should not be recommended as monotherapy. Clinicians who wish to recommend hydralazine for their patients should understand that while hydralazine may reduce blood pressure (based on non-randomized cross-over trials), to the best of our knowledge, there are no known clinical studies which have associated the use of hydralazine with a reduction in all cause mortality, myocardial infarction, or stroke. In addition, despite poor reporting of treatment-emergent adverse effects, clinicians must weigh the risks of potential serious side effects associated with the use of hydralazine, such as reflex tachycardia, hemolytic anemia, glomerulonephritis, vasculitis, as well as lupus-like syndrome against the potential benefits of blood pressure reduction with no proven beneficial effect on adverse cardiovascular outcomes.” The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

23. Summary • Implications for Practice sections differ widely across empty reviews; • some Implications for Practice sections are very brief, while others are quite lengthy; • some Implications for Practice sections are difficult to interpret or vaguely worded; • many Implications for Practice sections appear to refer to research without a formal citation; and • whilst the majority of Implications for Practice sections do not make recommendations, some do. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

24. Conclusions • Only 15.7% of empty reviews provide recommendations. • Recommendations appear to be based on: • Evidence from studies cited in Background, • Evidence related to adverse effects of the intervention, • Standard practice or practice guidelines, • Theory, and/or • Excluded or ongoing studies. • A substantial proportion of implications for practice sections provide very minimal information, even regarding the scope of the review. The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com

25. Discussion Points Raised by Analysis • What to report in Implications for Practice? • Should evidence discussed in Implications for Practice be specifically cited? • What is the minimum that should be said? • Should evidence of adverse events from excluded studies or other research be mentioned? • Additional points? The Cochrane Empty Reviews Project cochrane.empty.reviews@gmail.com