Can Consumers Improve the Quality of Cochrane Reviews?

Can Consumers Improve the Quality of Cochrane Reviews? Carol Grant-Pierce, Carol Sakala, Dell Horey Gill Gyte, Sonja Henderson and Sandy Oliver 12th Cochrane Colloquium Ottawa, Canada October 2-6, 2004

Cochrane Reviewers’ Handbook:10.2 Consumer Involvement Essential, to help ensure that reviews: • are targeted at problems important to people • take into account outcomes important to people • are accessible to people making decisions • adequately reflect variability in values, conditions & healthcare in different countries Cochrane Pregnancy and Childbirth Group

Consumer Participation in Research • Growing involvement of consumers in Cochrane and other research • Growing literature on consumers and research • Cochrane Review: Editorial peer-review for improving the quality of biomedical studies Cochrane Pregnancy and Childbirth Group

Pregnancy & Childbirth Group Consumer Panel: Background Established 1998 Since January 1999, comments on all protocols & reviews going through editorial process: over 300 documents altogether 1st 2 years described in Birth 2001;28(2):133-7 For current editorial process, see: The Editorial Team. Pregnancy and Childbirth Group. About the Cochrane Collaboration (Collaborative Review Groups (CRGs)). In: The Cochrane Library 2004, Issue 3. Cochrane Pregnancy and Childbirth Group

Cochrane Pregnancy & Childbirth Group Editorial Process Contact Editor Authors Start of Process Draft Protocol/Review • Referees • Internal peer • External peer • Statistician Office Office • Consumer Panel • Consumer Panel Coordinator • Regional Consumer Coordinators • EuropeAustralasiaNorth AmericaLow-, mid-income nations • 4512104 • Active Consumers

PCG Consumer Panel:Why Evaluate? • Assess impact & resource allocation • Improve administrative process • Provide feedback to PCG participants • Obtain needed resources in future • Improve quality of Cochrane & external research Cochrane Pregnancy and Childbirth Group

Consumer Panel Evaluation:Phase I Interviews with 4 Editors, Review Group Coordinator, 5 authors, 5 consumers, all 3 Consumer Coordinators What is their assessment of • whether the quality of reviews is improved? • how the process works? With support from Cochrane Collaboration Steering Group Discretionary Fund Cochrane Pregnancy and Childbirth Group

Editor & Author Results:FavorableAssessment of Impact • Strong overall assessment • Adds value to professional refereeing • Identifies overlooked outcomes • Improves accessibility • Helps represent range of perspectives Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Favorable Assessment of Impact Editor: “The input in general is fantastic. It’s the highest quality we get.” Editor: “As an editor, well there’s no doubt some of the best, most constructive comments about the review come from the Consumer Panel.” Author: “I think we get excellent constructive feedback on the reviews.” Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Favorable Assessment of Impact Author: “Certainly in terms of outcome measures that were felt important to consumers, … we hadn’t actually considered those as important outcomes and in fact, when we thought about it, they were.” Author: “The language, will be accessible to someone who is not doing medicine all the day.” Editor: “Some of the approaches the consumers guide us into thinking through are really probably new … from a health profession point of view.” Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Concerns About Impact • Input with “agenda” or emotionality perceived as less generally relevant • Outcomes may not be perceived as valid if not already in literature • Unwieldy to make all technical information comprehensible to general audience Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Concerns About Impact Author: “Well the only negative [comment] is that I don’t think you can explain absolutely everything so that absolutely everyone can understand it.” Editor: “It’s just been this problem of sometimes emotionalism that gets in the way.” Author: “There is a real tension about who one is writing this for; … on the whole, when I’m writing, I think I’m writing this for other doctors.” Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Favorable Assessment of Process • Dedicated and skillful coordinator works well with consumers and professionals • Helpful to have summaries that highlight key points, along with full detailed collated feedback Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Favorable Assessment of Process Author: “I think [the consumers] take it very seriously. … they’re not destructive.” Author: “It’s good to actually get a lot of criticism.” Author: “We do get … very constructive comments by and large, and also a good summary from [coordinator], and … it’s generally quite easy to respond to.” Author: “The feedback was very professional, and professionally handled.” Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Favorable Assessment of Process Editor: “It works well because we’ve got a really good coordinator…. a dedicated person that can coordinate & work with consumers, who understands what they are saying at the same understands what the review needs.” Editor: “[Feedback is] very ably summarized. We have a covering note that reminds us that even though we might have a long list, it’s about the following, of issues that need to be considered & dealt with…. Teasing out from all the information there what are really the key points is helpful for a reviewer & editorial process. Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Concerns About Process • Volume of full collated feedback can be overwhelming • Some feedback difficult to address if first available later in process • Inadequate input from low-income countries Cochrane Pregnancy and Childbirth Group

Editor & Author Results:Concerns About Process Author: “I think it’s simply my memory of these 18 pages arriving and just thinking, ‘Can’t do this.’” Editor: “If the consumer said at the protocol stage, there are a number of important things missing … I would ask the reviewers to think seriously about including [them]…. If it was at the review stage, I probably wouldn’t draw attention to that. Editor: “I know it’s been a struggle all along to get input from women in low-income countries.” Cochrane Pregnancy and Childbirth Group

Phase 1 Evaluation: Summary • Apart from helping to identify review topics, appears to contribute to Handbook objectives • Skill mix of Principal Consumer Coordinator central to success, including provision of summary cover note, diplomacy and content knowledge • Large volume of feedback can be both strength and challenge • Need to develop better outreach to consumers in low- & middle-income countries Cochrane Pregnancy and Childbirth Group

Consumer Panel Evaluation:Phase II Content analysis to assess whether consumer input • adds value to input from authors & peer referees • improves quality of final review RCTs evaluating different models of consumer participation Cochrane Pregnancy and Childbirth Group

For Information, To Participate, or To Obtain Copy of Slides Gill Gyte PCG Principal Consumer Coordinator (+ Europe, low- & middle-income countries) ggyte@cochrane.co.uk Carol Sakala PCG Consumer Coordinator for North America sakala@maternitywise.org Dell Horey PCG Consumer Coordinator for Australia dhorey@ozemail.com.au Cochrane Pregnancy and Childbirth Group

Can Consumers Improve the Quality of Cochrane Reviews?