New methodological standards for Cochrane reviews first output from the MECIR project

1. edit New methodological standards for Cochrane reviews first output from the MECIR project Rachel Churchill Co-ordinating Editor representative on Steering Group School of Social and Community Medicine, University of Bristol Julian Higgins Methods Groups representative on Steering Group MRC Biostatistics Unit, Cambridge, UK & Centre for Reviews and Dissemination, University of York

2. Outline • Background and purpose of MECIR • Development of methodological standards for conduct of intervention reviews • The first set of standards • Some examples • Next steps and discussions around implementation

3. Background • Coordinated by MARS (Methods Application and Review Standards) • Co-convened by Julian Higgins and Rachel Churchill to facilitate interaction between Methods Groups and CRGs; supports Editor in Chief and CEU • Coordinating Group from MARS and the CEU (Rachel Churchill, Julian Higgins, Jackie Chandler, David Tovey and Toby Lasserson)

4. MECIR • Stands for Methodological Expectations of Cochrane Intervention Reviews • Purpose is • to specify methodological expectations for Cochrane protocols, reviews, and updates of reviews on the effects of interventions • to ensure that these methodological expectations are supported and implemented across the Collaboration

5. Developing a question and deciding the scope of the review • Searching for studies • Selecting studies and collecting data • Assessing risk of bias in studies • Analysing data and undertaking meta-analyses • Interpretation and presenting results • All registered entities (16/6/11 – 8/8/11) • 51% responded • - CRGs (35/53) • - MGs (8/15) • - Centres (5/14) • - Centre branches (2/15) • - Fields (5/11) • - CEU, TWG, Author rep MECIR Working Groups Collaboration-wide consultation 1. WGs submittedlong list of items 2. WGs asked to consider Mandatory itemsvs Highly Desirablevs Good practice and Rationale for each Development process Coordinating Group

8. Further work by Coordinating Group, before extensive consultation and feedback from all Cochrane entities All registered entities (16 June ‘11 – 8 August ‘11) 51% entities responded CRGs (35/53) MGs (8/15) Centres (5/14) Centre branches (2/15) Fields (5/11) CEU, TWG, Author rep Consultation process

9. Consultation feedback (Aug – Oct 2011) • High levels of support for most items • Decision around status item description, expectation and rationale (merging or omitting as appropriate) and provided associated Handbook reference for each standard • 100 standards, reduced to 80 • Provided responses to comments and final decision regarding status, description, and rationale for each item (see Annex 2) www.editorial-unit.cochrane.org/mecir

10. Summary of MECIR standards • 80 standards for the conduct of CIRs structured around: • Setting research question to inform the scope of the review • Setting the eligibility criteria for including studies in the review • Selecting outcomes to be addressed • Planning the review methods • Searching for studies • Selecting studies into the review • Collecting data from included studies • Assessing risk of bias in included studies • Synthesizing the results of included studies • Summarizing the findings • Reaching conclusions

11. Some examples

12. Example of one we got right

13. Pre-defining clear and unambiguous criteria for interventions and comparators (standard 7) Define in advance the eligible interventions and the interventions against which these can be compared in the included studies (Mandatory) Agreed/Disagreed: • CRGs: 30 / 0 • Centres: 7 / 0 • Fields: 4 / 0 • Methods Groups: 6 / 0 • Left unchanged

14. Example of one we had to work on

15. Excluding studies without useable data (standard 40) Do not exclude studies on the basis of there being no usable data (either because the data are inconvenient format or because they are not reported at all) [Draft] (Mandatory) Agreed/Disagreed: • CRGs: 23 / 5 • Centres: 6 / 1 • Fields: 4 / 0 • Methods Groups: 6 / 0 • Three suggestions to change to Highly Desirable

16. Excluding studies without useable data (standard 40) We will not implement this in our group Hugely important RevMan will protest if a study is ‘included’ with no data at all somewhat confusing agree if the outcomes of interest were measured as an outcome Clarify that this relates to the formal “exclusion” of studies from the review should be rephrased

17. Excluding studies without useable data (standard 40) (Mandatory) Include studies in the review irrespective of whether measured outcome data are reported in a ‘usable’ way.

18. Some key ones

19. Some key conduct standards • Consider important potential adverse effects: Mandatory • Clarify role of outcomes: Mandatory • Search trials registers: Mandatory • Double (independent) data extraction: Mandatory • Risk of bias tables: Mandatory • Summary of findings tables: Highly desirable • The five GRADE considerations (study limitations, consistency of effect, imprecision, directness and publication bias) to assess the quality of the body of evidence: Mandatory

20. The new stuff

21. Standards not currently in the Handbook • Rerun or update searches for all relevant databases within six months before publication of the review or review update, and screen the results for potentially eligible studies: Mandatory, BUT • Incorporate fully any studies identified in the rerun or update of the search within six months before publication of the review or review update: Highly Desirable • Compare magnitude and direction of effects reported by studies with how they are presented in the review, taking account of legitimate differences: Mandatory • Consider in advance whether issues of equity and relevance of evidence to specific populations are important...: Highly desirable

22. Other MECIR outputs • Guidance on implementation of standards for updates to be produced • to be produced • Reporting standards (for protocols, reviews, updates) • in draft • Working Groups to be invited to input • consultation: Collaboration and external (users) • Good practices and Common errors • in draft • will continually evolve

23. Facilitating implementation A mixture of • Distribution to entities and individuals • Handbook amendments • prominent incorporation of standards throughout • extension/clarification of guidance as appropriate • Training materials (Training Working Group) • Check lists for authors, editors, peer reviewers • building on existing author checklist (Editorial Resources Committee) • Modifications to RevMan • Baseline audit

24. Concluding remarks • MECIR conduct standards are (almost all) clarifications of existing Handbook guidance • Adopting the standards should ensure that Cochrane intervention reviews are consistently of high quality • We hope you will support and implement them • and contribute to development of further expectations, and future evaluations of them all • The standards will be updated over time • For more, see www.editorial-unit.cochrane.org/mecir • Finally, THANK YOU to all individuals and entities who have contributed so helpfully so far to the development of the expectations...

25. Particular thanks to Working Groups Rachel Marshall Ruth Mitchell Donal O’Mathuna Anna Noel-Storr Georgia Salanti (Methods lead, WG5) Doug Salzwedel Margaret Sampson Jelena Savovic Holger Schünemann (Methods lead, WG 6) Ian Shemilt Nandi Siegfried Jonathan Sterne (Methods lead, WG4) Britta Tendal (Methods lead, WG3) David Tovey Peter Tugwell Lucy Turner Claire Vale Julia Walters Helen Worthington (Co-Eds lead, WG 5 and Co-Eds co-lead, WG6) Janelle Yorke Doug Altman Mohammed Ansari (Methods lead, WG1) Sally Bell-Syer Patrick Bossuyt Deborah Caldwell Christopher Cates Rachel Churchill (Co-Eds lead, WG4) Mike Clarke (Co-Eds co-lead, WG2) Jan Clarkson (Co-Eds co-lead, WG6) Philippa Davies Marina Davoli (Co-Eds lead, WG1) Ruth Foxlee Chantelle Garritty Davina Ghersi (Co-Eds co-lead, WG2) Julie Glanville (Methods co-lead, WG2) Peter Herbison Julian Higgins Sophie Hill (Co-Eds lead, WG3) Toby Lasserson Edith Leclercq Carol Lefebvre (Methods co-lead, WG2) Jessie McGowan