“Meeting the challenge of maintaining Cochrane Reviews”

“Meeting the challenge of maintaining Cochrane Reviews” Author: Dr David Tovey FRCGPEditor in Chief, The Cochrane Library

Plan • Introduction • Previous Cochrane project • “Fit for purpose” project • Future challenges and opportunities for co-operation • Questions AHRQ 2012 Annual Conference

Acknowledgements • Sally Hopewell, Yemisi Takwoingi, Alex Sutton, Rachel Marshall and Bazian Ltd. • This project was funded by the NIHR Cochrane – NHS Engagement Award Scheme (project number 10/4000/01). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.

Acknowledgements • Cochrane Musculoskeletal Group • Cochrane Pain, Palliative and Supportive Care Group • Cochrane Infectious Disease Group • Cochrane Wounds Group • Cochrane Neonatal Group • Cochrane Airways Group • Karla Soares-Weiser AHRQ 2012 Annual Conference

Introduction: Protecting Archie Cochrane’s vision • “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials” AHRQ 2012 Annual Conference

Introduction: Protecting Archie Cochrane’s vision • “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials” Credible& Safe AHRQ 2012 Annual Conference

Introduction: the size of the challenge AHRQ 2012 Annual Conference

Introduction: the size of the challenge “The boulder in my rucksack” AHRQ 2012 Annual Conference

Previous Cochrane projects: the updating officer project • 8 reviews selected • On average these reviews each took 6.4 months (range 3-11 months) to update from receipt of the search strategy to submission for editorial review. • The main challenges: • lack of familiarity with individual Review Groups’ methods • out of date methodology within the existing reviews • lack of subsequent author commitment • lack of wish for ongoing commitment • Would imply the need for a small army of updating officers!! AHRQ 2012 Annual Conference

“Fit for purpose”project Three objectives of the project: • To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status • To explore whether targeted consultancy could facilitate updating and identify process efficiencies AHRQ 2012 Annual Conference

Methods • Project conducted by Bazian, co-applicants in the NHS engagement award. • Stakeholders in the NHS (clinicians, consumers, commissioners etc.) formed a panel. • The panel had two meetings, and communicated via email before and after meetings.

Results: prioritisation criteria in the tool

Results: testing the tool • The tool was tested on 19 Cochrane Reviews, which found: • There is a level of judgement when assigning scores. • A certain level of knowledge of the field is helpful when assigning scores • Needs to be customised (notable absentee parameters..) • Recommended that the results of using the tool are discussed with external stakeholders including patient and carer representatives.

“Fit for purpose” project Three objectives of the project: • To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status • To explore whether targeted consultancy could facilitate updating and identify process efficiencies AHRQ 2012 Annual Conference

Methods • We refined and amalgamated two complementary methodologies for prioritising systematic review updates: • a qualitative tool based on a broad range of updating signals (Loudon 2008). • formal statistical methods which assess when the inclusion of new studies is likely to change a review’s conclusions (Sutton 2009).

Output: Decision tool

Step 1: Is the clinical question answered or no longer relevant?

Step 2: Are there any new factors to consider?

Steps 3 & 4: Are there new studies? Are the conclusions likely to change?

Statistical prediction tool

About metarank • Based on minimal information on the new evidence • assumes an update strategy is in place such that number of new studies and their sample sizes are known • ‘Signals’ of the need to update implemented as a STATA user-written function • Performs simulation of several meta-analyses, each with one or more new studies of different sizes

Metarank output • Summary of results for each meta-analysis includes details of the original meta-analysis and the signals detected • Table with all reviews in the dataset ranked in order of priority • by a given signal • or the total of all signals triggered • Includes descriptive information for the collection e.g. • average number of trials and participants in the collection of meta-analyses • number of trials in the largest meta-analysis • largest number of participants in a meta-analysis

Advantages of metarank • Assessment of the likelihood of any definable criteria changing in a meta-analysis • Ongoing studies can be taken into account • aid in predicting potential ‘shelf life’ of a review in the light of new or accruing evidence • To provide information with respect to further research needed such as the number of new trials and the number of participants • useful information to end users of reviews in assessing the stability or validity of a review

Limitations of metarank • Relies on availability of some information on new trials • Simplistic approach used in deriving the predictive distribution for new studies • Other issues external to a review may need to be taken into account such as • the rate at which new evidence on a topic evolves • public heath significance etc. • No criteria developed to establish when enough evidence has accrued on a given topic and review that it is deemed decisive and not worth conducting further primary research

Decision tool: summary • The decision tool provides a set of criteria that can be used to assess whether to update a Cochrane Review. • The tool can be applied to a single Cochrane Review or can be used to prioritise a suite of reviews (e.g. those from an individual Cochrane Review Group)

Results: testing the tool • One Cochrane Review Group so far • Cautiously positive - “helpful structure to assess each review’s eligibility for updating”“a transparent way to explain decisions around updating to stakeholders” • Time consuming but might improve with experience • Some assumptions around the statistical tool “ too crude” • Need to have wider experience and use AHRQ 2012 Annual Conference

“Fit for purpose” project Three objectives of the project: • To identify those Cochrane Reviews that NHS stakeholders regard as the most important to update – prioritisation by topic • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status • To explore whether targeted consultancy could facilitate updating and identify process efficiencies AHRQ 2012 Annual Conference

Aims • We aimed to provide short and focused periods of assistance, to incentivise authors, and to address barriers to updating. • The intended approach was to retain both the responsibility for producing the update within the authors, and editorial support for within the CRGs. AHRQ 2012 Annual Conference

Methods • One-year project (July 2010 to July 2011) • Involved 6 CRGs • Each CRG offered 37 days’ of assistance for updating 1 to 6 Cochrane Reviews (between November 2010 and April 2011) • Tasks performed by reviewers from inside and outside Cochrane without specific topic knowledge AHRQ 2012 Annual Conference

Tasks offered to CRGs and authors AHRQ 2012 Annual Conference

Results: assessment of tasks AHRQ 2012 Annual Conference

Results: CRG and author feedback • Evaluation form: all CRGs responded, most authors responded (1 on holiday, 1 no response) • Scale 1-5 (1 greatly improved, 5 greatly worsened) • All authors and managing editors who responded to the question on rolling out an updating service for The Cochrane Collaboration were in favour of an updating service (2 CRGs did not respond) AHRQ 2012 Annual Conference

Results: status of updates • We worked on 14 Cochrane Review updates • Mean time spent: 35.5 hours (median 26.71 hours, range 4.5 to 109.75 hours) • Lots of activity but.... • In October 2011, no updates yet published AHRQ 2012 Annual Conference

Future challenges: • Whose priorities? • Expectations and the increasing professionalisation of systematic reviews • The rise and rise of methods advances • How to balance updates versus new reviews? • Authorship of updates AHRQ 2012 Annual Conference

Conclusion • Updating is critical to patient safety and credibility • Increasing recognition of need to prioritise (but this isn’t as easy as it seems) • Transparency to user/reader is critical • Different approaches but no “one size fits all” solution AHRQ 2012 Annual Conference

Conclusion: How can we work more effectively together? • Sharing intelligence • “Keep up” initiative • Shared surveillance • Sharing the workload • Sharing data • “Wiki” approaches • Novel approaches • Use of data mining and semantic technologies • Limited search and “Summary updates” AHRQ 2012 Annual Conference

thank you for listeningdtovey@cochrane.org AHRQ 2012 Annual Conference

Authorship of updates • Scenario: A review is updated that originally included 11 studies and now has 17. The text includes 35% the same text as the previous version, across all sections. In addition a summary of findings table has been added.The author team has changed completely and none of the previous authors have contributed to the update. • Questions How should the work of the previous authors be credited? AHRQ 2012 Annual Conference

ICMJE guidance on contributorship Authorship credit should be based on • Substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; • Drafting the article or revising it critically for important intellectual content; and • Final approval of the version to be published. Authors should meet conditions 1, 2, and 3. AHRQ 2012 Annual Conference

Authorship of updates • As authors of the updated review • Lead author included on the update • Should be negotiated on an individual basis • Original authors not included as authors but acknowledged within the review • Some other approach AHRQ 2012 Annual Conference

“Meeting the challenge of maintaining Cochrane Reviews”