Theme 5. Cochrane Reviews: updating

1. Theme 5. Cochrane Reviews: updating • 9:30 – 10:10 Theme overview and CRG experiences of using tools/frameworks • 10:10 – 10:30: Q&A • 10:30 – 11:00 Working groups • 11:00 – 11:15 Feedback

2. Overview • How often should Cochrane Reviews be updated? • Which Cochrane Reviews should be a priority for updating? • How should updates of Cochrane Reviews, and Cochrane Reviews no longer being updated, be classified in The Cochrane Library?

3. Updating frequency • CRGs are not able to update all Cochrane Reviews every two years. The reasoning behind updating decisions is not transparent to readers. • The burden of updating is ever increasing • Cochrane Reviews may become out of date at different times • Updating too soon may introduce bias • 80% of 617 respondents moderately or strongly supported a move from the current policy of aiming to update all Cochrane Reviews every two years to a ‘needs-based’ approach

4. Prioritising Cochrane Reviews

5. Prioritising Cochrane Reviews

6. Classification framework

7. Experiences

8. Original criteria

9. Revised criteria

10. Cochrane Airways Group Prioritisation exercise using the decision flowchart and STATA program

11. Context • ~250 reviews • 166 reviews over 2 years out of date • Don’t have a prioritisation policy and make decisions to update based on author availability • Looking to: • make decisions transparent • direct limited resources at most important topics

12. What we did • Considered 21 reviews with the highest number of hits and cites for prioritisation • TSC ran searches • ME/TSC/RA screened titles and abstracts • Used prioritisation flow chart and STATA tool • Extracted outcome data for the STATA tool where possible even if review currently being updated

13. Flow chart

14. Issues with flow chart • Seemed very complicated at first, but was OK to use • We created a spreadsheet to reflect the flow of work from the decision tool and record comments on reviews • We didn't go searching for abstracts or papers if they weren't available; however we felt that the new studies identified represent the bulk of new research for the review • Some updates were already in progress and we consulted authors to try to get the number of hits

15. Innovated table to record decisions

16. Issues with STATA tool Possibly too crude? • need to select a single outcome • extract outcome data often from abstract only. • Unresolved issues with continuous data and GIV • However, once data had been extracted the STATA tool was easy to use and may find application in some circumstances

17. Reviews that reached stage x in the flowchart • Stage 1: 4 review questions appeared answered or partly answered • Stage 2: 13 reviews needed some sort of work. 7 were unclear. Most needed SOF • Stage 3: We did searches for all of them to test out the STATA tool (so could save time here in future). Some searches challenging to go through • STATA: we got sufficient data to run 6 studies through STATA tool but could not analyse the continuous data or the GIV data for unresolved technical reason

18. Outcomes • 8 indicated for update for other reasons, 5 possible updates and 8 not to be updated (already up-to-date, irrelevant review question, not enough new data) • We didn’t feel that the STATA tool was worth the additional workload and that it was too crude for our needs, however now we know how it works, may use in future

19. Issues/comments • Estimate it took around 5 days in total to do this work. Plus 2 hours with a statistician on STATA. • This is a reasonable amount of time compared to the editorial process for an ill-conceived update! • The flow-chart tool provided • a helpful structure to assess each review • a transparent way to summarise evidence to stakeholders

20. Issues/comments • Other groups may need to modify tool and consider which of their reviews to put through the tool • We would need a further conversation with an advisory board on the obtained information to rank them/decide relevance of review question • Need a transparent way to select reviews for this tool • With more time, may be possible to screen references carefully enough to be of use to authors

21. Classification framework pilot Objectives • To assess whether CRGs other than the CIDG could classify their Cochrane Reviews using the framework • To identify any problems with the framework and potential solutions • To determine how long it takes CRGs to classify Cochrane Reviews Methods • CRGs who nominated themselves classified a sample of 20 Cochrane Reviews selected by CEU, and fed back experiences

22. Results and next steps Results • Thirteen CRGs completed the pilot • CRGs were able to classify Cochrane Reviews of interventions, and found the process manageable and useful • CRGs faced several issues when making judgements, and we adjusted guidelines and guidance • CRGs varied in the amount of time they took to classify Cochrane Reviews Next steps • Consider whether framework is suitable for non-intervention reviews • User test framework on readers of CDSR, with mock-ups of potential views • Implementation

23. Queries from CRGs • How should Cochrane Reviews be classified if there is more than one comparison, and the comparisons have different categories? • How should a Cochrane Review be classified if trials are ongoing? • Could there be more subdivisions in the categories, for example, to highlight whether an author is or isn’t available? • When should a Cochrane Review be classified as up to date? If there are no new studies, can the Cochrane Review be labelled ‘Current, Up to date’? And what should happen if a few small studies are identified that are unlikely to change the conclusions? • Is the term ‘Current, Up to date’ misleading when a full systematic update search as not been done? • How frequently should these statuses be applied? • For the classification ‘Current, No update intended’, how should CRGs make this judgement? • Should there be a final search before a Cochrane Review is classified as ‘Historical, No update intended’. • Should authors be consulted before making a judgement about the classifications?

24. Revised framework

25. Revised framework in light of queries

26. Recommendations “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” “Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...” “Ensure decisions around prioritisation are transparent to readers.” “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

27. Working groups Group 1: “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” AND “Ensure decisions around prioritisation are transparent to readers.” Group 2:“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...” Group 3: “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

28. To consider in working groups • Agree/disagree? Consensus? • If agreement, high or low priority (compared with other strategic session paper recommendations)? How urgent? • If disagreement, tweak, rework, or discount? • Implementation? • Changes to tools or framework? • Affect on CRGs’ monitoring? • Changes to Review Manager and Archie? • Changes to The Cochrane Library website? • Changes to CRG entity websites? • Changes to resources (e.g. Cochrane Handbook)? • Barriers? • Methods? • External perceptions?