1 / 56

Comments on Cochrane Reviews

Comments on Cochrane Reviews. Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City. Introductions. We are: John Hilton Editor, Cochrane Editorial Unit Gavin Stewart Associate Editor, Wiley Jo Garner Senior Business Analyst, Wiley

harper-glass
Download Presentation

Comments on Cochrane Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comments on Cochrane Reviews Approaches to managing feedback WORKSHOP 22 September 2013 Cochrane Colloquium, Quebec City

  2. Introductions We are: • John Hilton Editor, Cochrane Editorial Unit • Gavin Stewart Associate Editor, Wiley • Jo Garner Senior Business Analyst, Wiley • Toby Lasserson Senior Editor, Cochrane Editorial Unit Feedback Editor, Airways Group + 19 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

  3. Introductions We are: • John Hilton Editor, Cochrane Editorial Unit • Gavin Stewart Associate Editor, Wiley • Jo Garner Senior Business Analyst, Wiley • Toby LassersonSenior Editor, Cochrane Editorial Unit Feedback Editor, Airways Group + 19 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

  4. Introductions We are: • John Hilton Editor, Cochrane Editorial Unit • Gavin Stewart Associate Editor, Wiley • Jo Garner Senior Business Analyst, Wiley • Liz Dooley Managing Editor, ARI Group ME Support + 21 other roles Disclosures: Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation. Who are you?

  5. Workshop • Part 1:What’s the latest? • Overview, data, guidance, support • What do you need? What do you need to know? • Part 2: Case studies • Some real-life examples • What would you do? And why? Any barriers, hurdles or brick walls? • Part 3: The way ahead • Improving the process and the display • How can you help? • Part 4: Is it worth it? • Isn’t everyone using Twitter these days? • What do you think?

  6. Submitting comments

  7. Submitting comments

  8. Submitting comments

  9. Submitting comments • Comments are also submitted via other channels: • Direct contact to CRG • From OVID • Direct contact to Editor in Chief or CEU • Summaries website (not at the moment)

  10. Processing comments • What does CEU do? • Comments go to Editor in Chief, and two editors • EiC looking if intervention required (with CRG) • Editors looking for cross-CRG issues (e.g. multiple submissions) • Support • Tracking

  11. Processing comments • What gets filtered out: • Comments or enquiries about the website functionality, access, display, etc • Comments that are offensive, nonsensical, or inconsequential • Comments that arise from system errors or system testing • Duplicates

  12. Processing comments •  CRG-led process • Managing Editor • Feedback Editor • Co-ordinating Editor(s) • Feedback contributor(s)

  13. Processing comments •  CRG-led process • Managing Editor • Feedback Editor • Co-ordinating Editor(s) • Feedback contributor(s) • Author(s) • Editor(s)

  14. Processing comments •  CRG-led process • Managing Editor • Feedback Editor • Co-ordinating Editor(s) • Feedback contributor(s) • Author(s) • Editor(s) • Editor-in-Chief • CEU Editor(s) • Steering Group, CEO • Interfering third parties

  15. Acting on comments • Questions, questions, questions • Is it in English? • Is it coherent and not clearly wrong? • Is it about the content of the review? • Should it be published? • How much editing does it need? • Are the authors going to respond? And when? • If authors won’t or can’t respond, what then? • How long will the whole process take? • Is anyone in editorial base conflicted? • Has contributor declared conflicts? • Does it trigger an amendment? • Does it trigger an update? • Does it raise broader issues?

  16. Acting on comments • Questions, questions, questions • Is it in English? • Is it coherent and not clearly wrong? • Is it about the content of the review? • Should it be published? • How much editing does it need? • Are the authors going to respond? And when? • If authors won’t or can’t respond, what then? • How long will the whole process take? • Is anyone in editorial base conflicted? • Has contributor declared conflicts? • Does it trigger an amendment? • Does it trigger an update? • Does it raise broader issues? • What have I missed? Process Judgement call? Workflow?

  17. Comment outcomes • Validity • Valid: take action • Not valid: take no further action • Usefulness • Translate • Edit • Drastically cut or ask for resubmission • Suitability for publication • Publish • Don’t publish • Author response • Not needed • Needed but not forthcoming • Received and published • Action on review • No action needed • Review amendment • Review update • Changes at next update Communication

  18. How many comments?

  19. Which groups? • In the first half of 2013…

  20. Which groups? • In the first half of 2013… 4 Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group

  21. Which groups? • In the first half of 2013… 4 Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group 6 Airways Group

  22. Which groups? • In the first half of 2013… 4 Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds Group 6 Airways Group 8 Acute Respiratory Infections Group

  23. Which groups? • In the first half of 2013… 4 Anaesthesia Group Heart Group Pregnancy & Childbirth Group Wounds 6 Airways Group 8 Acute Respiratory Infections Group And 32 other CRGs received at least 1 comment

  24. Characteristics of comments • Study inclusion/exclusion • Interpretation of findings • Conflict of interest • Scope • Methods • Typos, errors

  25. Characteristics of comments • Study inclusion/exclusion • Interpretation of findings • Conflict of interest • Scope • Methods • Typos, errors One sentence 10 pages Detailed, well-written referenced letter Semi-coherent scribble Focused on single aspect of review Multiple issues, not all related to review

  26. Characteristics of comments • Study inclusion/exclusion • Interpretation of findings • Conflict of interest • Scope • Methods • Typos, errors One sentence 10 pages Detailed, well-written referenced letter Semi-coherent scribble Focused on single aspect of review Multiple issues, not all related to review • Retired academics • Cochranites • Students • Clinicians • Pharmaceutical companies or manufacturers • Persistent single-issue enthusiasts

  27. Where do comments come from?

  28. What about the unapproved comments? Example 1 Review Speed cameras for the prevention of road traffic injuries and deaths ContributorMaller Comment When there is accident happened, who will take responsibility? After several day or long time, how to know the scene when it happened? Car video recorder could record accident and store video data with gps information on memory card,  people could get accident reconstruction, it is very easy. more view: http://www.cheapcarcamera.com

  29. What about the unapproved comments? Example 2 Review Contributorwillywonka@gmail.com Comment

  30. What about the unapproved comments? Example 3 Review Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children ContributorRachel Bock Comment Test comment - please delete

  31. Known unknowns? • Who is submitting comments: students, clinicians, Cochranites, pharma companies? (and why?) • How many comments are published, and how long after submission? • How many reviews amended or updated as a result of feedback? • What aspects are commentors focusing on? (study inclusion, interpretation, outcomes, etc?) • Any more? What would it be useful to know? • ANY VOLUNTEERS?

  32. Cochrane Editorial & Publishing Policy Resource http://www.cochrane.org/editorial-and-publishing-policy-resource

  33. Cochrane Editorial & Publishing Policy Resource http://www.cochrane.org/editorial-and-publishing-policy-resource

  34. Cochrane Editorial & Publishing Policy Resource http://www.cochrane.org/editorial-and-publishing-policy-resource

  35. Cochrane Organisational Policy Manual http://www.cochrane.org/organisational-policy-manual

  36. Cochrane Handbook http://handbook.cochrane.org

  37. Cochrane Editorial Unit website http://www.editorial-unit.cochrane.org/cochrane-library-feedback

  38. So what’s in those new guidelines? • Guidance on how feedback should be managed • When to publish, when not to publish • Guidance on reasonable timescales • Roles of ME and FE • Meeting expectations of contributors • Where to get support • What to do with non-English comments Your feedback will be very welcome.

  39. Feedback editors: neglected? • Communication to and between FEs has been lacking • Need Role description (purpose, expectations) • Induction package? • Page on Cochrane Training page? • Independence of role • Cochrane.orgfeature • Forum

  40. So what else do you/we need?

  41. Case studies • 1. • 2. • 3. • 4.

  42. What’s in that CEU proposal? • Use the term ‘comments’ • Decouple comments from review, but link to version • Display comments parallel to review version • Make comments citable (DOI) • Expectations about reasonable turnaround time • Transparency and tracking • Archie workflow for comments So that was a while ago. What happened?....

  43. Cochrane-Wiley Roadmap • A set of work streams fulfilling agreed contractual projects • Next 18 months • Feedback scheduled for Q2 2014 • Project will involve: • CEU, Wiley, Cochrane Tech team • CRGs • You?

  44. Goals for feedback project • Make process more efficient and transparent • Phased approach to first improve the user experience for: • the Contributor • the Reader of feedback • De-couple publishing feedback from the review or protocol • Make the individual comment citable in its own right

  45. Requirements for user feedback • A Contributor is able to submit a comment • The system shall save and acknowledge the comment • Wiley Editorial may moderate the comment • The system shall trigger an editorial workflow that may result in publication of the comment along with a response • A Reader will be able to view comments and responses • A Reader will be able to cite the comment • A Search User will be able to search comments and responses

  46. Proposed approach – submission

  47. Proposed approach – submission

  48. Proposed approach – display

  49. Improve processing • A system for moderation • Separate the publishing of feedback from the review, retain a link between them: • Feedback as a new Content type? • Changes to Archie, through XDPS and to Online Library • Add the ability to cite individual comments & author responses • Improve the robustness of the feedback-response loop • Feedback items having their own workflow?

  50. Why all the bother? • So we can “…amend reviews in the light of new evidence…to reflect the emergence of new data, valid feedback, solicited or unsolicited, from whatever source” • Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 1994;209:862-5.

More Related