1 / 13

Implementation Survey Results – Systematic Review Questions

Implementation Survey Results – Systematic Review Questions. Next Steps: Implementation Workshop on Standards for Systematic Reviews and Clinical Practice Guidelines Institute of Medicine Tuesday, May 10, 2011 Presenter: Marguerite Koster Committee on Standards for Systematic Reviews.

Download Presentation

Implementation Survey Results – Systematic Review Questions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Implementation Survey Results – Systematic Review Questions Next Steps: Implementation Workshop on Standards for Systematic Reviews and Clinical Practice Guidelines Institute of Medicine Tuesday, May 10, 2011 Presenter: Marguerite Koster Committee on Standards for Systematic Reviews

  2. Stakeholder Survey Overview • Purpose: Solicit input from stakeholder groups re barriers to and opportunities for implementing the SR/CPG standards. • E-mail with survey link sent to key SR/CPG stakeholders (N=442) • Registered workshop attendees • Members from IOM SR/CPG and “Knowing What Works” (KWW) committees • Workshop attendees and speakers from earlier public SR/CPG/KWW workshops • Follow-up reminder e-mail sent one week before survey closing • Response rate = 8.6% (n=38) • Results may not be generalizable to all workshop attendees

  3. Results: Implementing the SR Standards • More than 80% (25/31) of respondents are considering implementing the SR standards • Most cited reasons not to implement standards: • Don’t have the funding • Don’t have the staff resources • Disagree with standards (too cumbersome, no evidence) • All “non-implementers” stated that the new standards will still impact their organization • Timeline for implementing standards: • 33% within 6 months • 38% within 12 months • 29% longer than 12 months • 61% of respondents report the standards will make them more likely to collaborate with others on SRs

  4. Results: Standards for Initiating the SR

  5. Key Comments: Initiating the SR • User/Stakeholder Input • Limited resources for user/stakeholder engagement unless they are volunteers – limits variety of perspectives • No mechanism to involve patients or consumers in SR process • Early stakeholder involvement “a death wish by 1,000 paper cuts” • Conflict of Interest: • Difficult to identify and address properly; content experts often have conflicts • Rely on self-report of potential COI from participants outside our organization • SR Protocols: • Do PICO, but not full SR protocol – takes additional time/resources • Making protocol publicly available for peer review and responding to comments will be a major burden • No resources to engage a third party to manage peer review

  6. Results: Standards for Finding/Assessing Individual Studies

  7. Key Comments: Finding/Assessing Individual Studies • Searching for evidence • Difficulty finding skilled librarians/information specialists • Will not use limited time on grey literature, unpublished data, or non-English studies; require evidence from peer reviewed journals • Addressing biases in studies • Greatest obstacle is selective outcome reporting & publication bias • Without access to prospectively-registered protocols, it is difficult to know whether we have all of the relevant data • Screening/selecting studies • Dual reviewing will require additional time and resources; will likely affect the number of SRs we are able to conduct with existing resources • Critical appraisal of studies • Critical appraisal of individual studies by each important outcome will require more time and resources

  8. Results: Standards for Synthesizing the Body of Evidence • Key Comments: • Lack of expertise in conducting meta-analyses

  9. Results: Standards for SR Reporting • Key Comments: • Business model involves earning money from SRs...if they are free, what's the new model? • SR publication is up to the journal – cannot ensure that all SRs will be free and open access for the first 12 months • Issues re adherence to confidentiality and journal publication embargoes • Implementing public access through a web site or other venue would require additional time and resources

  10. Key Comments: Ways to Overcome Barriers • Infrastructure support for the labor-intensive tasks; increase efficiency of existing staff • Add more methodologists to the panels; prepare an educational program on conducting SRs and MAs for those and other panelists • Conduct fewer SRs and choose topics more judiciously • May have to start with dual reviewing for only a percentage of abstracts vs. the entire body of abstracts • Will have to be more careful in selection of important outcomes, as each outcome must now be evaluated across the body of evidence • Be transparent about any shortcomings in the SR • Ensure contractors include these steps in their process • Continue to register SRs with Cochrane and Campbell Collaborations • Rigorous COI disclosure; ensure balanced input (i.e.,majority of input from those without COI)

  11. Key Comments: Support/Resources Needed • A trusted third party to manage the peer review process without charge • More help from AHRQ with comparative effectiveness reviews • Access to trial protocols • Database of vetted stakeholders and experts • Training to meet the IOM SR standards, e.g., AHRQ, Cochrane or other organizations that meet the standards could host online/web methodology workshops to assist organizations in meeting standards • Being able to provide input to AHRQ on SR topic selection, especially on topics of importance to healthcare organizations/practicing MDs • More funding/grants to conduct SRs • Wider distribution of standards at the national level to ensure they are recognized and implemented. • More specifics on some standards (methods, timing of public access)

  12. Key Comments: Preferences for Collaboration • Collaboration in composition of review teams • With agreement on IOM SR Standards, adopt other organizations' work and divide the topics for SRs among collaborators • Have a national group (PCORI, AHRQ, etc.) actively sponsor, fund and/or organize a collaborative effort, rather than leaving it to individual groups to organize themselves • Have an ongoing collaboration, involving multiple stakeholder groups, that will agree on topic selection for SR development, methodology standards (to meet IOM reports), sharing of resources, etc. • Work with organizations such as AHRQ and the Cochrane and Campbell Collaborations, to prepare systematic reviews/meta-analyses as a foundation for guidelines, using GRADE methodology • A dedicated, funded collaboration site that can provide the infrastructure for collaboration

  13. Thanks to all survey participants!

More Related