Implementation Survey Results – Systematic Review Questions

1. Implementation Survey Results – Systematic Review Questions Next Steps: Implementation Workshop on Standards for Systematic Reviews and Clinical Practice Guidelines Institute of Medicine Tuesday, May 10, 2011 Presenter: Marguerite Koster Committee on Standards for Systematic Reviews

2. Stakeholder Survey Overview • Purpose: Solicit input from stakeholder groups re barriers to and opportunities for implementing the SR/CPG standards. • E-mail with survey link sent to key SR/CPG stakeholders (N=442) • Registered workshop attendees • Members from IOM SR/CPG and “Knowing What Works” (KWW) committees • Workshop attendees and speakers from earlier public SR/CPG/KWW workshops • Follow-up reminder e-mail sent one week before survey closing • Response rate = 8.6% (n=38) • Results may not be generalizable to all workshop attendees

3. Results: Implementing the SR Standards • More than 80% (25/31) of respondents are considering implementing the SR standards • Most cited reasons not to implement standards: • Don’t have the funding • Don’t have the staff resources • Disagree with standards (too cumbersome, no evidence) • All “non-implementers” stated that the new standards will still impact their organization • Timeline for implementing standards: • 33% within 6 months • 38% within 12 months • 29% longer than 12 months • 61% of respondents report the standards will make them more likely to collaborate with others on SRs

4. Results: Standards for Initiating the SR

5. Key Comments: Initiating the SR • User/Stakeholder Input • Limited resources for user/stakeholder engagement unless they are volunteers – limits variety of perspectives • No mechanism to involve patients or consumers in SR process • Early stakeholder involvement “a death wish by 1,000 paper cuts” • Conflict of Interest: • Difficult to identify and address properly; content experts often have conflicts • Rely on self-report of potential COI from participants outside our organization • SR Protocols: • Do PICO, but not full SR protocol – takes additional time/resources • Making protocol publicly available for peer review and responding to comments will be a major burden • No resources to engage a third party to manage peer review

6. Results: Standards for Finding/Assessing Individual Studies

7. Key Comments: Finding/Assessing Individual Studies • Searching for evidence • Difficulty finding skilled librarians/information specialists • Will not use limited time on grey literature, unpublished data, or non-English studies; require evidence from peer reviewed journals • Addressing biases in studies • Greatest obstacle is selective outcome reporting & publication bias • Without access to prospectively-registered protocols, it is difficult to know whether we have all of the relevant data • Screening/selecting studies • Dual reviewing will require additional time and resources; will likely affect the number of SRs we are able to conduct with existing resources • Critical appraisal of studies • Critical appraisal of individual studies by each important outcome will require more time and resources

8. Results: Standards for Synthesizing the Body of Evidence • Key Comments: • Lack of expertise in conducting meta-analyses

9. Results: Standards for SR Reporting • Key Comments: • Business model involves earning money from SRs...if they are free, what's the new model? • SR publication is up to the journal – cannot ensure that all SRs will be free and open access for the first 12 months • Issues re adherence to confidentiality and journal publication embargoes • Implementing public access through a web site or other venue would require additional time and resources

10. Key Comments: Ways to Overcome Barriers • Infrastructure support for the labor-intensive tasks; increase efficiency of existing staff • Add more methodologists to the panels; prepare an educational program on conducting SRs and MAs for those and other panelists • Conduct fewer SRs and choose topics more judiciously • May have to start with dual reviewing for only a percentage of abstracts vs. the entire body of abstracts • Will have to be more careful in selection of important outcomes, as each outcome must now be evaluated across the body of evidence • Be transparent about any shortcomings in the SR • Ensure contractors include these steps in their process • Continue to register SRs with Cochrane and Campbell Collaborations • Rigorous COI disclosure; ensure balanced input (i.e.,majority of input from those without COI)

11. Key Comments: Support/Resources Needed • A trusted third party to manage the peer review process without charge • More help from AHRQ with comparative effectiveness reviews • Access to trial protocols • Database of vetted stakeholders and experts • Training to meet the IOM SR standards, e.g., AHRQ, Cochrane or other organizations that meet the standards could host online/web methodology workshops to assist organizations in meeting standards • Being able to provide input to AHRQ on SR topic selection, especially on topics of importance to healthcare organizations/practicing MDs • More funding/grants to conduct SRs • Wider distribution of standards at the national level to ensure they are recognized and implemented. • More specifics on some standards (methods, timing of public access)

12. Key Comments: Preferences for Collaboration • Collaboration in composition of review teams • With agreement on IOM SR Standards, adopt other organizations' work and divide the topics for SRs among collaborators • Have a national group (PCORI, AHRQ, etc.) actively sponsor, fund and/or organize a collaborative effort, rather than leaving it to individual groups to organize themselves • Have an ongoing collaboration, involving multiple stakeholder groups, that will agree on topic selection for SR development, methodology standards (to meet IOM reports), sharing of resources, etc. • Work with organizations such as AHRQ and the Cochrane and Campbell Collaborations, to prepare systematic reviews/meta-analyses as a foundation for guidelines, using GRADE methodology • A dedicated, funded collaboration site that can provide the infrastructure for collaboration

13. Thanks to all survey participants!