Karen Schoelles, MD, SM ECRI Institute

1. Launching the ECRI Guidelines TrustTM: Challenges and Opportunities in Building A New Repository for Trustworthy Guidelines Karen Schoelles, MD, SM ECRI Institute

2. Disclosure • I have no relevant financial relationships with commercial interests to disclose. • As a matter of fact, I can’t have any as an employee of ECRI Institute!

3. Guidelines Trust™ https://guidelines.ecri.org/ Please register for access at no charge

4. A Bit of History • ECRI Institute ran the National Guidelines and Quality Measures Clearinghouses for AHRQ since the beginning (1997, 2001) • Funding for the Clearinghouses ended July 16, 2018 • We heard from a wide range of stakeholders that they hoped we would do something to provide a guideline repository

5. Sample stakeholder comments “As a medical librarian, I need to find good-quality evidence and information quickly for providers in a hurry. NGC makes my life easier, and … plays a role in my organization's ability to confidently provide good and evidence-based patient care by knowing that we are in line with national guidelines.”

6. Sample stakeholder comments “NGC has been an important part of the Evidence Based Medicine researching process taught and well executed at my institution. Taking it away without a proper replacement would make completing valuable protocols much more difficult.”

7. Sample stakeholder comments “It is a priceless resource. The standards the NGC upholds changed the quality of an association I worked for. We went from consensus guidelines to evidence-based. Producing evidence-based guidelines costs money and if an organization can skate by producing consensus-based guidelines, they have one more reason to do so if no entity is raising the bar.”

8. The Gordon and Betty Moore Foundation • ECRI was awarded a grant from the Gordon and Betty Moore Foundation to develop Guidelines Trust. Users may register for free access to the guideline content. • The Foundation was established to create positive outcomes for future generations. They foster path-breaking scientific discovery, environmental conservation, patient care improvements and preservation initiatives. • The Foundation is funding development of Guideline Briefs, TRUST Scorecards, and monthly newsletters. They are also supporting a Technical Advisory Panel to assist ECRI in their efforts to sustain the repository after the funding period.

12. Guideline Brief Template

15. Clinical Practice Guidelines We Can Trust IOM 2011: Clinical practice guidelines are statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options http://www.nationalacademies.org/hmd/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx

16. Research and Reporting Methods |19 March 2019 Developing and Testing the Agency for Healthcare Research and Quality's National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument J. Jane Jue, MD, MSc; Sarah Cunningham, MA; Kathleen Lohr, PhD; Paul Shekelle, MD, PhD; Richard Shiffman, MD, MCIS; Craig Robbins, MD, MPH; Mary Nix, MS; Vivian Coates, MBA; Karen Schoelles, MD, SM

18. Cross-walk comparison of NEATS items, IOM standards and the AGREE II tool • AGREE II=Appraisal of Guidelines for Research and Evaluation II; CPG=clinical practice guidelines; COI=conflict of interest; GDG=guideline development group; IOM=Institute of Medicine (now the National Academy of Medicine); NEATS=National Guideline Clearinghouse Extent Adherence to Trustworthy Standards. • Annals of Internal Medicine 2019. doi:10.7326/M18-2950.

19. TRUST Scorecard • Transparency and Rigor Using Standards of Trustworthiness • Assesses clinical practice guidelines (CPGs) against the Institute of Medicine (IOM) standards for trustworthy guidelines (2011) • Uses National Guideline Clearinghouse Extent Adherence to Trustworthy Standards (NEATS) Instrument

20. Identifying information Additional information

21. Covers 7 domains, 8 IOM standards • Every CPG with brief also gets scorecard • Evaluation done by trained ECRI reviewers • CPG developers review and can comment or provide additional information

22. The Guidelines Trust went live Nov. 21, 2018 • 61 nationally and internally recognized medical specialty societies and developer organizations have given permission to summarize and assess their guidelines • 389 guidelines with Briefs and Scorecards • 710 additional link outs to guidelines

23. An International Audience • 6,000+ registered users from around the globe. • Registrants are from 75 countries in addition to the U.S.

24. Top 10 Member Audiences

25. Improving Patient Care with Evidence • Clinical decision support (CDS) 5 rights: • Right information • Right person • Right format • Right channel • Right point in workflow

26. Improving Patient Care with Evidence • Clinical decision support (CDS) 5 rights: • Right information • Right person • Right format • Right channel • Right point in workflow • We (the Guidelines Trust) hope to help with • “Right information” • Matched to the right actor • Ability to measure impact

27. What evidence should contribute to the “right information”? • Trustworthy evidence: IOM standards for systematic reviews and clinical practice guidelines • Strong evidence supporting strong recommendations • Conditional recommendations with specification • Current evidence

28. Knowledge Representation Framework – How Far Can We Go? 6 months to a year 6 months to a year Probably not Boxwala AA, Rocha BH, Maviglia S, et al. J Am Med Inform Assoc2011;18:i132-i139.

29. If you are building a guideline repository, why not… • add these steps into our workflow? • Assessing implementability • Creating decision logic • Creating valuesets • Use CQL authoring tool to prepare content for use in CDS and e-Quality Measures

30. A Shared Starting Point Clinical Practice Guideline (Guidelines Trust) Guideline Recommendations Database Defined clinical concepts Links to value sets Semi-formal Measure Semi-formal CDS Formal (or functional) Decision Support Formal (or functional) Quality Measure Clinical Decision Support and Measure Development EHR Registry Adapted from presentations by Dr. Richard Shiffman

31. Guidelines Trust Guideline Recommendation Database Recommendation Recommendation Implementation Ready Recommendations Guidelines Recommendation Processing Encoding Recommendation Recommendation Recommendation Recommendation Recommendation

32. Guideline Recommendations Database • What do we want? • To be able to use recommendations as a unit of analysis • To link similar guideline recommendations • To search for guidelines affecting a population or recommending an action • A shared starting point for Clinical Decision Support and Quality Measures development • How do we want it built? • Populated using a transparent and reproducible process • Scalable and adaptable to changes in content • Using current technology and standards • Able to support future advances in guidelines

33. Support For Aggregate Recommendation Development Source Recommendation: 1 Source Recommendation: 2 Source Recommendation: 3 Subject Matter Expert Panel Source Recommendation: 4 Aggregate Actionable Recommendation Source Recommendation: 5 Source Recommendation: 6 Source Recommendation: 7

34. Where We are Now • We have defined the process for selecting guidelines • We are testing extracting recommendations from similar guidelines • Our IT team is developing the database to house our content and link it to the Guidelines Trust • We are training team members on content extraction and encoding activities • We are mapping GEM to the Guidelines Trust to support automated data migration

35. Thank you! • kschoelles@ecri.org