Evaluation of the National Guideline Clearinghouse™

1. AHRQ Annual Conference September 2011 Evaluation of the National Guideline Clearinghouse™

2. Acknowledgements Thank-you to: • Survey respondents • Focus group participants • Key informants • AMIA and ACCP • AHRQ • Mary Nix • Judi Consalvo • Gov Delivery Staff • Marjorie Shofer • HIT Portfolio • AMA and AHIP • ECRI Institute • NGC EvaluationPET • Florence Chang • Belinda Ireland • Richard Shiffman • Katrin Uhlig • Cally Vinz Evaluation Project Team Members AFYA, Inc. • Michelle Tregear • Jenice James • Debra Dekker • Craig Dearfield • Ajay Bhardwaj • Robin Pugh-Yi The Lewin Group • Carol Simon • Jaclyn Marshall • Jacob Epstein

3. Evaluation Objectives • Gain a better understanding of how NGC: • Is used by its stakeholders (including AWARENESS among key stakeholders) • Supports dissemination of evidence-based clinical practice guidelines and related documents • Has influenced efforts in guideline development, implementation, and use • Can be improved

4. Data Collection Methods Mixed-Methods Approach Key Project Milestones CIPP evaluation framework – Logic model to develop key questions Developed instruments which were Informed by the PET Received OMB clearance Feb 2011 Conducted survey, focus groups, and interviews (Mar – Jul 2011) • Quantitative Data • Web-based Survey • Skip logic and Branching • Respondents solicited by e-mail lists for AHRQ, AMA, and AHIP • Qualitative Data • Focus groups (4) • Stakeholder-specific • Key informant interviews (26) • Mix of stakeholders

5. Survey Sample Characteristics Referral Source Respondent Demographics Survey Reach Majority from the U.S. (87%) Occupation Mix 56% - Providers, clinicians, nurses 13% - Researchers, librarians, or similar 12% - Consultants, managers, administrators 19% - Other Survey Sample Majority familiar with AHRQ (99%)

6. Survey Respondent Counts NGC awareness and use, demographics, other guideline source questions NGC questions Stakeholder-specific questions

7. NGC Awareness, Use, and Other Guideline Sources

8. Results – NGC Awareness Key Notes • Majority of respondents derived from AHRQ e-mail invitation • Awareness of NGC greatest among AHRQ and AHIP respondents • AHRQ Opportunity to increase physician awareness of NGC • Awareness of NGC in 2011 substantially higher than in 2001 evaluation N=9,289 N=42 N=49

9. Results – Guideline Sources Used • Survey Findings • Most use multiple sources to find CPGs (3-5 most common) • PubMed, government sources, general search engines, medical/professional societies • Qualitative Findings: • NGC often cited as“first go-to source” Key Finding Most NGC users equally satisfied or more satisfied with NGC compared to other guideline sources

10. Results – Degree to which NGC Fulfills Users’ Specific Needs when Using NGC “indicate the degree to which use of the NGC Web site fulfills your needs for:” Key Finding NGC is doing well on these needs

11. Results – NGC Compared to Other Sources in Meeting Needs for Guidelines Finding NGC does betteror about equal compared to other sources This is supported by qualitative findings

12. NGC – Trustworthiness and Inclusion Criteria

13. Results – Trustworthiness • Survey Findings • 75% -very good or good • 80% - would definitely,very likely, or probably recommend NGC to colleague • No differences by key stakeholder group • Qualitative Findings: • Differences by key stakeholder group • Guideline developers and informaticians less trusting • Others note that they “trust” the content because “it comes from AHRQ”

14. Results – NGC Inclusion Criteria • Survey Findings • 63% - appropriate • By length-of-use:longer users more likely to find the criteria too loose • No differences by keystakeholder group • Qualitative Findings: • Differences by key stakeholder group • Guideline developers/informatics specialists said they are ‘too loose’ • Policymakers, medical librarians, researchers generally satisfied

15. Results – NGC Inclusion Criteria • Qualitative Findings (Guideline Developers and Informaticians): • “…the quality of the criteria was good when NGC started out but it has gotten more complicated...” • “…If the goal is to be ‘all inclusive,’ then the criteria are fine. [but]….there needs to be some other ways to separate the wheat from the chaff.” • “They could raise the bar.” • “I’d say too loose…. I think there’s a belief that: A) NGC creates these guidelines…and then, B) there’s also a belief that NGC somehow has a very rigorous process for only allowing certain guidelines in, or certain types of very high quality guidelines, or that it’s an endorsement of these guidelines. And it isn’t.” AHRQ Opportunity to revisit NGC’s Inclusion Criteria

16. Results – NGC Age Criterion AHRQ Opportunity to revisit NGC’s Age Criterion Survey Findings ~Equal # say 5 yearsis “appropriate” or “too long” Reduce to 3 years most common selection No differences by keystakeholder group Qualitative Findings: Common theme: the age criterion is too long

17. Stakeholder-Specific NGC Influences

18. NGC Influences – StakeholdersKey Take-Aways from Survey Findings • Respondents, by group, indicated NGC greatly influenced: • Guideline developers’ ability to identify guidelines, and develop and use quality measures • Health providers’ ongoing learning activities, clinical decision-making processes, and ability to identify guidelines • Medical librarians’ ability to meet their client’s needs • Medical librarians’ and researchers’ ability to identify current and high quality guidelines • Measure developers’ measure development activities and approach to identifying guidelines • Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

19. NGC Influences – StakeholdersKey Take-Aways from Survey Findings • Respondents, by group, indicated NGC greatly influenced: • Guideline developers’ ability to identify guidelines, and develop and use quality measures • Health providers’ ongoing learning activities, clinical decision-making processes, and ability to identify guidelines • Medical librarians’ ability to meet their client’s needs • Medical librarians’ and researchers’ ability to identify current and high quality guidelines • Measure developers’ measure development activities and approach to identifying guidelines • Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

20. NGC Influences – Guideline Developers Participating in NGC • Survey Findings (n=199) • NGC guideline submitters reported greater NGC influence for guideline updating frequency, and how organizations document or report their guidelines • Qualitative Findings (n=24) • NGC serves primarily as a source for locating guidelines • NGC’s age criterion applies some “pressure” to stay current • NGC has had little influence on how guideline developers do their work– e.g., methodology, reporting AHRQ Opportunity to increase knowledge among guideline developers about how to create and report trustworthy guidelines

21. NGC Influence – Dissemination of CPG’s Survey Findings for Participating Developers • Question: How would you rate NGC's dissemination of your organization’s guidelines? • Noteworthy finding • 65% said excellent or good • 21% were neutral • 14% said fair or poor AHRQ Opportunity to identify additional efforts to enhance the dissemination of guidelines

22. NGC Enhancements

23. Likely use of Potential EnhancementsSurvey Findings Examples of Potential Enhancements Ratings of guideline quality Subject-specific e-mail alerts Access to archived guidelines Additional data download options and xml formats AHRQ Opportunity to invest in major enhancements that will increase the value of NGC • Enhancements • Two thirds said they would “definitely,” “very likely,” or “probably” use the following NGC enhancements if available: • Ratings of guideline quality • Subject-specific e-mail alerts • Enhancements for providers • 72% - having NGC content at point-of-care would be useful • 66% - would take CME if available • Enhancements for informaticians • 52% - NGC Summaries as XML file according to the Guideline Elements Model (GEM) or other similar

24. Additional Enhancements – Common Themes Qualitative Findings Additional Common Themes • Guideline Developers • Commentary/responses to guidelines from users • Guideline developer conferences / methodology workshops • Informatics Specialist • Assessment of the executability of guidelines • Medical Librarians • Integration with other Web sites (PubMed) • Quicker access to “new” guidelines • Researchers • Assessment of attributes in IOM standards for guidelines • More information about treatment in multi-morbidity or comorbid conditions

25. Conclusions and Recommendations

26. Key Take-Aways for Expanding NGC’s Use and Impact • AHRQ Opportunities include: • Build on NGC’s user base, including with health providers • Revisit the NGC inclusion criteria • Revisit the guideline age criterion • Increase knowledge among guideline developers about how to create and report trustworthy guidelines • Enhance guideline dissemination efforts • Invest in major enhancements to the NGC Website that will provide significant added value