CMS Measure Development Education & Outreach

1. MACRA Info Session #7 CMS Measure Development Education & Outreach • Environmental Scans for Measure Development and Introduction to the Environmental Scan Support Tool (ESST) Presenters: Jeff Geppert and Jarrod Olson (Battelle) May 1, 2019 2:00-3:00pm EST

2. Vision and Goals: MACRA Info Sessions • An ongoing process to engage measure developers through education and outreach in quality measure topics. • Elicit feedback that will help CMS design resources that can help all of those interested in healthcare quality improvement better understand the goals of quality measurement. • Education • Outreach • Dedicated Websites • Measure Development Roadmaps • Listserv opportunities

3. Agenda Overview of environmental scans for measure development What should be included in an environmental scan? Demo of ESST Examples

4. Overview of Environmental Scans for Measure Development

5. Measure Lifecycle Timeline

6. What is an Environmental Scan? • Part of the information gathering stage of measure development • Builds the case for quality measures and helps to establish the measurement plan • A strong environmental scan will include: • Information about relevant scientific and grey literature • Related measures currently in use or under consideration • Economics of the measure construct • Stakeholder (including patient) considerations

7. Objectives of an Environmental Scan • Explain why the measure is needed (i.e., opportunities for improvement, gaps in existing measure landscape) • Show how it will further CMS aims and objectives • Describe the strength of the evidence supporting the measured process/outcome

8. Why Conduct an Environmental Scan? • Identify measurement gaps • Determine areas of controversy relevant to quality measures • Gain insight into existing measures, methods, and outcomes which could be adapted to suit a new topic • Better understand stakeholder input • Increase the likelihood that a measure development project succeeds

9. Priority Planning • Environmental scans can be broad or narrow • Depends on the number of measures to be developed and the focus of each • Before initiating the environmental scan, consider the problem(s) that should be addressed • Is it a high priority problem? • Does it align with the Meaningful Measures Framework?

10. What Should Be Included in an Environmental Scan?

11. Components of an Environmental Scan • Literature review • Peer-reviewed literature • Grey literature • Clinical practice guidelines review • Measures review • Measures search • Call for Measures • Stakeholder input

12. 1. Literature Review:What to Include • Peer reviewed publications (e.g PubMed) • Published in the last five years Data collected in the last ten years • Existing literature reviews and systematic reviews • Grey literature • Unpublished government studies and reports • Working papers from professional societies • Manuscripts produced by international agencies (e.g. World Health Organization) and non-profit organizations Example of grey literature: NQF reports

13. 1. Literature Review: Interpreting the Evidence • Strength of Evidence: • Quantity of studies • Consistency of findings • Grading of body of evidence • Controversy and contradictory evidence • Information about disparities What if evidence is mixed? If recent studies have emerged which contradict established findings, they must be included in the report, even if their results are not deemed strong enough to base new measures on.

14. 1. Literature Review:Supplementary Information • Include a list of relevant citations, details about the characteristics of each study (e.g., population, sample size, study type, data sources), and which measure evaluation criteria each study addressed. • If developing measures for CMS, measure developers must submit a complete explanation of the search process used to develop the literature review. This includes: • Online databases searched(e.g. ESST) • Keywords included • Boolean logic used to identify relevant studies

15. 2. Clinical Practice Guidelines Review • Identify recent clinical practice guidelines applicable to the measure topic • American national healthcare programs • American federal agencies • International guidelines • Document the strength of the evidence base for the guideline • Was a grading system used? • Measure developers are not required to grade evidence themselves

16. 3. Measures Review: Where to Search Existing and Related Measures • Searching existing measures helps measure developers identify measurement gaps and existing measures which could be adapted to suit a new topic. • Information about existing measures should be obtained from a variety of databases and sources. • CMIT (and the ESST) • NQF’s Quality Positioning System • PCPI

17. 3. Measures Review: How to Search Existing and Related Measures • In order to develop related measures with harmonization in mind, look for: • Measures used in the same setting, but for a different topic • Measures used in a different setting, but for the same topic • Measures constructed in a similar manner • Quality indicators • Accreditation standards • NQF preferred practices for the same topic

18. 3. Measures Review: Call for Measures • If the search for existing and related measures does not yield sufficient results, a Call for Measures may be necessary. • For information-gathering purposes only • Notify stakeholder organizations prior to release • Seek related candidate measures and measure concepts • Measures and measure concepts submitted should be evaluated using the measure evaluation criteria and included in the environmental scan report.

19. 4. Stakeholder Input: Soliciting Input • Stakeholder input can be used to frame the environmental scan and to guide the search process • Stakeholders to engage include: • Patients and caregivers • Front line clinicians • Professional societies • Payers • Measure development experts

20. 4. Stakeholder Input:Best Practices • Seek stakeholder input early and often • Iterative process • Seek input formally and informally • Document processes throughout the Environmental Scan • Keep notes regarding decision rationale (e.g., literature review search parameters, need for Call for Measures)

21. Discussion Questions

22. Overview of Environmental Scan Support Tool (ESST) • Automates environmental scan search and prioritization of biomedical literature • Monthly updates for all Process and Outcome measures in CMIT

23. Demo https://cmitmms.cms.gov – Controlled Access site https://cmit.cms.gov – Public site

24. How ESST Works • How does it work? • Artificial Intelligence (AI) identifies the most relevant PubMed content from the Measure Information Form • Clinical Quality Measures (CQM) ontology “describes” a measure, identifying the population, health status or process, the change concept, and the expected output to categorize

25. Using ESST • Available with login from the CMIT Controlled-Access site • Request a login at https://cmitmms.cms.gov/CMIT/ViewAccountRequest • Example Use Case: Review a measure • Login to CMIT • Search for the measure • Review the environmental scan results

26. Features Coming Soon • Public site, not requiring controlled access (Released April 27) • Expanded biomedical publication databases • Psychinfo (September 2019), others to follow • De Novo Measure Scan (i.e. enter search terms for a new measure) (September 2019)

27. Environmental Scans: Resources and Templates The CMS MMS Blueprint • Chapter 9 – Information Gathering • Chapter 10 – Environmental Scan • https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/MMS-Blueprint.html

28. Discussion Questions

29. Upcoming MACRA Info Sessions Planned Upcoming Webinars: • May 22, 2019 – Re-Introducing the eCQI Resource Center and the CMD Workspace • June 12, 2019 – eCQM Testing: Making the Most of Limited Testing Data • July 17, 2019 – CMS National Impact Assessment Overview and Measure Evaluation To suggest topics for upcoming Info Sessions, email: MMSsupport@battelle.org

30. Upcoming Public Webinars Planned Upcoming Webinars: • June 25, 2019 and June 27, 2019 – Measuring Quality to Improve Quality: Strengths and Challenges of Clinical Quality Measurement • July 24, 2019 and July 25, 2019 – Patient-Centered Quality Measurement: What It Is and How to Get Involved

31. Contact Information • Battelle Measures Management System Contract Holder Contact: MMSsupport@Battelle.org • CMS Kimberly Rawlings: Kimberly.Rawlings@cms.hhs.gov

32. Examples of Environmental Scans

33. Example 1: Measuring What Matters

34. Example 1: Identifying Measures • Identified relevant process and outcome indicators • Public domain • Tested for reliability and validity • Included measures that had been endorsed by general healthcare quality organizations (e.g., the National Quality Forum), and sources specific to the palliative care population (e.g., PEACE Palliative Care and Hospice) • 75 indicators met inclusion criteria and were classified by care domain

35. Example 1: Evaluating Measures • A TEP rated the technical strength of the 75 indicators identified in the search. • Measures were rated on: • Importance • Clinical logic • Reliability and validity • Feasibility • The panel determined that 34 measures were scientifically sound enough to merit review by a clinical user panel.

36. Example 1: Evaluating Measures • A clinical user panel of physicians, nurses, social workers, pharmacists, and chaplains evaluated the measures deemed scientifically sound. • Measures were rated on: • Meaningfulness to patients/families • Actionability for providers/organizations • Potential for impact • Panel ratings identified 10 quality indicators in six domains suitable for evaluating the quality of palliative care.

37. Example 2: Hypothetical Environmental Scan

38. Example 2: Hypothetical Environmental Scan Environmental Scan: • Determine literature of interest (peer reviewed, grey) • Determine databases of interest (i.e., PubMed and Medline) and additional searches (e.g., Google) • Consult clinical practice guidelines from several organizations (i.e., US Preventive Services Taskforce, the American College of Cardiology/American Heart Association, and the European Society of Cardiology) • Convene a TEP of experts in cardiology, patients, representatives of various payers, and front-line clinicians

39. Example 2: Hypothetical Environmental Scan Environmental Scan – Continued: • Discuss evidence to identify gaps in the literature as well as measurement gaps relevant to quality of care • Search for existing quality indicators used by government organizations (e.g., CMS) and professional societies (e.g., American Heart Association) • Identify important measurement gaps • Develop a set of draft measures which were sent to the TEP for feedback and modified based on that feedback