Building the Science of Health Care Quality Improvement Intervention

1. Building the Science of Health Care Quality Improvement Intervention Denise Dougherty, Ph.D. Senior Advisor, Child Health and Quality Improvement AHRQ Annual Conference 2010 Coordinator and Moderator Implementation, Change, and Improving Health Care Quality and Safety: Lessons Learned From AHRQ’s Implementation Science AwardsSeptember 28, 2010

2. Overview (15 minutes) • Introductory Comments: Why and how does AHRQ Focus on Implementation Science? • An emerging framework • Implementation Science grantees work • Rita Mangione-Smith • Carrie Byington • Interactive Discussion

3. T1 T2 T3 The “3T’s” Road Map to Transforming U.S. Health Care Improved health care quality and value and population health Basic biomedical science Clinical efficacy knowledge Clinical effectiveness knowledge Key T1 activity to test what care works Clinical efficacy research Key T2 activities to test who benefits from promising care Outcomes research Comparative effectiveness research Health services research Key T3 activities to test how to deliver high-quality care reliably and in all settings Measurement and accountability of health care quality and cost Implementation of Interventions and health care system redesign Scaling and spread of effective interventions Research in above domains Source: JAMA, May 21, 2008: D. Dougherty and P.H. Conway, pp. 2319-2321. The “3T’s Roadmap to Transform U.S. Health Care: The ‘How’ of High-Quality Care.”

4. Relevant Provisions in New Laws: American Recovery and Revitalization Act (ARRA) • “Meaningful Use” of Health IT = • Beyond getting the electrons in place • Using health IT to improve quality and safety of health care • Funding for Comparative Effectiveness Research (CER) beyond purely clinical interventions • System redesign • Enhanced registries for QI and CER • Accelerating Implementation of Comparative Effectiveness Findings on Clinical and Delivery System Interventions by Leveraging AHRQ Networks • Other (http://www.ahrq.gov/fund/granarch.htm#RFA)

5. New Laws: Patient Protection and Affordable Care Act • Demonstration Projects for Quality Improvement • “demonstration” — 312 mentions • “pilot” — 80 mentions# • Creation of the Center for Medicare and Medicaid Innovation (CMI) • National Strategy for Quality Improvement • More (see CRS report) # http://e-caremanagement.com/pilots-demonstrations-innovation-in-the-ppaca-healthcare-reform-legislation/ http://www.aamc.org/reform/summary/ph.pdf

6. CHIPRA – CMS Quality Demonstration Grants to States • Aims: A) Experiment with, and evaluate the use of new measures for quality of Medicaid/CHIP children's health care; B) Promote the use of HIT for the delivery of care for children covered by Medicaid/CHIP; C) Evaluate provider-based models which improve the delivery of Medicaid/CHIP children's health care services; or D) Demonstrate the impact of the model Electronic Health Record format for children (developed and disseminated under section 401(f)) on improving pediatric health, and pediatric health care quality, as well as reducing health care costs. E) Broad systems approaches/medical home • 10 awards made Feb. 2010 to individual States and consortia of States • National Evaluation (in planning stage –AHRQ has lead) http://www.cms.gov/CHIPRA/15_StateDemo.asp

7. AHRQ Funding to Test and Disseminate Strategies to Improve Quality and Patient Safety • ACTION II • Program Announcements • Evaluation of Spread of the Keystone projects (health care associated infections) • Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (forthcoming report) • Co-sponsorship of the NIH Science of D & I conference (2011) • Innovations Clearinghouse • Knowledge Transfer Projects • Value Exchanges • PAR 08-136

8. What are We Trying to Learn from all This Work? • Answer: Not Only the What, but the How and the Why of Healthcare Quality Improvement

9. Current State of the Science: QII and Evaluation Designs—A Personal View • Problem identification (vaguely defined) • Theory of action to solve the problem (often omitted, vague or in-name-only) • Interventions • vaguely described; • not replicable; • conceptual confusion between “intervention” and “implementation” • Focus on internal validity and related designs • Context of intervention/ implementation processes: • not considered or • considered post hoc and descriptive/idiosyncratic • effects of context/variation in context not considered in assessing results and variation in results • “qualitative” research does not mean standards of the qualitative research field • Lack of validated measures of contextual variables (leadership, culture, teamwork, resources) • For publication: design driven by clinical hierarchy of evidence standards (RCTs at patient level) • If not for publication: • threats to internal validity rarely considered; post only studies or simple pre-post without comparisons; implications for knowledge base not widely recognized. • Few comparison studies (one QI intervention to another; multiple settings)

10. Specific Example: Context– Multiple potential influences on QII Results • External factors – e.g.: • Regulatory requirement • Payments or penalties • Local sentinel event • Structural/organizational characteristics (organization site) • Culture, Teamwork, Leadership • Implementation Processes and Tools • Staff education and training • Audit and feedback • Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, forthcoming.

11. Specific Example: Quantitative Approaches to Context Heterogeneity - Progress • Premise: Context often moderates intervention effectiveness • This moderation effect can be represented statistically through the “intervention x context” interaction: • Yi = b0 + b1 × Ti + b2 × Ci + b12 × Ti × Ci + εi, • where i denotes the unit of analysis (usually the various sites in the study, but can also be dyads of sites in matched comparisons), Yi denotes the outcome measure, Ti denotes the intervention status (Ti=1 for intervention, Ti=0 for control), Ci denotes the contextual factor, Ti × Ci denotes the “intervention × context” interaction, εi denotes random error, b0 denotes the intercept for the model, b1 denotes the main effect for the intervention, b2 denotes the main effect for the contextual factor, and b12 denotes the moderation effect for the contextual factor, i.e., the influence of the contextual factor on intervention effectiveness. • Looks like progress, assuming we can quantify contextual variables • Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, Chapter 12, Special contribution from Naihua Duan, Columbia University, New York, New York forthcoming

12. Meta-Science Issues • IRBs • Study sections • Promotion and Tenure • Little collaborative research (understanding effects of variations in context) • Conflict between research and evaluation • Limited knowledge of evaluation “how to”?

13. Research in Implementation and Change While Improving Quality – The Answer? • PAR -08-136 • A relatively small attempt to specifically try to understand the how and why in a rigorous way • $300K/year • No dedicated pot of funds at AHRQ – highly competitive • This session: • Two examples – in process – methods and interim results, not definitive findings • Interactive discussion – • What would you add? • What else do you need to know?