1 / 25

Measuring Implementation Success

Explore approaches to measuring implementation success, understand the boundaries, and factors affecting interventions. Recognize the importance of implementation success as a distinct issue in research. Learn about conceptual and practical considerations in measuring implementation success.

johnsonjulia
Download Presentation

Measuring Implementation Success

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Implementation Success Workshop 2008 QUERI National Meeting December 11, 2008 Carol VanDeusen Lukas, EdD VA HSR&D Center for Organization, Leadership & Management Research

  2. Workshop objectives • Stimulate discussion about the measurement of implementation success • Explore approaches to measuring implementation success • Offer practical lessons for strategies & tools for measurement.

  3. Questions to consider • What do we mean by measuring implementation success? • Why do we care? • How do we do it? • Conceptual considerations • Practical considerations

  4. Implementation research is Research that supports the movement of evidence-based interventions and approaches from the experimental, controlled environment into the actual delivery contexts where the programs, tools, and guidelines will be utilized, promoted, and integrated into the existing operational culture (Rubenstein & Pugh, 2006)

  5. Implementing a clinical innovation

  6. What do we mean by measuring implementation success? • Implementationsuccess is the presence of the intervention in the delivery context & the consistent use of new practices. • Measuring implementation successis formally determining that presence & use

  7. Implementation success ≠ outcome effectiveness/success

  8. Boundaries of implementation success depend on what testing & want to learn

  9. Why do we care? • Contributes to understanding success or failure of intervention outcomes • Implementation success as independent variable • Interventions often fail because not implemented • Intention to treat loses information • Assumption of detailed replication neither realistic or appropriate

  10. Why do we care? • Essential to judging effectiveness of strategies for implementing innovative programs • Implementation success as dependent variable • Understanding factors affecting implementation

  11. Inventory of HSR&D implementation studies • Began with two portfolios • Implementation & management research • Service-directed projects • Screened for implementation studies • Interventions • Studies of factors affecting • Contacted PIs • Reviewed documents • Conducted telephone interviews

  12. Recognition of implementation success as a distinct issue varies • No recognition • Conflate implementation & outcomes • Assume innovation in place • Recognize variation but don’t measure • Know anecdotally what happening • Don’t worry about variation because expect treatment/control differences overall • Recognize and develop measures

  13. How do we measure implementation success? • Conceptual considerations • Practical considerations

  14. Conceptual considerations: Type of intervention • Levels of specification & complexity • Guidelines v programs v locally developed • Boundaries of the intervention • Clinical innovation • Clinical innovation + implementation strategies

  15. Type of intervention guides focus of implementation success measure (ISM) • Guideline only • E.g., hypertension guidelines • ISM: guideline adherence • Tools/strategy to support guideline use • E.g., CRC electronic screen notification, telehealth/ IVR to prompt colonoscopy screening • ISM: guideline & strategy adherence or instead of guideline • Program or clinical area • E.g., supported employment, diabetes expert team teleconference • ISM: adherence to program elements • Strategy to implement program – • E.g., TIDES collaborative care for depression • ISM: adherence to program elements and strategy

  16. Conceptual considerations:Operationally defining the intervention • Key elements • Replication v tailoring/ customization/ adaptation • Which features are core to the intervention? • Which features have to be present for the intervention to maintain its effectiveness? • Which can vary without compromising effectiveness?

  17. Conceptual considerations:Standards of implementation success • How judge success? • Fidelity – extent to which intervention is in place as designed • Scope – how widely used • Intensity – quantity and depth of use • Sustainability – persistence over time • RE-AIM – reach, effectiveness, adoption, implementation, maintenance

  18. Practical considerations:Data sources • Secondary v primary data? • Availability of existing data • Resource & logistical constraints of collecting new data • Qualitative, quantitative • Who collects? • Scale of intervention & role of researcher • Expectations/burden of implementation staff collection

  19. Practical considerations:Creating the measures • Who scores/rates? • Participant, researcher, consultant • Incorporating different perspectives • How scored? • Summary indicators • Components of complex innovations • Role of informed judgment • Rating, ranking, categories

  20. Practical considerations:Using the measures • How incorporate in analyses? • intervention effectiveness • success of implementation strategies • factors affecting implementation & outcomes? • Single v. multiple measures • Value & limits of composite indicators • Categorize sites • Covariates in quantitative analyses

  21. Closing thoughts and take-away lessons… • Conceptual considerations • Type of intervention • Operationally defining the intervention • Standard of implementation success • Practical considerations • Data sources • Creating the measures • Using the measures

  22. Elements of ACA model rated by field liaisons

  23. ACA summary rating by clinic staff

  24. Elements of VISN organizational collaboration model rated by research team members

More Related