Download
1 / 67
670 likes | 836 Views
Conducting Formative Evaluations Using a Consolidated Framework for Implementation Research. Diabetes QUERI Ann Arbor Center for Practice Management & Outcomes Research. Laura J. Damschroder, MS, MPH & Julie Lowery, PhD. Introduction.
E N D
Conducting Formative Evaluations Using a Consolidated Framework for Implementation Research Diabetes QUERI Ann Arbor Center for Practice Management & Outcomes Research Laura J. Damschroder, MS, MPH & Julie Lowery, PhD
Introduction • What questions do we ask when conducting implementation research? • Why didn’t the intervention work everywhere? • What can we do to ensure success? • How can we predict success or failure? • Etc. • So many models, so little time • Comprehensive framework needed with clear definitions of constructs 2
Purpose • Present the Consolidated Framework for Implementation Research (CFIR) • Describe the constructs • Show how we applied CFIR in a macro formative evaluation of the MOVE! Program • Question: Why was there such a wide variation in uptake 1.5 years after dissemination? • This presentation will combine the theoretical development of CFIR and application to illustrate constructs 3
Methods: Theoretical Development • Literature review of models applicable to implementation research • Targeted and non-systematic • 11 models were included • Constructs were identified along with evidence of their role in implementation • Theoretical and/or empirical • “Theme saturation” • When new models failed to produce new constructs, we stopped • Goal: build on what was already developed 4
Methods: Application in a Study • Qualitative study of barriers and facilitators of MOVE! Program uptake • MOVE! Weight Management Program in the VA • Purposive sample of 5 low & high uptake sites • Semi-structured interviews with 24 key stakeholders • 83% of those contacted and invited • Qualitative analysis • Deductive, using CFIR • Inductive, open to new themes • Team-based analysis • Strength of multiple perspectives • Test Face Validity of CFIR 5
Consolidated Framework for Implementation Research (CFIR) Intervention at Time0 Intervention at Time1 Internal Context Soft Periphery Soft Periphery Hard Core Hard Core Intervention Intervention External Context 6
Side Note: Dependent Variable • Implementation • The process of putting an intervention into use in an organization • The vehicle by which a new practice is assimilated into an organization • Implementation Effectiveness • Three general categories: • Widespread avoidance (non-use) • Meager and unenthusiastic use (compliant use) • Skilled, enthusiastic, consistent use (committed use) 7
Consolidated Framework for Implementation Research (CFIR) Intervention at Time0 Intervention at Time1 Process over Time Internal Context Soft Periphery Process Soft Periphery Hard Core Hard Core Intervention Intervention External Context 8
Intervention Intervention Consolidated Framework for Implementation Research (CFIR) Intervention at Time0 Intervention at Time1 Process over Time Internal Context Internal Context Process Soft Periphery Process Soft Periphery Hard Core Hard Core Intervention Intervention External Context External Context 9
Flow • Support for each construct in the literature • Empirical support from MOVE! qualitative study • Begin with Intervention • Evidence Building • Intervention Attributes 10
Intervention – Evidence BuildingHigh Uptake Sites • Strong positive influence of Evidence Strength • "...[providers] understand that a lot of problems are related to obesity. Absolutely everyone understands.” • One VISN used pilot study results to make a business case to obtain 9 dedicated FTEs for the program at VISN sites • Strong positive influence of Relative advantage – • “…with the help of MOVE information, MOVE literature, MOVE whatever…it sort of boosted [our existing program] more and we were able to expand more…” 13
Intervention – Evidence BuildingLow Uptake Sites • Source of the Intervention: • “Wow. Well there’s nothing like an unfunded mandate...to get...their blood boiling around here where workloads are so high everywhere else.” • The same pilot results were seen as weak evidence supporting MOVE!: • “…unfortunately, with the pilot study data not being very robust, we had a difficult time selling chief of staff and chief of medicine on the efficacy of the pilot study” • One low uptake site referred patients to an external program which they felt had greater relative advantage 15
Intervention – AttributesHigh Uptake Sites • Adaptability – Both high-performing sites perceived the MOVE! program as flexible and designed programs that fit within their particular context • Trialability – One site learned from the national pilot experience • Complexity • One site simplified a challenging implementation by taking an incremental approach • The other site viewed MOVE! as a relatively simple incremental change from their current program 18
Intervention – AttributesLow Uptake Sites • MOVE! as not sufficiently adaptable for one site: • “…all of our patients would need to be referred for an EKG prior to starting the program. There were some barriers that we didn’t understand…” • Implementation was particularly complex because of barriers throughout their organizations • One site was particularly troubled by packaging of MOVE! • “…the initial start up manual had very not positive pictures on it. Depressed looking, heavy sailors in stretched out white tee-shirts....” • The other site was impressed with quality of materials and suggested it be implemented outside the VA 20
EXTERNAL CONTEXTHigh Uptake Sites • Both sites put an emphasis on tailoring programming based on patient needs and requests • Many anecdotal stories about patients • “…one of our patients that is enrolled in MOVE…goes and talks to other patients in the waiting rooms saying what a great program it is, takes their names and…leaves it in my boxes…and we actually have a tremendously long waiting list…” 23
EXTERNAL CONTEXTLow Uptake Sites • The general atmosphere at the two low-uptake sites is a belief that it is a challenge to present MOVE as a viable alternative • “…our system is geared to paying people to be disabled….I think the commitment of the patients has to be up there among the top three [difficulties]...because…we live in a society of quick fix and if the medicine won’t do it; ‘you don’t expect me to starve for 10 weeks, do you?’” • “…and speaking very frankly, when I’m at the [community program] kickoff which is a very positive high paced environment…I’m out there with the ‘managing obese veterans everywhere.’ I frankly…had a hard time selling some of our veterans” 25
INTERNAL CONTEXTHigh Uptake Sites • Both sites had a high degree of “teamness” • “Very amicable, very very good, pleasant, very professional. I mean there isn’t a week that doesn’t go by that you know, we’re not communicating with each other and not really, we’re having a good time too with the group sessions…we’re all there to make the patients really change the way they’re eating and their activity habits…” • Meet regularly • “…we do this through our…lunch time. We keep it very short…It’s very difficult to and we have our other assignments and you …don’t have the free time to do it…to discuss obstacles, to discuss problems, to discuss you know, things that need to be discussed for us to be able to run this program properly.” 28
INTERNAL CONTEXTLow Uptake Sites • Did not have regular team meetings • Though one did have a multi-disciplinary team that takes turns leading the group visit • “Meet” through email • Lack of effective communication • Patients confused about what MOVE was • a movie? • Dance class? • Bariatric surgery? 30
Another Side Note • Confusion between • Culture • Climate • Other terms • Greenhalgh, et al • Receptive context for change • System readiness for change • Absorptive capacity (for new knowledge) • PARiHS • Readiness for change • Klein & Sorra • Implementation Climate 31
CFIR Terms • Culture • Implementation Climate • 6 constructs • Readiness for implementation • 4 constructs 32
INTERNAL CONTEXT–Implementation ClimateLiterature Support 33
INTERNAL CONTEXT–Implementation ClimateTension for change • Particularly clear for transition site • “…all of the group was pretty excited you know, because they had sat stagnant for a year…they had put up…posters [for] the MOVE program and all that stuff and they didn’t have anything set up so people were consulting to the MOVE program when there wasn’t even a program set up…” • Closely related to “Relative Advantage” for high uptake sites • Low tension for change at low uptake sites 35
INTERNAL CONTEXT–Implementation ClimateCompatibility • Compatible at high uptake sites in terms of “meaning” (difficult to disentangle from other constructs, however) • PCPs acknowledge the clear connection between MOVE!’s ability to induce weight loss which impacts important performance measures like blood pressure • …and in terms of fit with pre-existing program • At one low uptake site, some providers did not see this connection. However, MOVE! was compatible with physician champion values: • “…my just natural interest in this type of a program which is [a] more holistic…approach to managing some problems 37
INTERNAL CONTEXT–Implementation ClimateRelative Priority • At one high-uptake site, tying MOVE! into their bariatric surgery program increased priority • “…we were approved to start a bariatric surgery program bam, right away… and unfortunately all our doctors and administrative people are enormously interested a bariatric surgery…and all resources and interests funneled into bariatric surgery…we did everything backwards…in hindsight, it probably was a good way to do that because our criteria for eligibility for people to have bariatric surgery is that they must be enrolled in MOVE! for one year • One low-uptake site has to contend with many competing priorities that overshadow their MOVE! activities. • Staff work weekends and lunches to get through backlog of patients in primary care • MOVE! competes for space with other group classes 39
INTERNAL CONTEXT–Implementation ClimateIncentives & Rewards • One high-uptake site rewarded a clerk with a prize of recognition for stellar work on MOVE! • A physician champion at one low-uptake site shared that: • “…we had no incentive, you know, we didn’t get our boxes checked for getting this program implemented. You know, I didn’t get a raise, I didn’t get a bonus, nobody was patting me on the back…” • …particularly notable with lack of dedicated time 41
INTERNAL CONTEXT–Implementation ClimateGoals & Feedback • At one high-uptake site, anecdotal stories from successful patients motivate MOVE! team • The other high-uptake site reports weight loss and goals • Both sites emphasize outcomes other than weight loss • At one low-uptake site, VHA performance measures are emphasized but… • Detriment to MOVE! because weight loss is not a performance measure • One MOVE! team member works to track weight loss data from home because of lack of time at work 43
INTERNAL CONTEXT–Implementation ClimateLearning Climate • At high-uptake sites coordinators not afraid to test new strategies and share dreams for improvement • Act on “lessons learned” from VISN • “…I did express that to my Chief of Staff…that I would like to see here what I call a ‘MOVE Suite…’” • Weak Learning climate – coordinator talked to potential physician champion informally to avoid “getting arrows in her back” 45
INTERNAL CONTEXT–Implementation Climate • In Summary, Implementation Climate comprises: • Tension for change • Compatibility • Relative Priority • Incentives & Rewards • Goals & Feedback • Learning Climate 46
INTERNAL CONTEXT–Readiness for Impl.Leadership Engagement • High-uptake sites • VISN MOVE! Coordinators actively involved with local facilities and with VISN leadership • Help with data reporting and problem-solving • Help ensure MOVE! is visible with leadership • Local supervisors dedicate staff time for MOVE! • Low-uptake sites • “In the last year, there has been a change of leadership in primary care and that has made a big difference...they’re giving it all they can possibly give it, given the overall constraints” 49
