Program Theory and Logic Models Gareth Parry PhD Senior Scientist

1. Program Theory and Logic ModelsGareth Parry PhDSenior Scientist

2. Welcome and Introduction • Program Theory • Content Theory • Execution Theory • Logic Models • Practical

3. Program Theory

4. What Is A Program Theory? • Aconceptual explanation of how the program, as designed, will theoretically result in the desired improvement. • Comprises two major components: • Content Theory • Execution Theory

5. Kirkpatrick Framework Kirkpatrick Evaluation of Learning Framework has four levels:

6. Content Theory: What changes will teams make that will result in improved outcomes? Inputs Resources invested and activities performed by each improvementagent/entity Level 3 Process/Behavior Changes Level 4 Clinical, personal, organizational outcomes Level 1 Participant Experience Level 2 Learning Execution Theory: What will the improvement initiative do that will lead teams to adopt the process changes? Parry et al. Recommendations for Evaluation of Health Care Improvement Initiatives, 2013, AcadPeds.

7. Content Theory

8. Rapid Response Teams Detailed Tasks and Local Adaptations Core Concepts • Use a reliable method to identify deteriorating patients in real time. MEWS >=5 MEWS >=4 2 Nurses 1 Physician • When a patient is deteriorating, provide the most appropriate assessment and care as soon as possible 1 Nurse 1 Physician 1 Physician Theory Action

9. Driver Diagram: Example PRIMARY DRIVERS SECONDARY DRIVERS AIM • Use a reliable method to identify deteriorating patients in real time. Apply the MEWS Score Establish appropriate criteria for the MEWS Score Reduce inpatient mortality Train staff in the use of the MEWS Score Establish a communication system for an alert • When a patient is deteriorating, provide the most appropriate assessment and care as soon as possible Identify the team who will respond to an alert Train staff in how to proceed following an alert

10. Execution Theory

11. Milbank Quarterly, 2011 • Generating the pressure (will) for ICUs to take part • A networked community • Re-framing BSIs as a social problem • Approaches that shaped a culture of commitment • Use of data as a disciplinary force • Hard edges

12. Summary • A well described theory: • Provides a common understanding of the intention of the model. • Allows an evaluator to understand the core concepts that underpin the more detailed tasks carried out by those at the front lines. • Encourages testing and local amendment, informed by underlying principles. • Shows how all the pieces of a program are related. • Provides a high-level summary that can be the starting point for testing in other settings. Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach, 7th Edition. Sage, California 2004. Ovretveit J, Leviton L, Parry GJ. Increasing the generalisability of improvement research with an improvement replication programme. BMJ QualSaf 2011;20:i87-i91.

13. Logic Models

14. What Are They? • Diagram connection between what we want to accomplish and the way we plan to do that • Teams use them to: • Lay out plan • Expose gaps • Get multiple parties on the same page • Increase intentionality

15. What Are They?The Basic Logic Model: “If-Then” If you accomplish your planned activities to the extent you intended, then participants will see changes in knowledge and start testing If you accomplish your planned activities, then you will deliver the intended amount of product/ service to the intended audience If participants start testing, then participants will change their behavior/ processes If you have access to them, then you can use them to accomplish your planned activities If participants change processes, then you will see changes in patient and organizational outcomes Certain resources are needed to operate your program Activities Outputs Short-term Outcomes Mid-term Outcomes Long-term Outcomes Resources/Inputs 1 2 3 4 6 5 Execution Theory Content Theory Source: Adapted from WK Kellogg Foundation, Logic Model Development Guide

16. Project JOINTS: Background Aim: To assess whether a rapid spread approach applied at a state-level, will lead to increased adoption of the three interventions to reduce SSI in arthroplastichip and knee surgery. • Preoperative bathing or showering with chlorhexidine • Preoperative nasal screening for Staphylococcus aureus carriage followed by decolonization of S. aureuscarriers • Preoperative skin preparation with a long-acting antiseptic agent in combination

17. Project JOINTS: Execution Theory Summary Uptake will occur: • When interventions align with local and national priorities • By using charismatic leaders, highly respected within their field, to describe the changes. • By establishing clear and practical and ambitious aims at a local level. • By allowing for and encouraging local tailoring. • Through facilitation of a networked community.

18. Project JOINTS: Logic Model • Outputs • Nodes in 10 states recruited & assigned to cohorts; hospitals recruited for participation; relationship & support letters from national partners • Tools and communication structure for nodes and hospitals that support implementation • IHI in-person site visits • How-to guide • Improvement tools • Patient/family 1-pager • Surgeon/MD materials & calls • Measurement tools • Node meetings at 2011 and 2012 forum • Monthly node calls • Webinar call series • State-specific calls • Electronic communications (website, listserv, email) • Exemplar hospital application & awardees • Business Case • Hospital stories • Certificate of project completion • AORN and AAOS Support Letters • Hospital Town Hall Meetings Long Term Reduce surgical site infections in hip & knee patients in 10 states in the U.S. • Short Term • IHI: Establish & Support • Rapid Spread Network • Node • - Recruit hospitals • -Strengthen/ develop relationships with & among hospitals • - Assist hospitals with implementation & questions • -Increased capacity to coach hospitals with subject experts • -Create links to related initiatives • - Track hospital participation • - Link hospitals to IHI resources • - Provide feedback • - Suggest potential exemplar hospitals • -Raise awareness of Project JOINTS within state (media) • Hospital • - Set aims; form teams • - Access and use tools • - Increased knowledge of importance of bundle • - Increased leadership buy-in • - Increased surgeon, admin staff, lab, & patient buy-in • - Test & implement new processes & tools • - Share and learn from other hospitals • - Develop tools to share w/other facilities • - Customized implementation (e.g., Pre-ops joints class • - Provide feedback to nodes & IHI Medium Term In participating hospitals, >90% adoption of: Preoperative bathing or showering with chlorhexidine for 3 days prior to surgery Preoperative a)nasal screening for Staphylococcus aureus carriage followed by b) decolonization of S. aureus carriers Preoperative skin preparation w/a long-acting antiseptic agent in combination w/alcohol Inputs $500,000 funding Project & communication management team: PC, PM, communications, director, field manager, clinical director Content experts: Surgical, infection control, nursing, and improvement faculty from IHI Network of state nodes, hospitals & national partners Experience of sites that have done this before Communication & technology infrastructure Learning from campaign tools Evidence that is ready for spread • Activities • Recruitment • -Recruit nodes, hospitals, national partners • -Build/strengthen state-level relationships • -States assigned to cohort 1 or cohort 2 • Ongoing Development & Refinement of Content Materials • -Understand and summarize evidence of interventions • -Gather existing materials from early adopters • -Continuous development & refinement of materials to guide the work • Support Implementation • Build infrastructure where nodes & hospitals can share learning and adapt implementation to their setting: in-person, phone, web-based support • Revise activities and approach based on learning from cohort 1 Context 3yr federal grant funded project designed in two waves to work with all hospitals in 10 states that perform hip/knee replacements to reduce hip and knee SSIs The project is designed to use IHI’s strong relationships w/existing nodes, hospitals and nodes’ strong relationships w/hospitals to speed adoption of evidence based practices Assumptions: SSI reduction is a priority; IHI & nodes are a trusted source of content; to achieve goal must effectively recruit and then engage to support flexible implementation; developing an infrastructure for teams to learn from each other will support successful implementation External factors: - Other SSI focused projects e.g. Partnership for Patients , Dept of Health HAI reduction efforts and QIO 10th SOW detracted attention due to competing priorities and also prioritized reduction of SSIs - Shifting payment mechanisms for reimbursement for SSIs

19. Practical

20. Exercise • You have been asked to design a one year collaborative that aims to improve the uptake of three evidence-based interventions for stroke care: • Brain imaging within 24 hours of admission to hospital (CT scan)to confirm stroke type (ischaemic or haemorrhagic) and determine management. • Delivery of aspirin or an alternative antiplatelet (for patients where an antiplatelet is clinically indicated) within 24 hours of admission to modulate stroke complications and improve outcomes. For shorthand, we refer to this as “aspirin”. • Swallow screen within 24 hours of admission, to prevent unnecessary withdrawal of nutrition, support timely administration or modification of aspirin/antiplatelet delivery and highlight patients who need on-going management of swallow safety.

21. Exercise (30 minutes) • Use the logic model template to begin to map out, how you would describe the: • Inputs • Activities • Outputs • Short, medium and long-term outcomes • Work with others on your table • Think about the outcome you want to impact and work backwards

22. Exercise (15 minutes) • 10 minutes to talk through your logic models. • Reflect on: • What did you find helpful about drafting a logic model? • What was challenging? • What questions do you have? • How can you move forward? • 5 minute report out, for each table: • 1 thing that was helpful • 1 challenge • 1 question

23. Thank you! • Please send any follow up questions or suggestions to: • Gareth Parry email: gparry@ihi.org Twitter: @gjparry03

24. LOGIC MODEL TEMPLATE: Project Title (Month Year – Month Year) Medium term outcomes Long term outcomes Short term outcomes Outputs Inputs Activities Context What is the 2-3 bullet elevator speech? What background info is necessary to understand this project plan? What changes in processes do you expect as a result of your short-term outcomes? e.g., blood protocol implemented with 80% reliability What are the reach and products of the activities? e.g., 3 QI teams trained in improvement methods What changes in organizational and patient outcomes do you expect ultimately? e.g., 20% decrease in under-5 mortality What changes in learning do you expect to come from your activities and outputs? e.g., teams complete PDSA cycles What are you investing to get the work done? e.g., staff, funding, space, external partners, materials, tools What are you doing? e.g., training in QI, monthly data analysis calls Assumptions What are you assuming about what is needed to be able to deliver the program in the above planned way? e.g., leadership is on board and the will is strong, surgeons will make the time to attend trainings and lead their QI team External factors What factors outside of the project may be a barrier or facilitator to reaching your desired outcomes? e.g., new incoming government that prioritizes x, high staff turnover in health centers,