Testing and Measuring Changes

1. Testing and Measuring Changes Review of QI 103: Testing and Measuring Changes with PDSA Cycles

2. Lecture Objectives • Describe how to establish and track measures of improvement during the “plan” and “do” phase of PDSA. • Explain how to learn from data during the “study” phase of PDSA. • Explain how to increase the size and scope of subsequent test cycles based on what you’re learning during the “act” phase of PDSA.

3. Lecture Outline • Planning for data collection • Operational definitions • Key questions • Testing changes and collecting data • Sampling • Studying data for improvement • Run charts • Building your degree of belief over time • Linking tests of change

4. Using Data for Improvement

5. Defining Measures: Three Types Use unambiguous operational definitions

6. Defining Measures: Two Levels • Level 1: Project-level measures • Representing overall goals and assumptions • Documented on the Project Charter • Level 2: PDSA-level measures • Representing the specific changes you are testing • Documented on the PDSA worksheet

7. Identify the type and level of each measure in this QI project: • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • Percent of patient encounters in compliance with hand hygiene procedure • Number of hand sanitizer stations available

8. Identify the type and level of each measure in this QI project: • Your overall goal is to reduce rates of MRSA infection (project-level outcome measure) • Your change is to improve hand hygiene (project-level process measure and PDSA-level outcome measure) • Your first test is to try to improve hand hygiene by increasing the availability of hand sanitizer (PDSA-levelprocess measure)

9. How would you measure this outcome? • Access to health care

10. How would you measure this outcome? • There are many waysto measure “access” • Number of days to third next available appointment • Number of minutes from time of appointment to time to see clinician • Percent of “good” or “very good” answers on relevant patient satisfaction survey questions • Average daily clinician hours available for appointments

11. Which of these is a complete operational definition? • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • Percent of patient encounters in compliance with hand hygiene procedure • Number of hand sanitizer stations available

12. Which of these is a complete operational definition? • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • How do you define the occurrence of MRSA? What calculation will you use? • Percentage of patient encounters in compliance with hand hygiene procedure • How do you define a “patient encounter”? How will you calculate the percentage? • Number of hand sanitizer stations available • What area are you including in the count of sanitizer stations? How do you define “available” — what if the dispenser is empty?

13. Key Questions for Measurement • What is all the data you need to collect? (e.g., for a percent, you need both the numerator and the denominator) • Who is responsible for collecting the data? • How often will the data be collected? • How will the data be collected? • Make measurement as simple as possible!

14. Simplify Through Sampling • Simple random sampling • Proportional stratified random sampling  • Judgment sampling

15. Planning for Data Collection: Review • Establish: What do you want to learn about and improve? • Determine: What measures will be most helpful for this purpose? • Define: For each measure, what is the operational definition? • Designate: who, what, when, where, how

16. Using Data for Improvement

17. Are we getting better or worse?

18. Now, are we getting better or worse?

19. Using Data for Improvement

20. Building Degree of Belief Iterative test cycles; can be concurrent Increase size: 5X rule Broaden scope: Test in many different conditions

21. Acting on Tests of Change

22. Video https://youtu.be/Q4d7T_aBUPo http://www.ihi.org/education/IHIOpenSchool/resources/Pages/Activities/Provost-WhyShouldYouStartTestingChangesASAP.aspx

23. Discussion • Discuss the risks of starting a test of change earlier versus later. Which scenario do you think carries greater risk? • Provost implies that it’s important to see improvement quickly. Why do you think that is? • What do you think of the provocation “what can you do by next Tuesday?” Is it helpful? • Can you think of something to test by next Tuesday? http://www.ihi.org/education/IHIOpenSchool/resources/Documents/Facilitator_Provost-WhyShouldYouStartTestingChangesASAP.pdf

24. Exercise • Follow the instructions at http://www.ihi.org/education/IHIOpenSchool/resources/Documents/QI103_exercise.pdf • Form a team with at least one other person and pick a problem at your school or in your community to work on. The problem should lend itself to measurement. • Use the data collection planning checklist to help you create a data collection plan for the improvement project. Write down the answer for each question for a complete family of measures.