Measuring & Understanding Quality Improvement in Healthcare

1. Measuring & Understanding Quality Improvement in Healthcare Steve Meurer, MBA / MHS, PhD Vice President - Operations St Mary Medical Center Langhorne, PA

2. Focus Continuing the Journey • 1) Research Questions • Experience in Healthcare Operations • 2) Theory / Models • Search and Study • Develop Questions and Hypotheses • 3) Develop / Test • Develop 2 into something that can • help answer 1 • 4) Examine Results • How does 3 answer 1 Oral Exam Defense

3. Initial Research Questions Developed from 8 years of frustration in healthcare management knowing that I wasn’t equipped to provide appropriate support to clinicians What I Could Provide What I Needed to Provide Leadership Understanding of Healthcare Financial Direction Strategic Direction Management Capabilities A More Balanced Approach to Managing Assessment Data Management Study Design

4. Initial Research Questions • How do I know something works? • Continued requests for equipment, supplies and instruments • Do patients get better? • The End of Medicine • Is there one way to do a procedure that is better than another? • Significant variation in preference cards How do I measure quality?

5. Defining Quality • IOM – The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge. • Donabedian - The systematic measurement and evaluation of the predetermined outcomes of a process, and the subsequent use of information to improve the process based on expectations of the customer.

6. Theories / Models – Oral Exam • Ernest Amory Codman • end results idea – 1920s • Florence Nightingale (late 1800s) and Walter Shewhart (1920s) • Statistical Process Control • Edwards Deming, Joseph Juran and Kauru Ishikawa • Continuous Quality Improvement (CQI) • Avedis Donabedian • structure – process – outcome • Implicit vs. explicit criteria • Everett Rogers • Dissemination of Innovation • Don Berwick, Paul Batalden, Brent James and Steve Shortell • Recent literature

7. Structure – Process - Outcome Quality of healthcare can be assessed on the basis of structure, process (how care is delivered), and outcome (mortality, functional status, quality of life, and patient satisfaction) good measures of the first two are those that have a clear relationship to the third structure must proceed process which must proceed outcome • Structure Definition • Something arranged in a definite pattern of organization • Organization of parts as dominated by the general character of the whole

8. Implementing CQI is Largely Structure Medical Scott CQI Outcomes Organization Division of laborHR / Training Specialty Mix Coordination Customer Focus Resources Training/Experience Planning Work Load Power Process Access Resources Innovation Buildings Supplier Partner Information Information Leadership Leadership Policies/Procedures Tasks

9. Structural Dimensions of CQI Implementation Adapted from Shortell et al. 1996

10. Translating Theory To Research – Putting Shape to my Frustrations • 1. The healthcare system is broken • The IOM reports 2. Appropriate structure – “the forgotten, but important component of the quality triad” – is wrongly assumed as present  JCAHO • 3. CQI, if implemented properly, can be the appropriate structure • Managerial Philosophy 4. Healthcare providers are are finding it difficult to implement CQI Problem Solving Methodology

11. Research Question • Why haven’t healthcare organizations been able to use CQI to differentiate themselves in terms of quality? Answer: Accountability & Assessment

12. Accountability Individual motivation has not been successful Midnight at the Waldorf-Astoria Rhetoric, not Reality (The Halothane Study) Large scale environmental change is needed Environment, Organization, Micro-System, Pt • Current Motivators that may force change • Patient Safety – Medical Errors • “report cards” - PA, NY and CA • Increase in Costs and Premiums • Increase in the use of Alternative Medicine • Variation in processes • Increase in litigation

13. Assessment Issues & Research Questions 1. Low CQI knowledge level of senior leadership Do step by step instructions exist that assists researchers in studying CQI, and healthcare leaders in implementing CQI? 2. Implementing only a subset of the CQI domains Does a comprehensive survey and scale of CQI implementation exist? • 3.Poor measurement strategies • Are psychometrics examined appropriately in measuring CQI implementation? 4. Rhetoric does not equal the reality Does a measure exist that can be used to develop a quick snapshot of CQI implementation efforts in a hospital and is there a corresponding descriptive scale?

14. Step 1: Search the Literature Does a comprehensive, valid and easy to administer measurement tool exist that provides organization leaders with a descriptive scale and instructions for next steps? • Methods - • Used Ovid databases with a focus on business, psychology, sociology and healthcare • Keywords: measurement quality, continuous quality improvement, total quality management, implementation • Scrolled through quickly at least 10,000 references • most focused on implementing only a few domains of CQI • Articles not deleted included: Measuring Quality; Domains and Implementation of Quality

15. Quality OverloadUsing Medline through PubMed • from 1995 to the present / English only • ‘quality improvement’ = 8,848 • ‘continuous quality improvement’ = 1,100 • ‘quality’ in title = 17,466 • ‘quality improvement measure’ in title = 3 • from 2000 to present / English only • ‘quality improvement’ in title = 350 • 50 usable, 30 ‘easily findable’, 15 good, 1 measurement

16. Worldwide Measurement for QA/QI Structure AWARDS - too time intensive, gold standard domains • The Malcolm Baldrige Award Leadership, HR/Training, Process, Business Results, Customer Focus, Information Systems, Planning, Partnership • EFQM • US State-Based Awards ACCREDITATION & CERTIFICATION - questions on validity and too time intensive • JCAHO • IS0 9002 - 2000 Inspection, Contract, Public Responsibility, Innovation, Product Control, Servicing SURVEY INSTRUMENTS

17. Survey Instruments • Eleven worldwide surveys examining CQI as a managerial philosophy were analyzed (Tables - pges 1 & 2) • 8 from the US, 1 from Canada, the Netherlands, and Australia • 5 were specific to healthcare, including the ‘gold standard’ from Shortell et al. • Most examined psychometrics while very few provided a scale • The shortest (22 questions) was also the least comprehensive • The Baldrige Domains dominated

18. Survey Instruments • Weaknesses of the current surveys included: • Relatively few domains other than the Baldrige were even mentioned • The most comprehensive surveys are much too long (depth vs. breadth) • Strong potential for respondent bias if survey is only given to one level employee in an organization • Very few surveys provided a scale Published in the International Journal for Quality in Health Care 2001: Volume 13, Number 3: pp 197-207

19. Step 2: Develop and test a comprehensive and concise measure of CQI implementation • Initial Survey • 14 domains (Baldrige, EFQM, ISO) and 70 items, with each domain containing at least 4 items • All but a very few items were from the 11 surveys analyzed in Step 1 • Items were chosen by the researchers using a subjective analysis and whether or not the question could be answered using a 5 point Likert scale

20. Content Validity • The benefits of a content validity study for this study True experts in the field of CQI Past measures have gone through psychometric testing Excellent method of data reduction • Methods • Statistical method described in Grant & Davis (1997) and Lynn (1986) 1. Panel of Experts All either attend an invite only CQI symposium sponsored by Dartmouth, have recently taught CQI at a Masters level or are positional leaders of QI efforts in a healthcare organization  Best to have between 7 and 10 - this study had 7 from the US and 1 from England

21. Content Validity • Methods (cont.) 2. Scoring Grid (See Sample Grid - pge 3) • Each expert was emailed the scoring grid with definitions and instructions. Is the item clear and understandable? • 4 point scale Does the item represent CQI? • 4 point scale Match the item with a domain. • 1 through 14 representing each domain & 15 representing unable to classify

22. Content Validity • Methods (cont.) 3. Indices Inter-rater agreement (IR) = # of raters who scored an item as high / total # of raters • high defined as a 1 or 2 on both 4 point scales • acceptable IR > / = .70 Content Validity Index (CVI) = # of items where all experts rated high / # of items • acceptable CVI > / = .80 Domain congruence = % of time where experts chose the same domain as the investigators

23. Content Validity Results • After 4 analysis iterations where poorly rated items were deleted, the questionnaire included: • 22 items • 8 domains • Clarity IR of .91 (range of .85 - 1) • Representativeness IR of .93 (range of .87 - 1) • Clarity CVI of .73 using Lynn’s (1986) method • Representativeness CVI of .91 using Lynn’s (1986) method • Overall, the experts chose the same domain as the investigators in the original measure 76% of the time

24. Content Validity Results • Investigators added 6 items to ensure that every domain except for Supplier Partnership contained 3 items • Investigators changed some wording to increase clarity • Final survey was sent back to experts for comments • Other Results • The scoring grid took a lot longer to complete than originally thought • The leadership domain had the highest rate of agreement • QI must be differentiated from QA • Baldrige criteria dominate

25. Step 3: Develop a corresponding scale of CQI implementation • A 5 level corresponding scale (pge 4) was developed by the investigators from: • Roger’s Diffusion of Innovations • agenda setting, matching, redefining / restructuring, confirmation, clarifying, routinizing • Deming’s Continuous Quality Improvement • match domains with appropriate level • Samsa & Matchar • CQI as a problem solving methodology vs. a managerial philosophy • Characteristics: • CQI is a developmental process • Time is important • Scale provides focus for future quality initiatives

26. Steps 2 & 3 - Content Validity and Scale Published and Voted Best Student-Led Paper in the 2002 Business and Health Administration Proceedings,pges 198-204 Will be further published in Either Quality in Health Care or Hospital Topics

27. Step 4: Is the survey and scale easy to use and are the results practical? Pilot Study • Worked with the Missouri Hospital Association for contacts. • 83 Missouri hospitals eligible (above 40 beds), 40 participated • 5 responses from each hospital: CEO/COO, Director of Quality, a non-salaried MD, and 2 managers • Survey and results disseminated via email Hypotheses based on Paper 1 Weaknesses • 1. There will be measurable differences between and within hospitals. • 2. The survey will have high known-groups validity. • 3. The items and domains will differentiate between levels as hypothesized by the conceptual scale.

28. Pilot Study Methods • Known Groups Validity • Examined the relationship between the survey and: • state quality team winners >/= Level 2 • state quality organization winners >/= Level 3 • national quality award finalists >/= Level 3 • subjective quality assessment at 10 of the 40 hospitals • question 1 (pge 5) asking the participants to categorize their quality structure • Reliability • Cronbach’s Alpha for each domain, each title, and the overall measure

29. Pilot Study Methods • Between Hospital Variation • One-way ANOVA & Bonferroni • by size, region and ownership model • Within Hospital Variation • Repeated Measures ANOVA & Bonferroni • by title • Item and Domain Analysis • ANOVA & Bonferroni • determine which items and which domains discriminate well between different levels of the scale

30. Pilot Study Hospital Total LevelN = 40 (min of 90, max of 130)

31. Pilot Study Results • Sample Characteristics (pge 6) • 40 hospitals are significantly larger and more likely to be for profit and part of a system • Of the 200 returned surveys, there was less than 5% missing values and ‘I don’t know / NA’ • Known Groups Validity • 2 of the 12 did not score as hypothesized • hospitals were not significantly different than others • 4 of 9 (44%) similar for the subjective assessment • 40% agreement for question 1 assessment

32. Pilot Study Results • Reliability • Cronbach’s Alpha ranged from .54 (HR/Training) to .84 (Innovation) for the domains • Information .69; Process, Planning .77; Customer .78; Leadership .83 • Cronbach’s Alpha ranged from .88 (Director /Manager and QI Director) to .92 (MDs) for different groups • Cronbach’s Alpha was .94 for the overall measure

33. Pilot Study Results • Between Hospital Variation • Region was only attribute that was significant • Within Hospital Variation • Senior Executives significantly lower than QI Directors • QI Directors significantly higher than Managers / Directors • MDs significantly higher than Managers/ Directors • supports surveying more than one level employee

34. Pilot Study Results • Item and Domain Analysis (pge 7) • Of the 28 items, 4 did not show good differentiation between any of the levels • these should be either reworded or changed • Leadership showed significant differentiation between all levels studied • Planning showed significant differentiation between 2 of the 5 levels

35. Pilot Study Results

36. Findings • The survey is easy to administer • The survey provides a reliable and valid snapshot of CQI implementation in a healthcare organization • No known group exists • The scale is a practical method of providing hospital leaders with a roadmap for CQI implementation • Leadership is the most important component of implementing CQI Submitted to Health Services Research

37. A Likely Future Scenario • Patient Safety provides accountability to analyze quality and outcomes • Healthcare leaders see CQI as a methodology to improve patient outcomes • Hospitals use the survey and scale to help assess & implement CQI appropriately, which in turn eliminates structure issues discussed • Because of this, hospitals can effectively assess their processes and improve their outcomes

38. Future Research Questions Can a clearer snapshot of CQI implementation emerge using line worker responses, and senior leadership interviews? • Administered the survey to a 40 random line workers at 10 hospitals • Initial results include: • 50% return rate • high number of ‘I don’t know / NA’ responses • all hospitals overall employee score < 100 (Quality Assurance) • Senior hospital leadership meetings to discuss quality structure • Initial results include: • low knowledge of ‘quality’ among the senior leaders • structures developed with little statistical or facilitation resources

39. Future Questions Does a higher level of CQI implementation lead to better financial, quality and safety outcomes? If not, why? • Develop and Find Financial, Operational and HR Effectiveness and Efficiency Measures • Counte & Glandon, 1995 • Build one, clean database with CQI implementation scores and measures • Analyze to assess relationships

40. Future QuestionsIs CQI, as its described in the literature, an effective method for improving quality outcomes? • Change the four items that did not differentiate well 1. How many multi-disciplinary teams currently work to improve the processes of care in your organization? (Process) • more statistics, less teams • Human Factors Research • Toyota • Six Sigma 2. Do people in your organization know who their customers are? (Customer Focus) • not clear 3. Are employees in this organization encouraged to try new and better ways of doing things? (Innovation) 4. Is creativity actively encouraged in this organization? (Innovation) • healthcare has typically not been innovative and so these questions may need to be more specific

41. Acknowledgements and Questions Mentor – Dr. Counte Committee – Drs. Arrington, Rubio & Burroughs Dr. Dunagan Gretchen