Performance Evaluation in the Healthcare

1. Performance Evaluation in the Healthcare Heesuk Yun Korea Development Institute March 17, 2004

2. Need for Performance Evaluation in the Health Field • Expenditure for public health is rapidly increasing in Korea • Targets of public health actions is expanding beyond infectious diseases • Increasing emphasis on prevention, but efficacy  is hard to prove • Pressure of cost containment and Importance of accountability of resources invested in health field • The need for rigorous evaluation will increase

3. This paper attempts to • Summarize the essential elements and special characteristics of performance evaluation in the health field • Introduce the U.S. experience of devising indicators in the health field • Review standards for effective evaluation of health interventions • Extract  implications for establishing a performance evaluation in the health field in Korea

4. Objectives of Public Health Intervention • Multiplicity: objectives range from the most general such as reducing mortality to the very specific such as reading a health pamphlet. • Objectives and sub-objectives can be hypothesized corresponding to the steps or actions of a program. • A continuous series of events →divided into a hierarchy of sub-goals for evaluation purposes Each is  the result of successfully achieving the preceding goal, and a precondition to the next higher goal.

5. Reduction in morbidity or mortality Number or proportion of persons who are meeting prescribed and accepted standards Number of persons who change behavior pattern in accordance with the new knowledge Number or proportion of persons who change opinions or attitudes from the new knowledge Number or proportion of persons who learn the facts contained therein Number or proportion of persons who glance at or read it Number or proportion of persons who see the material Number or proportion of persons who receive the material Number of requests received for the material, or number distributed Number of pieces of literature available for distribution Pretesting of literature by special readability formulas Cumulative chain of objectives

6. Needs to prove the intervening assumptions through careful research • An assumption needed  whenever one moves from a higher-order objective to a lower one. Every lower-level objective must assume all of the assumptions made for all of the objectives above it. • Any program based on a false set of assumptions cannot be justified, even if sound evaluations are completed for each of the other objectives individually.

7. Special Characteristics of Performance Evaluation in the Health Field: Scientific rigor • Intervention logic: an explanation of what the public action is supposed to achieve and how it is supposed to achieve it • Causal assumptions are hidden in the logic • Involves scientific theory   • Identifying the hidden assumptions and investigating their uncertainties are more complicating

8. Older Than 17 Years .42 .78 No Social Support -.13 .55 .41 .75 .90 Economic Stress .35 Addictive Behavior Low Birth Weight .28 .89 .30 Family Stress .32 .35 Medical Risks .19 Smoking and Low Birth Weight

9. Addictive behaviors such as smoking and alcohol consumption have direct effect on low birth weight • Economic stress has direct effects on family stress and social support • Family stress and social support have direct effects on addictive behavior • The three stress variables have indirect effects on low birth weight • The standardized regression coefficients indicate the size of the change from various factors

10. Special Characteristics of Performance Evaluation in the Health Field: Contextual Complexity • Complicated contexts including historical, geographical settings, political, social and economic conditions, and influences of related or competing organizations • Need to consider the needs of the target population and the particular problems • Difficult to identify what effects are genuinely caused by a program and to separate these effects from other influences on the socio-economic problems • Needs to understand these intertwined factors, to design a context-sensitive evaluation, to interpret findings accurately and to assess the generalizability of the findings.

11. Political Context: • Health Care interventions are often created to reduce health disparities across socio-economic groups • Evaluations must consider political aspects among income groups, politicians, and interest groups • Evaluations must be conducted in a political context where groups compete in their own interests.

12. Methods in Performance Evaluation for Health Care Intervention: Customizing Tools for Evaluation • Health care's  special features due to scientific hypotheses or the contextual particularity • Need to understand the intervention specific components and process • Golden rule: there is no single evaluation methodology which is universally applicable for the entire health field. The choice of method determined by the particular evaluation problems

13. Prerequisites for Starting: Evaluation Infrastructure (1) Construction of Representative Frameworks for Evaluating Public Healthcare Intervention • Standardized framework supports a practical approach based on steps and standards applicable in public health settings. • Provides a guide for designing and conducting specific evaluation projects • Can be used as a template to create or enhance program-specific evaluation guidelines that further operationalize the step sand standards in ways that are appropriate for each intervention

14. Guided by the steps and standards in the framework: • Program planning will also evolve • Integrated information systems will support a more systematic measurement • Lessons learned from evaluations can be used more effectively to guide changes in public health strategies. • General framework can encourage evaluation research to be integrated with routine program operations.

15. Prerequisites for Starting: (2) Constructing Indicators for Major Healthcare Evaluation Types • System of criteria to measure the effects of an intervention • Where intervention logic is grounded on scientific knowledge and all the political, social, and economic interests, Indicators should function as the guiding rules • Should translate general concepts regarding the intervention, its context, and its expected effects into specific measures that can be interpreted, while providing a basis for collecting evidence

16. Developing indicators should be based on intervention logic • For each step of the model, qualitative and quantitative indicators  developed to suit the concept in question, linking assumption, the information available, and the planned usage of data. • By relating indicators to intervention logic, • Detect changes in performance faster than when relying on a single outcome as the only performance measure • Lines of responsibility and accountability are clarified as the measures are aligned with each step of the program strategy • Detect the consequences of intermediate effects on health outcomes of the program.

17. Prerequisites for Starting: (3) Preparing Date Sources for Evaluation • When an indicator is proposed for use in performance monitoring, appropriate data must be available to support an indicator • Data should be collected from the specific population of interest, within the relevant time frame, using valid, reliable, and responsive measures • Often, performance evaluations have to rely on data collected for another purpose • Evaluators must understand limitations on the applicability of the data.

18. Examining Indicators for Healthcare Evaluation: Experience of U.S. • Panel on Performance Measures and Data for Public Health Performance Partnership Grants • Recommend performance measures in ten areas: chronic disease; sexually transmitted disease (STD), human immunodeficiency virus (HIV) infection, and tuberculosis; mental health; immunization; substance abuse;  sexual assault, disabilities, and emergency medical services

19. Panel's Framework for Assessing Suggested Indicators • Health outcome measures are widely used but insufficient • Many measures of health outcomes are affected by various factors that are not under the health intervention's control → changes in outcomes cannot be attributed only to specific program effectiveness. • Many important public health objectives, such as lowering the incidence of cancer and HIV infection, cannot be achieved over short periods of time to derive an outcome measure.

20. Health outcome: change in the health of a defined population related to an intervention. • Risk status (intermediate outcome): change in the risk demonstrated or assumed to be associated with health status. • Process: what is done for the defined groups as part of the delivery of services, such as performing a test or procedure or offering an educational service. • Capacity: the ability to provide specific services, such as clinical screening and disease surveillance

21. Example: CHRONIC DISEALSE • Prevention of chronic disease is the primary goal of many health programs, • But chronic disease incidence and mortality data are not useful because • The expected time period between  prevention activities and the effect exceeds the time that health departments are generally willing to wait to assess the intervention's effectiveness • Potential chronic disease measures are focused on risk reduction and screening, and process measures.

22. Potential Risk Status Measure • Percentage of (a) persons aged 18-24 and (b) persons aged 25 and older currently smoking tobacco • Who eat five or more servings of fruits and vegetables per day • Who do not engage in physical activity or exercise • Who had their blood pressure checked within past 2 years

23. Examples of Process Measure • Nutrition Program Strategy: Percentage of schools with menus that meet dietary guidelines • Physical Activity Program Strategy: Percentage of worksites with worksite wellness programs • Smoking Program Strategy: Percentage of vendors who illegally sell smoking tobacco to minors • Screening Program Strategy: Percentage of persons with diabetes receiving diabetes health education

24. Examples of Capacity Measures • Resources: Number of full-time health department employees for chronic disease prevention • Proficiencies: Number of key surveillance systems and data sets that are establish and maintained • Planning: Percentage of population served by systematic community planning process, with leadership provided by the official health agency • Community Involvement: Proportion of health department programs that operate within the framework of a community coalition or have a community advisory group

25. Summary • Performance evaluation in the health field is an increasingly important area • Evaluation of public health action: use of rigorous scientific methods and contextual complexity requiring socio-economic and political awareness. • Setting up an evaluation system requires building infrastructure 1) Framework for evaluating public health intervention needs to be devised  2) Indicators need to be collected and assessed for future use 3) Appropriate data sources for each indicator should be gathered and organized. and following up data collection should be made within a reasonable time.