PA 519– Health Programs Planning and Evaluation Second Semester 1439/ 1440

1. King Saud UniversityCollege of Business AdministrationDepartment of Health Administration Masters` Program PA 519–Health Programs Planning and Evaluation Second Semester 1439/ 1440 Mohammed S. Alnaif, Ph D.alnaif@ksu.edu.sa Dr. Mohammed Alnaif

2. HEALT H PROGRAM DEVLOPMENT and EVALUATION Learning Objectives Describe Program development, program planning and program evaluation. Understand the cycle of program planning and program evaluation. Introduces types of program evaluation. Use of the public health pyramid in program planning and evaluation. Dr. Mohammed Alnaif

3. Police EMS Corrections MCOs Health Department Philanthropist Mosques CommunityCenters Nursing Homes HomeHealth Parks Doctors Schools Elected Officials Hospitals MassTransit Environmental Health Civic Groups CHCs Fire Tribal Health Employers Laboratory Facilities Economic Development Drug Treatment Mental Health The Public Health System (Stakeholders) Dr. Mohammed Alnaif Source: Centers for Disease Control and Prevention

4. HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Objectives Expected outcomes Target population Mechanism(s) to deliver services (the intervention) Criteria for participating in the program A conceptual framework that provides rationale for program existence (sometimes called the Development Hypothesis). Dr. Mohammed Alnaif

5. HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Project planning involves a series of steps that determine how to achieve a particular community or organizational goal or set of related goals. This goal can be identified in a community plan or a strategic plan. Project plans can also be based on community goals. A Program is a group of clear, related, complementary activities that are intended to achieve a desired outcome among the target group(s). Dr. Mohammed Alnaif

6. HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): When organizations set out to make the world a better place, they develop programs to effect changes for individuals or groups. Programsvary, but the basic components of programs are essentially the same. All programs have objectives and a set of activities carried out by staff, volunteers and participants that are intended to achieve those objectives. Programsoperate within an organization and often within a broader system of similar programs. Dr. Mohammed Alnaif

7. HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Programdesign is the process that an organization uses to develop a program. It is most often an iterative process involving research, consultation, initial design, testing and redesign. A program design is the plan of action that results from that process. Dr. Mohammed Alnaif

8. HEALT H PROGRAM DEVLOPMENT and EVALUATION Five reasons to do program design and development You want to make sure that: There is really a need for the program; The design you are considering actually meets the needs of the intended participants; The program you are considering has been shown to be effective; The program has been appropriately adapted to fit your context; and The program is actually working for the participants. Dr. Mohammed Alnaif

9. HEALT H PROGRAM DEVLOPMENT and EVALUATION Stakeholder consultation Programs can have many stakeholders with many different points of view, depending on whether they are staff, participants, funders or other community service providers. Balancing the values and principles of a program with the practicalities of implementation often means compromise. But what compromises are likely to undermine the core values of the program? Stakeholder consultation can help to determine what is important, particularly to those implementing and those using the program. It can also contribute to adapting a model. Dr. Mohammed Alnaif

10. HEALT H PROGRAM DEVLOPMENT and EVALUATION What a program design includes: Need for the program – assesses what a population needs, what is available in community and the gap between what is needed and what is available (e.g. an organization serving youth notes that many of its participants need mental health supports, but need first to determine whether there is a broader need and whether there are existing programs). Program objectives and outcomes – what the program is intended to accomplish for the particular target population. Dr. Mohammed Alnaif

11. HEALT H PROGRAM DEVLOPMENT and EVALUATION What a program design includes: Program principles – these are the fundamental norms, rules and values that guide the program (e.g. being inclusive, client-focused). Program components – a specific set of inter-connected activities. provided. Budget/cost – what it will cost to implement the program. Funding strategy – how the program will be funded. Can it be done within the current organizational budget? Usually not, so organizations look to funders like the municipality, province, federal government. The funding strategy needs to include start-up and long-term funding. Governance – some programs are set within an existing organization or some programs may be collaboration among two or more organizations. The design needs to include the process for decision-making (e.g. an organization may have a board program committee that has responsibility for design of new programs as well as redesign of existing programs). Dr. Mohammed Alnaif

12. HEALT H PROGRAM DEVLOPMENT and EVALUATION Dr. Mohammed Alnaif

13. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Identify a potential need - This is not evidence of the need for a program; it is only evidence of the needs of people within the community or a population. This may create an opportunity for new program which will usually require additional resources or. Conduct a needs assessment – in order to determine the needs of a specific target population gather and analyze data taken from different sources in a systematic way, looking at the needs of the population and the existing programs in order to determine whether there are gaps. Dr. Mohammed Alnaif

14. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Research potential programs for filling need – deciding on how best to address an identified need requires searching for and reviewing existing programs. Contextual issues related to the demographic population, location and magnitude of the need are considered before selecting an appropriate program. Select an evidenced-based design – after a thorough review of potential programs, select one that shows strong or promising evidence of being effective. This means the design has been previously implemented and evaluated and has demonstrated some ability to produce a desired outcome for the target group. If such a program does not exist or is not suitable for the context, components of a program may be selected and adapted or a new program may be designed. If the program is new or innovative, that should be noted with reasons given for trying something new. Dr. Mohammed Alnaif

15. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Develop the program design – using a logic model (or theory of change) as a tool, create a detailed plan that specifies program activities, resource allocations and expected outcomes. The theory of change specifies the underlying assumptions of the program. Program design consolidates all of the information gathered during the development process and sets out the program’s outcomes, outputs, activities and resources. Dr. Mohammed Alnaif

16. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Pilot the program – before implementing a full-scale program, it is important to test whether it works in your particular context, if possible. Piloting includes developing the program on a small scale so you can gauge the level of readiness to deliver the program, determine whether it is making a difference for program users, and assess what, if any, changes are needed. A good pilot sets the stage for ongoing monitoring. Dr. Mohammed Alnaif

17. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Roll out the program – if the pilot indicates the program works, you need to select a site, hire and train program staff, establish administrative processes, start service delivery, and carry out ongoing monitoring and quality control. Monitor and evaluate – while testing or implementing the program, it is important to learn what’s working well and what isn’t. It is also helpful to review the program’s original goals. Dr. Mohammed Alnaif

18. HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program Monitor and evaluate – By using a systematic process that gathers and analyzes data from a variety of sources, steps can be taken to modify and improve the program before rolling it out or as it gets implemented. This also supports accountability to funders. Consult with stakeholders – speaking with and gathering information from all people involved with the program generates buy-in and surfaces diverse perspectives during the planning process. Stakeholder consultation must be genuine and responsive to be effective, and should take place throughout the planning process. Dr. Mohammed Alnaif

19. HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation Recently planners have been attempting to foster connections with public health professionals and improve health through planning actions. As issues such as obesity, asthma, venomous pollutants, and mental health concerns have grown in prominence, there is increasing interest in the potential for changes in the built environment to alleviate such problems. Dr. Mohammed Alnaif

20. HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation We need to distinguish between program planning and planning for a program evaluation, which can take place at any program stage of development. Health Planning is the identification and elaboration (within existing resources) of means and methods for providing in the future, effective health care relevant to identified health needs for a defined population. Dr. Mohammed Alnaif

21. HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation Aim of health planning is to maintain and improve health status of a given community. It is achieved through provision of health services which are accessible, effective, and equitable and of a quality to ensure their appropriate utilization. A plan of operations is the (written) summary of the implementation plan specifying sequence and timing of, and responsibilities for, implementation of main objectives, (expected) results and activities. Dr. Mohammed Alnaif

22. HEALT H PROGRAM DEVLOPMENT and EVALUATION Rationale for Health Planning Planning of delivery of effective health services to the population with in resources provided Translationof “new policy” statement into operational plan Re-planningon the basis of an already existing plan for the purpose of reviewing existing health problems and needs and rendering services more effective and efficient Emergenceof a new problem (e.g. SARS, natural disaster) Planningof health services for a population where no organized health care delivery system as yet, or where an existing one is being extremely revised or re-organized. Dr. Mohammed Alnaif

23. HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The current thinking about programs and evaluation is that the activities constituting program planning and program evaluation are cyclical and interdependent(FIGURE 1-1) The stages are cyclical to the extent that the end of one program or stage is flows almost seamlessly into the next program or planning activity Interdependenceof activities and stages ideally result from information and data feedback loops that connect the stages Dr. Mohammed Alnaif

24. (FIGURE 1-1) HEALT H PROGRAM DEVLOPMENT and EVALUATION Priority established Health Program Planning Statement of health problems Assessment of Community needs and Assets Health status changes Program and Evaluation Planning Findings from the evaluation Evaluation design and methodology Effects Evaluation Process theory and effect theory delineated Findings from the evaluation Participant/Recipient Health Outcome and Impact Process Evaluation Program Implementation of Process and Effect Theory Intervention effect Dr. Mohammed Alnaif

25. HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The cycle begin with a trigger event, such as awareness of health problems; a periodic strategic planning effort; a process required by a stakeholder, such as a grant renewal or a newly available fund for a health program. Failure of a health program The trigger might also be a news media expose or a legal action. Dr. Mohammed Alnaif

26. HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The trigger event or situation leads to the collection of data about the problem, the characteristics of the people affected, and their perception of the health problem. Based on the data from the needs assessment; Program development begin, Problems and their solutions are prioritized. Dr. Mohammed Alnaif

27. HEALT H PROGRAM DEVLOPMENT and EVALUATION Major Components of the Planning Phase Developing the program theory; Assessment for organizational and infrastructure resources for implementing the program; Setting goals and objectives that are derived from the program theory; Implementation of the program; Evaluation plan and cycle. Dr. Mohammed Alnaif

28. Evidence and Practice Based Planning Framework 1 Define Issues 7 Evaluation 2 Situational Assessment 6 Implementation 3 Discussion & Recommendation 5 Allocate Resources 4 Approve Decision Dr. Mohammed Alnaif

29. HEALT H PROGRAM DEVLOPMENT and EVALUATION Major Components of the Planning Phase Developing the program theory; Assessment for organizational and infrastructure resources for implementing the program; Setting goals and objectives that are derived from the program theory; Implementation of the program; Evaluation plan and cycle. Dr. Mohammed Alnaif

30. HEALT H PROGRAM DEVLOPMENT and EVALUATION Program life cycle A program may begin as a pilot; The program may not rely on any existing format or theory; Model programs; Regardless of the stage in a program’s life cycle then major planning and evaluation stages of community assessment and evaluation are carried out; The life cycle of a health program is reflected in the evolution of hospice care. Dr. Mohammed Alnaif

31. HEALT H PROGRAM DEVLOPMENT and EVALUATION The Fuzzy Aspects of planning Several Paradoxes pervade health planning; That planning is shaped by the same forces that created the problems that planning is supposed to correct. Prosperity in our modern world has its own associated health risks. What may be easier and more effective may be less acceptable (active protection, passive protection). Those in need ideally but rarely trigger the planning of health programs. The fact is that planning is intended to be successful. Most planning is for making changes not for creating stability. Dr. Mohammed Alnaif

32. HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. The first is that a solution, remedy, or appropriate intervention can be identified or developed and provided. Without this assumption, planning would be pointless. It is fundamentally an optimistic assumption about the capacity of the planners, the stakeholders, and the state of the science to address the health problem. The assumption of adequate capacity and knowledge is actually tested through the process of planning. Dr. Mohammed Alnaif

33. HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Planning leads to the allocation of resources needed to address the health problem. This assumption is challenged by the reality that four groups of stakeholders have interests in the decision making regarding health resources and each group exists in all program planning. Members of the target audience for the health program Health payers, such as insurance companies and philanthropic funding agencies Individual healthcare providers and healthcare organizations and networks The general public are effected by how resources are allocated for health programs Dr. Mohammed Alnaif

34. HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Another assumption about those involved is that they share similar views on how to plan health programs. During the planning process their points of view and cultural perspectives will likely come into contrast. Planners need to know what is relevant and important for the problem at hand. There are competing purposes and values. Effective planning requires tolerance, freedom, and fairness Dr. Mohammed Alnaif

35. HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. The resulting conflicts among the stakeholders for the limited resource apply whether they are allocating resources across the healthcare system or among programs for specific health problems. Limited resources, whether real or not, raise ethicalquestions of what to do when possible gains from needed health programs or policies are likely to be small, especially when the health program addresses serious health problems. Dr. Mohammed Alnaif

36. HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Another assumption about the planning process is that it occurs in an orderly fashion and that a rational approach is best. Four key elements are inherent in planning; uncertainty, ambiguity, risk, and control. The presence of each of these elements contradicts the assumption of rational approach, and each generates its own paradoxes. Dr. Mohammed Alnaif

37. HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control The process of planning is affected by the limits of both scientific rationality and the usefulness of data to cope with the uncertainties, ambiguities, and risks being addressed by the planning process (see TABLE 1-3). Uncertaintyis the unknown likelihood of a possible outcome. Four type of uncertainty are identified: types and amount of resources, technological, market receptivity to the product, and organizational. Dr. Mohammed Alnaif

38. Dr. Mohammed Alnaif

39. HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Ambiguity is doubt about a course of action stemming from awareness that known and unknown factors exist that can decrease the possibility of certainty. In this sense, ambiguity results in uncertainty Both uncertainty and ambiguity pervade the planning process because it is impossible to know and estimate the effect of all relevant factors. Dr. Mohammed Alnaif

40. HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Ambiguity is the characteristic of not having a clear or single meaning. Change, or the possibility of change, is a possible source of ambiguity. When ambiguity is ignored, the resulting differences in interpretation can lead to confusion and conflict among stakeholders. Openly and constructively addressing the ambiguity and any associated conflict can lead to innovations in the program. Dr. Mohammed Alnaif

41. HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Risk is the perceived possibility or uncertain probability of an adverse outcome in a given situation. Risk is not just about taking chances but it is also about uncertainty and ambiguity ( as in the case with estimates of cure rates and projections about future health conditions). Risk is pervasive and inherent throughout the planning process in terms of deciding who to involve and how, which planning approach to use, which intervention to use, and I estimating which health problem deserves attention. Dr. Mohammed Alnaif

42. HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Control, as in being in charge of managing, is a natural reaction to the presence of ambiguity, conflict, and risk. It can take the form of directing attention and allocating resources or of exerting dominance over others. Addressing the ambiguity, uncertainty, and risk that might have been the trigger for the planning process require less—not more—control. Those who preside over and influence the planning process are often thought of having control over solutions to the health problem or condition. Dr. Mohammed Alnaif

43. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Some think of evaluations as falling into one of two broad categories: formative and summative evaluations. Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program. Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals. Dr. Mohammed Alnaif

44. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Several major types of activities are classified as evaluations. each type of activity requires a specific focus, purpose, and set of skills. Which of these evaluations is most appropriate depends on the stage of your program: A community needs assessment Process evaluations Effect evaluations Outcome evaluations Impact evaluations Cost evaluations Comprehensive evaluations Meta-evaluations Dr. Mohammed Alnaif

45. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Unfortunately, contrasting summative evaluations with formative evaluations as the two broad categories adds more confusion to the evaluation terminology. Summative evaluations, in the strictest sense, are done at the conclusion of a program to provide a conclusive statement regarding program effects. The term summative evaluation is sometimes used to refer to either an outcome or impact evaluation. The term formative evaluation is used to refer to program assessments that are performed early in the implementation of the program and used to make changes to the program. Dr. Mohammed Alnaif

46. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Community needs assessment (also known as community health assessment) is a type of evaluation that is performed to collect data about the health problems of a particular group. Needs Assessment, determines who needs the program, how great the need is, and what can be done to best meet the need. Dr. Mohammed Alnaif

47. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation The data collected for this purpose are then used to tailor the health program to the need and distinctive characteristics of that group. A community needs assessment is a major component of program planning because it is , done at an early stage in the program planning and evaluationcycle. Dr. Mohammed Alnaif

48. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Process evaluations, which begins at the same time that the program starts, and focus on the degree to which the program has been implanted as planned and on the quality of the program implementation. Process evaluations are known by a variety of terms, such as monitoring evaluations, depending on their focus and characteristics. Dr. Mohammed Alnaif

49. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation The underlining framework for designing a process evaluation comes from the process theory component of the overall program theory developed during the planning stage. The process theory delineates the logistical activities, resources, and interventions needed to achieve the health change in program participants or recipients. Information from process evaluation is used to plan, revise, or improve the program. Dr. Mohammed Alnaif

50. HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Effect evaluations answer a key question: did the program make a difference? Evaluators seek to use the most rigorous and robust designs, methods, and statistics possible and feasible when conducting an effect evaluation. Findings from effect evaluations are used to revise the program and may be used in subsequent initial program planning activities. Dr. Mohammed Alnaif