Intro to Program Evaluation

1. Intro to Program Evaluation Defining Terms

2. Defining Evaluation • Evaluation is the systematic investigation of the merit, worth, or significance of any “object” Michael Scriven • Programis any organized action/activity implemented to achieve some result

3. These must be integrated… What do we do? • Continuous Quality Improvement (CQI) cycle. • Planning—What actions will best reach our goals and objectives. • Performance measurement— How are we doing? • Evaluation—Why are we doing well or poorly? Why are we doing well or poorly? How do we do it? How are we doing?

4. Introduction to Program Evaluation CDC’s Evaluation Framework

5. Framework forProgram Evaluation 5

6. Step-by-Step 1. Engage stakeholders: Decide who needs to be part of the design and implementation of the evaluation for it to make a difference. 2. Describe the program: Draw a “soup to nuts” picture of the program—activities and all intended outcomes. 3. Focus the evaluation: Decide which evaluation questions are the key ones

7. Step-by-Step Seeds of Steps 1-3 harvested later: 4. Gather credible evidence: Write indicators and choose and implement data collection sources and methods 5. Justify conclusions: Review and interpret data/evidence to determine success of failure 6. Use lessons learned: Use evaluation results in a meaningful way.

8. CDC’s Framework for Program Evaluation Steps 1 Engage stakeholders 6 Ensure use and share lessons learned 2 Describe the program Standards Utility Feasibility Propriety Accuracy 3 Focus the evaluation design 5 Justify conclusions 4 Gather credible evidence Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11).

9. Introduction to Program Evaluation Step 2. Describing the Program

10. CDC’s Framework for Program Evaluation Steps 1 Engage stakeholders 6 Ensure use and share lessons learned 2 Describe the program Standards Utility Feasibility Propriety Accuracy 3 Focus the evaluation design 5 Justify conclusions 4 Gather credible evidence Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11).

11. You Don’t Ever Need a Logic Model BUTYou Always Need a Program Description Don’t jump into planning or evaluation without clarity on: • The big “need” your program is to address • The key target group(s) who need to take action • The kinds of actions they need to take (your intended outcomes or objectives) • Activities needed to meet those outcomes • “Causal” relationships between activities and outcomes

12. Logic Models and Program Description • Logic Models : Graphic depictions of the relationship between your program’s activities and its intended effects

13. “Complete” Logic Model Long-term Outcomes/Impacts Intermediate Outcomes Short-term Outcomes Inputs Outputs Activities What the program needs… What the program does… Who or what will change because of the program… Context and Assumptions External factors that influence getting to outcomes

14. What the program and its staff actually do Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes/Impacts Inputs Outputs Activities Context Assumptions

15. Results of activities: Who/what will change? Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes/Impacts Inputs Outputs Activities Context Assumptions

16. Inputs Activities Activities Outputs Outcomes Short-term Long-term Recruit and engage Corner Store Owners Provide technical assistance Neighborhood Corner Stores stock and sell healthier food options • Funding: • Federal • Foundation Community/Systems Increased variety and quality of fresh fruits and vegetables offered Link stores with local food producers Work with local organizations to engage in community education & outreach Increased number of corner stores selling more fresh fruits and vegetables Riverside Health Department Program Staff Healthy and affordable fruits and vegetables are available at Corner Stores Increased revenue at corner stores Conduct community outreach and education around nutrition Research and Best Practices Neighborhood residents are aware of fresh fruits and vegetables at corner store Individual Engage schools in nutrition education and outreach Increased awareness of availability of fruits and vegetables at corner stores Improved overall dietary quality among neighborhood residents Conduct nutrition education to parents, teachers, staff, and students Corner Store Owners Students, parents, teachers and staff have information and access to healthier food options Increased purchasing of fresh fruits and vegetables at corner stores by neighborhood residents Recruit and train Youth Advocates Reduced income and racial/ethnic disparities in obesity and nutrition Local Community Organizations Plan and conduct Health Summit with community leaders Community leaders are aware of neighborhood nutrition needs Improved nutritional knowledge and attitudes among neighborhood residents Riverside School District and local schools Reduced prevalence of childhood and adult obesity Produce film of local nutrition environment Neighborhood nutrition needs are documented

17. Informs Two Steps in CDC Evaluation Framework • In Step 1. Engage Stakeholders: • Who are major stakeholders for our efforts? • Where in this model do they want to see success? • Who needs to be engaged upfront to ensure use of results? • In Step 3. Setting Evaluation Focus: • Today, 1 year, 5 years, 10 years, where in the model should I be measuring changes? • If no change, where should I look for problems?

18. Intro to Program Evaluation Step 1. Engaging Stakeholders

19. CDC’s Framework for Program Evaluation Steps 1 Engage stakeholders 6 Ensure use and share lessons learned 2 Describe the program Standards Utility Feasibility Propriety Accuracy 3 Focus the evaluation design 5 Justify conclusions 4 Gather credible evidence Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11).

20. Who are Stakeholders? • Three major groups: • Those served or affected by the program • Those involved in program operation • Primary intended users of the evaluation findings

21. Which Stakeholders Matter Most? Who is: Affected by the program? Involved in program operations? Intended users of evaluation findings? Of these, who do we most need to: Enhance credibility? Implement program changes? Advocate for changes? Fund, authorize, expand program?

22. Knowing Your Audience • What actions do you want them to take? • What do they need or want to know? • Why should they care? • What is the most convincing way to communicate with them?

23. Triggering the Audience’s Emotions Fairness Communities are unfairly burdened Ingenuity Communities can and do work together Prevention Responsibility to prevent problems

24. Intro to Program Evaluation Step 3. Setting Evaluation Focus

25. CDC’s Framework for Program Evaluation Steps 1 Engage stakeholders 6 Ensure use and share lessons learned 2 Describe the program Standards Utility Feasibility Propriety Accuracy 3 Focus the evaluation design 5 Justify conclusions 4 Gather credible evidence Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11).

26. Evaluation Can Be About Anything • Evaluation can focus on any/all parts of the logic model • Evaluation questions can pertain to • Boxes---did this component occur as expected • Arrows---what was the relationship between components

27. Phases and Types of Evaluation FORMATIVE SUMMATIVE Source: Based on slides from Jennifer Nichols, Porter Novelli

28. (Some) Potential Purposes Test program implementation Show accountability “Continuous” program improvement Increase the knowledge base Other… Other… 28

29. (Some) Potential Purposes Test program implementation Show accountability “Continuous” program improvement Increase the knowledge base Other… Other… 29

30. Inputs Activities Activities Outputs Outcomes Short-term Long-term Recruit and engage Corner Store Owners Provide technical assistance Neighborhood Corner Stores stock and sell healthier food options • Funding: • Federal • Foundation Community/Systems Increased variety and quality of fresh fruits and vegetables offered Link stores with local food producers Work with local organizations to engage in community education & outreach Increased number of corner stores selling more fresh fruits and vegetables Riverside Health Department Program Staff Healthy and affordable fruits and vegetables are available at Corner Stores Increased revenue at corner stores Conduct community outreach and education around nutrition Research and Best Practices Neighborhood residents are aware of fresh fruits and vegetables at corner store Individual Engage schools in nutrition education and outreach Increased awareness of availability of fruits and vegetables at corner stores Improved overall dietary quality among neighborhood residents Conduct nutrition education to parents, teachers, staff, and students Corner Store Owners Students, parents, teachers and staff have information and access to healthier food options Increased purchasing of fresh fruits and vegetables at corner stores by neighborhood residents Recruit and train Youth Advocates Reduced income and racial/ethnic disparities in obesity and nutrition Local Community Organizations Plan and conduct Health Summit with community leaders Community leaders are aware of neighborhood nutrition needs Improved nutritional knowledge and attitudes among neighborhood residents Riverside School District and local schools Reduced prevalence of childhood and adult obesity Produce film of local nutrition environment Neighborhood nutrition needs are documented

31. Process Evaluation The type and quantity of services provided What actually happens during implementation—implementation “fidelity” The number of people receiving services The number of coalition activities and meetings How much money the project costs The staffing for services/programs 31

32. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Tangible products of activities Context Assumptions Stage of Development 32

33. Resource “platform” for the program Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Context Assumptions 33

34. (Some) Potential Purposes Test program implementation Show accountability “Continuous” program improvement Increase the knowledge base Other… Other… 34

35. Outcome Evaluation Results of program services Changes in individuals Knowledge/awareness Attitudes Beliefs Changes in the environment Changes in behaviors Changes in disease trend 35

36. Inputs Activities Activities Outputs Outcomes Short-term Long-term Recruit and engage Corner Store Owners Provide technical assistance Neighborhood Corner Stores stock and sell healthier food options • Funding: • Federal • Foundation Community/Systems Increased variety and quality of fresh fruits and vegetables offered Link stores with local food producers Work with local organizations to engage in community education & outreach Increased number of corner stores selling more fresh fruits and vegetables Riverside Health Department Program Staff Healthy and affordable fruits and vegetables are available at Corner Stores Increased revenue at corner stores Conduct community outreach and education around nutrition Research and Best Practices Neighborhood residents are aware of fresh fruits and vegetables at corner store Individual Engage schools in nutrition education and outreach Increased awareness of availability of fruits and vegetables at corner stores Improved overall dietary quality among neighborhood residents Conduct nutrition education to parents, teachers, staff, and students Corner Store Owners Students, parents, teachers and staff have information and access to healthier food options Increased purchasing of fresh fruits and vegetables at corner stores by neighborhood residents Recruit and train Youth Advocates Reduced income and racial/ethnic disparities in obesity and nutrition Local Community Organizations Plan and conduct Health Summit with community leaders Community leaders are aware of neighborhood nutrition needs Improved nutritional knowledge and attitudes among neighborhood residents Riverside School District and local schools Reduced prevalence of childhood and adult obesity Produce film of local nutrition environment Neighborhood nutrition needs are documented

37. “Reality Checking” the Focus Based on “feasibility” standard: Stage of Development: How long has the program been in existence? Program Intensity:How intense is the program? How much impact is reasonable to expect? Resources: How much time, money, expertise are available? 37

38. Moderators: Contextual factors that will facilitate or hinder getting our outcomes Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Context Assumptions 38

39. Moderators/Contextual Factors Political Economic Social Technological 39

40. Did we get the inputs we needed/were promised? Process Evaluation Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Were activities and outputs implemented as intended? How much? Who received? Context Assumptions Stage of Development

41. Outcome Evaluation Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Which outcomes occurred? How much outcome occurred Context Assumptions Stage of Development

42. Efficiency Evaluation Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities (How) was implementation quality related to inputs? Context Assumptions Stage of Development

43. Causal Attribution Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Outputs Activities Did outcomes occur because of our activities and outputs? Context Assumptions Stage of Development

44. Intro to Program Evaluation Step 4: Gathering Credible Evidence

45. CDC’s Framework for Program Evaluation Steps 1 Engage stakeholders 6 Ensure use and share lessons learned 2 Describe the program Standards Utility Feasibility Propriety Accuracy 3 Focus the evaluation design 5 Justify conclusions 4 Gather credible evidence Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11).

46. Evaluation Plan

47. Evaluation Plan—Core

48. What is an indicator? Specific, observable, and measurable characteristics that show progress towards a specified activity or outcome. 48

49. Not “Collect Data” BUT “Gather Credible Evidence” Narrowing from 100s of ways to collect data: Utility: Who’s going to use the data and for what? Feasibility: How much resources? Propriety: Ethical constraints? Accuracy: How “accurate” do data need to be? 49

50. Not “Analyze Data” BUT “Justify Conclusions” Utility: Who’s going to use the data and for what? Feasibility: How much resources? Propriety: Ethical constraints? What does “ethical” mean? Accuracy: How “accurate” do we need to be? What does “accurate” mean? 50