Evaluating the Initiative

1. Evaluating the Initiative Community Tool Box Curriculum Module 12

2. What have been your previous experiences in evaluation?Is your partnership or coalition currently undergoing any type of evaluation?

3. Concepts and Attributes of Evaluation Evaluation: Systematic investigation of the merit, worth, or significance of an object or effort

4. Concepts and Attributes of Evaluation Evaluation as: • A Process: ongoing, interactive (repeating) • A Product: information used for continuous improvement

5. Why an Evaluation? • Help understand how efforts work • Ongoing feedback can improve community work • Gives community members voice and opportunity to improve efforts • Hold accountable those groups doing, supporting, and funding the work

6. A Bill of Rights and Responsibilities for Community Evaluation • Contribute to understanding • Contribute to improvement • Encourage participation of all stakeholders • Respond to interests of different stakeholders • Capture the dynamic nature of the work • Provide clear and timely information • Be practical, with the benefits outweighing the costs • Do no harm to the community • Help clarify the initiative’s contribution to more distant outcomes • Strengthen capacity

8. Concepts and Attributes of Evaluation DIALOGUE: Which of these attributes is particularly important to your community initiative? How will this attribute be assured in your evaluation?

9. Overview of an Evaluation Plan • Identify stakeholders • Describe the program • Focus the evaluation design • Gather credible evidence • Make sense of data and justify conclusions • Use information to celebrate, make adjustments and communicate lessons learned

10. “We should make things as simple as possible, but not simpler.” -- Albert Einstein

11. Determining Who Cares and What They Care About Identify Who Cares – Stakeholders: • Those involved in operating the program or initiative • Those served or affected • Primary intended users of the evaluation (e.g., funders, staff, researchers)

12. DIALOGUE: Who cares about your effort and its effects?

13. Identify What They Care About: • Different stakeholders may have different perspectives. Ask. What evaluation questions or aspects are of interest to: • Community groups • Grantmakers and funders • Outside researchers

14. Evaluation Stakeholders & Interests Program or Initiative: ______________________

15. Developing Evaluation Questions • An evaluation question seeks information stakeholders want to know about the functioning of the program • Examples: • Did the intervention have the desired effect? • With whom? • Under what conditions?

16. Categories of Evaluation Questions • Process • Planning and implementation issues • Outcome • Attainment of objectives • Impact on participants • Impact on the community

17. Process Measures Measures for planning and implementation • How important are the goals? • How well was the effort planned? • How well was it implemented? • Who participated? How many? • Did those most affected contribute to planning, implementation, and evaluation? • How satisfied are stakeholders withthe program?

18. Outcome Measures Attainment of objectives • How well has the program met its stated objectives?

19. Developing Outcome Measures Impact on participants • How much and what kind of difference has the program made for its targets (e.g., in knowledge, behaviors, outcomes)? • From the perspective of participants (and outside experts), how significant were the effects?

20. Developing Outcome Measures Impact on the community • How much and what kind of difference has the effort made on the community (e.g., in community and systems changes, rates of behavior, population-level outcomes)? • From the perspective of participants (and outside experts), how significant were the effects? • Were there any unintended consequences? Positive? Negative? • Is the community’s capacity to address problems improved—across issues, over time?

21. ACTIVITY: What evaluation questions related to process measures or outcome measures is your group interested in?

22. Evaluation Questions Program or Initiative: _______________________

23. Developing Meaningful Questions Consider: • What exactly was it that you originally hoped to accomplish? • How did you plan to do so? • What does your strategic plan suggest? • What does your logic model suggest—activities, outputs, and outcomes?

24. Evaluating the Initiative “Wonder is the beginning of wisdom.” --Anonymous

25. Gathering Evidence to Address the Evaluation Questions • Evidence—information that could be used to assess the merit or worth of a program • Gathering credible evidence—Overview: • Indicators of Success • Sources of Evidence • Quality of Evidence • Quantity of Evidence • Logistics for Gathering Information

26. Indicators of Success Translate expected effects into specific measurable units • Examples include: • Program outputs (e.g., services delivered) • Participation rates • Levels of satisfaction • Intervention exposure, or dose • Changes in communities and systems • Changes in behaviors of participants, populations • Population-level outcomes

27. DIALOGUE: What are some indicators of success for the Austin Healthy STEPS Initiative?

28. Sources of Evidence • People and Surveys • Documents (e.g., archival data, statistics) • Observation and Experimentation • More than one source helps make evidence more compelling

29. DIALOGUE: What are some possible sources of evidence for the Austin Healthy STEPS Initiative?

30. Quality of Evidence • Appropriateness and integrity of information • Reliability • Validity • Relationship to the evaluation questions • High quality data are both accurate and sensitive

31. Quantity of Evidence • Sufficient data to draw conclusions • Enough to answer evaluation questions • Adequate for data analysis • Enough to draw conclusions • Not more than is necessary

32. Logistics for Gathering Information • Methods • Timing • Infrastructure • Attention to cultural norms • Guarantee of confidentiality

33. Selecting Evaluation Methods • Surveys about satisfaction and importance of the effort • Goal attainment reports • Behavioral surveys • Interviews with key participants • Archival records

34. Selecting Evaluation Methods • Observations • Self-reports, logs or diaries • Documentation systems and analysis of contribution • Community-level indicators of impact • Case studies and experiments

35. Ongoing Documentation and Evaluation • Document unfolding of intervention: • Community and systems changes (i.e., new or modified programs, policies and practices) • Analysis of contribution to population-level outcomes—Amount by: • Goal • Intensity of behavior change strategy • Duration • Penetration to Targets throughSectors in Places

36. Assembling Evidence Chart Program or Initiative: ______________________

37. Gathering Evidence to Address the Evaluation Questions DIALOGUE: What types of evaluation methods would be useful to the STEPS initiative? DIALOGUE: Which of these evaluation questions would be of interest to key stakeholders for your initiative? What are indicators of success? What evaluation methods would you use?

38. Evaluating the Initiative “The best test … is how well the modeler can answer the questions, ‘What do you know now that you did not know before?’ and ‘How can you find out if it is true?’” --James M. Brower

39. Using Evaluation Data toLearn and Make Adjustments Making Sense of the Data and Justifying Conclusions • Standards • Analysis and Synthesis • Sense Making and Interpretation • Judgments • Recommendations

40. Using Evaluation Data toLearn and Make Adjustments DIALOGUE: How will the Austin Healthy STEPS group use the information to make judgments about the merit of the effort?

41. Some Ways to Use the Information • Celebrate accomplishments • Make adjustments • Communicate lessons learned

42. Using the Information • Plan for use • Consider the actual evaluation results • Encourage use of learning from the evaluation process

43. Keeping Your “Eyes On The Prize” • Ultimate goal is to improve outcomes • May take too long to be useful • Crucial to document intermediate outcomes to support movement toward more distant outcomes

44. Using Evaluation Data toLearn and Make Adjustments DIALOGUE: How might the STEPS initiative use their findings to learn and make adjustments? ACTIVITY: Who might use the evaluation findings to learn and make adjustments to your program? Use the following chart to map out the key questions you are going to ask, potential findings, and related recommendations.

45. Evaluation Questions of Interest to Key Stakeholders (e.g., Did the program result in behavior change? Do people like the program?) Actual (Potential) Findings and Key Conclusions (e.g., The results showed that there was only a slight increase in… The findings suggest that …) Actual (potential) Recommendations (e.g., Consistent with these findings, we recommend that… including…) Using an Evaluation Information Chart

46. Communicating the Findings to Relevant Audiences How, when, and what to communicate from an evaluation is a strategic decision.

47. Sharing Lessons Learned Strategy Issues • Timing • Style • Tone • Message Source • Vehicle • Format of the information products

48. Communicating the Findings to Relevant Audiences Broadening Your Dissemination • Formal evaluation • Methods for delivering evaluation data: • Press releases • Internet distribution, web pages, links • Storytelling • Presentations and publications • Consulting by group members or teams • Use multiple and diverse methods

49. ACTIVITY: Use the chart on the next slide to identify the message, audience, plan and channel to communicate the findings of your evaluation.

50. The Message - Actual (Potential) Findings Plus Recommendations Communications Plan AUDIENCE Who should receive this message? SOURCE Who should deliver it? CHANNEL How should it be delivered? (e.g., personal contact, written report, media, professional presentation or publication) Communicating Results (e.g., The results show… The findings suggest… We recommend…)