Program Evaluation

1. Program Evaluation Needs Assessment Survey of Local Coalitions

2. How many evaluators does it take to change a light bulb? • One to do a needs assessment • One to do a feasibility study • One to do a qualitative study to find out what bulb to change • One to empower the bulb to change • One to tender a contract for further study • One to write performance indicators for success • One to do a cost benefit analysis of the best bulb to buy • One to do a meta-evaluation showing that all previous studies have left everyone in the dark

3. SO.......how many evaluators does it take? None, actually. Evaluators don't change bulbs, that's an implementation problem! -Program Evaluation gets a lot of flack!

4. But it can save your assets!

5. Objectives • Understand the purpose of program evaluation • Review local coalition survey results

6. Most commonly asked PE question • How do I know I’m making a difference? • Answer : Institute best practices… • With fidelity, innovation, and critical interpretation. • Unfortunately, outcome evaluation is expensive and rarely feasible with community intervention.

7. Most commonly asked PE question • How do I know I’m making a difference? • Answer : Institute best practices… • With fidelity, innovation, and critical interpretation. • Unfortunately, outcome evaluation is expensive and rarely feasible with community intervention.

8. Process evaluation helps implement best practices withfidelity. • Is my programming “evidence-based”? • Nicotine patches are great. Stick one over each eye and you can't find your cigarettes. • Is the “best practice” appropriate for my population? • What does the tobacco science say about adapting best practices to my population? • Am I appropriately following the science?

9. Process vs. Outcome Evaluation Processevaluation: Determines whether best-practices were adopted properly. Outcome evaluation: Determines whether program was successful.

10. Process Evaluation Assesses the delivery or implementation of a program. What happened and why? How is this different from what was planned? To be evaluated: • Activities • Materials • Delivery • Numbers of staff • Audience • Logistics

11. Process Evaluation TATU Example: How many youth participated in TATU trainings? What were the youths’ perceptions of each training? What were the implementation problems?

12. Outcome Evaluation Measures what progress the program has made towards its goals. To be evaluated: • Behaviors • Attitudes • Knowledge • Skills

13. Outcome Evaluation Example goals to evaluate for TATU: Short term – youth who participate in TATU gain outreach skills. Intermediate – TATU youth reach fellow students. Long term – Youth who TATU participants reach actually use skills to abstain when tobacco is available.

14. UW Tobacco Surveillance & Evaluation Program “It is now proved beyond doubt that smoking is one of the leading causes of statistics.” http://www.medsch.wisc.edu/mep/

15. UW Tobacco Surveillance & Evaluation Program • Indirectly provides outcome evaluation of Wisconsin’s coordinated tobacco control activities. • Fields surveys. • Analyzes data to monitor state trends. • Identifies progress and emerging challenges. • Future – more geographically specific information; better disparities data. http://www.medsch.wisc.edu/mep/

16. Conclusion : Uses of program evaluation • Integral part of every successful organization. • Clarifies and re-focuses program goals and strategies. • Verifies that current strategies are still appropriate. • Refreshes staff expertise in best practices and evidence-based programming. • Documentation for supporters, funders, and stakeholders.

17. Survey Results • Results reported here are for fully-funded coalitions ($20,000 or more). • 95% of fully-funded coalitions responded (40 of 42 coalitions that were contacted). • 76% of coalitions responded in total (55 of 72). • 94% of coalitions funded at $10,000 or more responded (44 of 47).

18. Survey Results:Is P.E. built into your regular program functioning?

19. Survey Results : The majority of coalitions (85%) engaged in some program evaluation.

20. Regional Distribution of Coalitions

21. Coalitions distributed geographically, rather than proportionate with population.

22. Number of full-time equivalent employees (FTEs) in Wisconsin: • 33 FTEs statewide • 0.82 per coalition (33 hours per week) • About 40¼ total staff.

23. FTEs by Region:

24. Indication of urban-efficiency gains. • Building partnerships is vital.

25. Most coalitions are funded entirely through a TPCP contract. • This raises the potential for instability given a potential change in state government. • What steps are coalitions taking to diversify?

26. Priority Populations (Choose three) • Targeting youth has likely paid off. • After rising through 1999, youth cigarette use now drops every year.

27. Priority Populations (Choose three) • More resources needed for 18-24 year olds.

28. Priority Populations (Choose three) • Less than 50% target policymakers. • “Other” populations reported include health care providers.

29. Coalition Objectives(Check all that apply) “Other” reported goals: • Reducing tobacco use among pregnant women • Increasing the tobacco tax.

30. Major Coalition Activities (Check all that apply) • Coalitions selected an average of 10 activities. • All selected more than one activity.

31. Major Coalition Activities (Check all that apply) • Less than 100% recruit supporters and seek community partnerships. • Less than ¼ pursue alternative funds.

32. Major Coalition Activities (Check all that apply) “Other” activities: • working with state partners. • helping worksites create individual policies. • asthma workgroups. • the N-O-T program. • Oral health screenings in disparate populations.

33. Back to Program Evaluation…

34. Current Program Evaluation Activities(Check all that apply) • More than 50% document their basic program activities. • Herein lies a wealth of experience to share.

35. Current Program Evaluation Activities(Check all that apply) “Other”: • focus groups • police reports • CDC’s SPF indicators • WI Wins data • First Breath statistics • parent surveys • oral health surveys • data from local agencies

36. In your own words… • Coalitions’ answers to open-ended survey questions: • What questions would you want a program evaluation to answer? • Please describe any sources of information/data that might be available for use in program evaluation.

37. 1. What questions would you want a program evaluation to answer? • Open-ended responses were grouped into: • process questions • outcome questions • technical needs

38. Process Questions: • Is the program operating as intended? • Are training materials and meeting times best? • Is the program structure optimal? • Are we reaching the intended population? • Are our activities “best practice”? • Are we using best practices for helping pregnant women quit? • Generally, what is the ranking of most- to least-valuable practices? • What is the need/demand for our program? • How important are non-cigarette tobacco products?

39. Process Questions: • Is the program operating as intended? • Are training materials and meeting times best? • Is the program structure optimal? • Are we reaching the intended population? • Are our activities “best practice”? • Are we using best practices for helping pregnant women quit? • Generally, what is the ranking of most- to least-valuable practices? • What is the need/demand for our program? • How important are non-cigarette tobacco products?

40. Outcome Questions: • What difference has the program made? • Are we changing attitudes about second hand smoke? • Have our education materials impacted attitudes about smoke-free workplaces? • What behavior changes resulted? • Have pledge cards increased smoke-free homes? • How many/who benefits? • Progress on goals of multi-year action plan? • Do results justify the costs?

41. Technical needs: • Where can we find detailed local statistics? • Where to find numbers of youth smokers, by school and by grade? • How can City of Milwaukee data be separated from Milwaukee county data? • How do we write survey questions that identify attitude and behavior changes? • Where can we find template surveys that can be changed to fit the needs of our community? • How can we use program evaluation data to promote tobacco control? • What are indicators of whether progress is being made?

42. 2. Please describe any sources of information/data that might be available for use in program evaluation. • Open-ended responses were grouped into: • Qualitative • Quantitative

43. Qualitative • Testimony from youth, school leaders, adults who work with youth, partners, law enforcement. • Documentation of earned media. • Minutes of Board of Health meetings. • Supporter data base. • Mid-year and year-end reports on consolidated contract tobacco objectives. • Community needs assessment (every 5 years). • Community Health Assessment data.

44. Quantitative • Statewide surveys/data: • YRBS, BRFS, UW Burden of Tobacco, UW survey of smoke-free municipal and government-owned buildings, Worksite survey, WIC data on tobacco use prevalence • Local surveys: • Dane County Youth Assessment, PRIDE survey in county schools, South West Youth Survey (SWYS), evaluation of local smoke-free ordinance, county smoke free air/tobacco use survey (400 residents surveyed), community health assessment data, survey data collected by community health improvement coalitions, oral health screening results • Program surveys/data: • Friends Helping Friends (youth peer group) survey data, WI Wins compliance checks data, First Breath data, Quit-line and Fax to Quit statistics, pre- and post-test data for participants, member survey

45. Next Steps… • Potentially, there may be regional trainings in program evaluation methods in each of the DHFS service areas. • Show of hands: • No PE training or experience • Some PE training • Some PE experience • Some PE success

46. Local coalitions team: Sue Marten, Chair Dianna Forrester Kathy Cahill Themis Flores Ramos Connie Olson Heidi Foster Deb Bruning Paula Silha Deb Gatzke Karen Hagemann Jody Moesch-Ebeling T&TA Lynn Habrik Kathy Cahill Deb Bruning Mary Hilliker Sue Marten Randy Glysch Cindy Musial Sue Marten Thanks!