Evaluating Your Program: How Do You Know If It Worked?

1. Evaluating Your Program:How Do You Know If It Worked? Susan Donaldson Cooperative Extension Bringing the University to You

2. What is the purpose of the evaluation? • To improve a program, or your own teaching • To help others (including a funding agency) understand the program and its results • To guide future actions/programs • To determine if the program is worth the cost • To win support and/or funding • For academic interest (!)

3. PLAN THE EVALUATION AT THE SAME TIME AS YOU PLAN THE PROGRAM! What do I want to know, and how will I know it?

4. Design your program with specific learning or action objectives in mind • What results do you want from your program? • Knowledge (understanding) • Beliefs, attitudes and opinions (psychological states) • Behavior (What actions do you want them to take?)

5. Outputs are not the same as outcomes or impacts • Outputs are things you produce, including brochures, other printed materials, numbers of events, etc. • Grantors often need these numbers • Producing a brochure does NOT guarantee learning or outcomes!

6. Some questions you might ask… • What do people do differently as a result of the program? • Who benefits, and how? • Are the program’s accomplishments worth the resources invested? • What are the social, economic, environmental impacts?

7. Some questions you might ask… • What are the strengths and weaknesses of the program? • How well does the program respond to the need that caused you to design it? • How efficiently are resources being used? • How well does the program fit in the local setting?

8. What information do you need to answer the questions? Indicators: how will I know success when I see it? • Participation • Reactions (maintain interest, accept leadership responsibility) • Learning (sufficient knowledge and skills developed) • Actions (knowledge is applied; plans are made; collaboration occurs) • Impact (successful weed management planning and implementation occurs; decrease in infested acres, etc.)

9. Kinds of information • Numerical (statistical) • Narrative What will the recipients consider to be credible information? What best meets their need?

10. Sources of information • Surveys • Observations • Interviews • Tests • Group techniques • Case studies • Photographs • Document review • Testimonials • Etc…

11. The Truckee Meadows example Identify the audiences: • City and County government • Citizens and neighborhood advisory board members • The general public

12. City and County Government • Message: We have a problem. You have a legal responsibility. Failure to take action will increase future costs. • Desired action: Form a functional CWMA • Hook: Forming a CWMA will help get funding.

13. City and County Government Evaluation: PARTICIPATION • Does CWMA form? • Do city and county personnel participate fully? REACTION • Does participation continue over time? LEARNING • Is the ability to manage weeds improved? ACTIONS • Do collaborations develop among entities? • Is budget allocated to weed management? IMPACT • Are weeds managed?

14. What happened? • 36 city, county, agency, and citizen representatives attended the first meeting • Attendees agreed that a CWMA was needed • 12 weeds of concern were prioritized for group effort • Attendees decided to continue to meet

15. What happened (outcomes)? • Staff now attend annual trainings • An MOU and strategic plan were developed • Washoe County and City of Reno are collaborating • County has hired a Natural Resources Manager; position will coordinate weed management • Grants were obtained • Seasonal mapping staff were hired • Control budgets are still not adequate • Issue is easily de-prioritized

16. Indicators PARTICIPATION • Does CWMA form? (YES) • Do city and county personnel participate fully? (NO) REACTION • Does participation continue over time? (YES) LEARNING • Is the ability to manage weeds improved? (SOMEWHAT) ACTIONS • Do collaborations develop among entities? (YES) • Is budget allocated to weed management? (NOT ENOUGH) IMPACT • Are there fewer weeds? (COULD BE A LOT BETTER!)

17. CAB & NAB members • Message: Weeds are affecting your neighborhood, and you can help fight them • Action: Incorporate weed cleanups into neighborhood cleanup days • Hook: We’ll supply bags, information and help • Reaching the audience: Personal contact with each group via a “canned” presentation at a regular meeting

18. CAB & NAB members Evaluation tools: • Number of “champions” recruited (PARTICIPATION) • Number of projects incorporating weeds (ACTIONS) • Neighborhoods have fewer weed problems (IMPACT)

19. The campaign: • Presentations were made at 12 NABs and CABs in April and May reaching 240 people • Each group was given a binder of information • One “champion” was recruited; many asked for information and assistance • Number of events: one in 2004, five in 2005

20. General public • Message: Weeds are ruining your neighborhood • Action: Identify, pull, and dispose of yellow starthistle) • Hook: The “ouch!” factor – this affects YOU!

21. General public • Methods: PR campaign using movies, buses, hotline • Money: NDOA grant • Evaluation tool: Number of calls to hotline (LEARNING & ACTION) • Future evaluation tools: Hits on website, reports filed on website, random phone survey (need funding)

22. What happened? • A $10,000 grant was obtained for an education and outreach campaign • Campaign elements included: • presentations to advisory boards • promotion of a weed hotline • bus placards • slides in movie theaters

23. 2004 PR campaign • Washoe County donated a phone line for a hotline • Hotline magnets distributed at public events • Advertised via news releases and PSAs • Volunteers agreed to transcribe messages on a monthly basis

24. Bus Placards 9 buses for one month (potential viewership 150,000) Cost: $1566.34

25. Movie Slides 3 different slides shown on half of all Reno/Sparks screens in movie theaters (21 total) for 6 weeks

26. Movie Slides Potential viewership = 126,600 people

27. Movie Slides Number of times repeated = 8,820/slide Cost: $2370.50

28. Hotline Calls Logged by Month Number of Calls

29. Did it work? • Increase in hotline calls was small relative to amount of PR • Calls to hotline dropped off once PR stopped (and fall came) • Need better method to evaluate public awareness • Campaign elements were changed for 2005