Impact Statements 2005

1. Impact Statements 2005 Joseph L. Donaldson Extension Evaluation Specialist Linda K. Bower David Perrin Eastern Region Program Leaders

2. What is a program? • An Extension program is a planned educational response to an identified need. • A program is a planned series of interrelated educational strategies. • These strategies are designed to help people make social, economic or environmental improvements.

3. What is evaluation? • Think of evaluation as asking questions to learn about our program so we can continually improve and communicate program outcomes to our clients and stakeholders.

4. What is impact? • Improvements in learning, actions or conditions • Improvements in social, economic or environmental conditions • Outcomes of their participation – So What???

6. What happens to my impact statement? • State funding documents • Federal impact statement database • Federal Annual Accomplishment Report • Performance Appraisal • County accountability

7. What problems with impact statements are we trying to solve? • Describing activities, outputs, and inputs that are not impact • Describing everything that was achieved – with little if any focus on real need • Describing only impacts that will occur in the future • Undocumented impacts

8. Format for 2005 Impact Statement • CSREES Goal Number: • Title: • Issue: (relevance) • What has been done: (response) • Impact: (results) • Funding: • Contact:

9. “Issue” Describes the NeedsTwo Kinds • Needs that are known: focus groups, surveys, observation, nominal group technique and interviews • Needs that are unknown: data indicators, investigative reporting and environmental scanning • Comes from your annual plans

10. “What has been done” Describes Extension’s Response • Inputs: agent time, contact hours, volunteer time • Outputs: educational programs presented, farm visits, home visits, training events, news articles, newsletters, activities, materials produced, and participation levels

11. “Impact” Describes the Outcomes • Learned something – knowledge, attitudes, skills, aspirations • Motivated to take action – recycle, vote, adopt practices, eat right • Living conditions improved - Environment, Quality of Life, Economic

12. Bennett/Rockwell TOP Model

13. What is impact? • Environment: reduced pesticide usage, reduced contaminants in air, water, soil, increased recycling, increased waste management • Quality of Life: decrease in illness, improved eating habits, increase in academic scores • Economic: dollars saved, increased profitability, cost-benefit analyses, increased production, decrease of loss, jobs created, reduced debt, increased savings

14. Issue StatementShow the Need • The Example County Chamber of Commerce identified poor workforce skills among the county’s greatest concerns. The County 4-H Advisory Council expressed concern that middle school youth need more opportunity to gain and use the communication skills essential to workforce success.

15. Priorities to OutcomesWorkforce Prep Outcome In a pre and post-test survey, 6-8th grade participants will self-report the following on the TN 4-H LSES Public Speaking survey: • The proportion who now have the confidence to speak in front of others will increase from 40% to 80%.

16. Outcomes • The proportion who have acquired the skills to research a topic for a speech will increase from 30% to 80%. • The proportion who can deal with nervousness when speaking will increase from 10% to 40%.

17. Outcomes, Post-Test Only Post-Test only TN 4-H LSES survey will show that among 6-8th graders: • 80% have gained the confidence to speak in front of others. • 80% can now research a topic for a speech. • 40% can now deal with nervousness when speaking in front of a group.

18. Program Planning and Evaluation Model • Help us to work in logical ways • “Logic Models” • Helps us plan inputs, outputs and outcomes

19. Program Title:________________________________________________________ Issue:

20. Example County Workforce Prep Adapted from: UW-Extension. Program development in UW-Extension: Getting started in Extension programming. University of Wisconsin Distance Education/Digital Media Unit.

21. Example County Financial Well-Being

22. Example County Battling the Musk Thistle

23. Example County Parenting Education Adapted from: UW-Extension. Program development in UW-Extension: Getting started in Extension programming. University of Wisconsin Distance Education/Digital Media Unit.

24. Is This Logical? • Issue: Youth need to develop decision-making skills for better nutrition. • What has been done: Monthly programs were presented on the topic Decisions for Health, and poster art and bread-baking were popular activities. • Impacts: 57 clubs were organized. The teachers report that “this was the best year ever.” • Is this logical??

25. Is This Logical? • Issue: Youth need to develop decision-making skills for better nutrition. • What has been done: Monthly programs were presented on the topic Decisions for Health, and poster art and bread-baking were popular activities. • Impacts: 57 clubs were organized. The teachers report that “this was the best year ever.” • No, it’s not logical

26. Is This Logical? • Issue: A recent survey showed that parents of 4-H horse project members feel the most important outcome of the program should be the ability to set and achieve goals. • What has been done: Monthly meetings emphasized horse nutrition. • Impacts: 10 youth exhibited in the Regional 4-H Horse Show. • Is this logical??

27. Is This Logical? • Issue: A recent survey showed that parents of 4-H horse project members feel the most important outcome of the program should be the ability to set and achieve goals. • What has been done: Monthly meetings emphasized horse nutrition. • Impacts: 10 youth exhibited in the Regional 4-H Horse Show. • No, it’s not logical

28. Is This Logical? • Issue: Youth are not engaged in their community… • What has been done: Citizenship in Action involved 450 youth. Youth have planned and conducted 15 service projects. • Impact: Scores on the citizenship life skill evaluation showed an increase of 50%..... • Is this logical?

29. Is This Logical? • Issue: Youth are not engaged in their community… • What has been done: Citizenship in Action involved 450 youth. Youth have planned and conducted 15 service projects. • Impact: Scores on the citizenship life skill evaluation showed an increase of 50%..... • Yes, it is logical

30. Where do we show impact? • Planned (60%) • Most contact hours since that’s where we have the most impact • Measurable outcomes • Important to your stakeholders • Base programs

31. Evaluation Plans • Watch them: Observation • Examine what they did or said: Records • Ask them: Interviews or questionnaires • Document these for evaluation

32. Helpful IdeasEvaluation Plan • 3 month follow-up survey • Scale: Before = did before program Started = started since program Plan = plan within three months No Plan = no plans NA = does not apply to me

33. Evaluation PlanFoods Project Follow-up Below are some food practices. For each practice, please circle the answer to show what is true for you.

34. Observation • What are you aiming for? • Have criteria written down • Checklist is an observation • Scorecard is an observation

35. EXAMPLE Checklist for Prevention of Cross Contamination Between Meats and Ready to Eat Foods  Uses separately labeled or color-coded cutting boards  Changes disposable gloves between tasks  Washes hands before and after, when working with raw foods  Cleans and sanitizes food contact surfaces that touch contaminated food before they come in contact with cooked or ready to eat food  Stores raw or contaminated food below cooked or ready to eat foods to prevent them from touching or dripping fluids

36. Examine Records • Benchmarks • But, records are often not available • So, make your own • Record/diary/checklist

37. Focus Group Interviews • 5-7 people • Ask them what happened as a result of the program • Ask follow-up questions • Record the answers • Data can be hard to analyze, but it’s rich

38. Pre-Test and Post-Test • No need to say “pre-test” or “post-test” to your participants • “It’s time to begin. First, you will complete a very short exercise that will help you prepare for our program.” • “The last part of our workshop will allow you to review some things you may or may not have learned the past few months.”

39. Post-Then-Pre Common Example Please think about your 4-H livestock project now and before this program. Please check which, if any, practices you do now (left) and then check which, if any, practices you were doing before this program (right). NOW BEFORE THIS PROGRAM ____ 1. I keep records of weaning weights. ____ ____ 2. I have calculated costs for different feeds. ____ ____ 3. I have made nutrition decisions based on rations. ____

40. How important is livestock record-keeping to you? Some people feel that livestock record-keeping is no big deal. Others think that records provide a way to make better decisions and maybe even earn greater profits. What is true for you? Please place a “B” on one step to show how important you thought records were before our lessons. Please place an “N” on the step that shows how important you think records are now. Very Important Not Important

41. Tally Results • Four step increase = 50 people • Three step increase = 18 people • Two step increase = 10 people • One step increase = 9 people • Same or decrease = 1 person

42. Changing Attitudes for Better Decision-Making 88 4-H members in a recent UT Extension livestock record-keeping program completed an attitude scale that described how important they felt record-keeping was to making better decisions and earning higher profits. The scale measured their feeling both before and after the program. The attitude scale ranged from no change to a four level change, and 97% of participants now have a better attitude toward record-keeping. • The majority of participants (50 people; 56%) increased four levels on the record-keeping attitude scale. • 20% increased three levels. • 11% increased two levels. • 10% increased one level.

43. LSES Ideas • Collapse data • Focus on one or two outcomes • Focus on the survey means

44. Analyzing Data • Helpful to collapse data

45. Analyzing Data • Achieving Goals means: 2.5 pre-test / 3.5 post-test • 3.5 minus 2.5 = 1 • 5 divided into 1 = 20% • 20% gain in the abilities associated with achieving goals

46. Share positive evaluation results: • County Ag Committee • Advisory Committee • Target audience • Set goals – move to the next level • Elected officials • Apply for awards/dossier/share with your peers

47. What do I do with less than positive evaluation results? • Use in needs assessment • Improve or alter programs • Set goals • Set priorities • Eliminate ineffective programs • Share with Advisory Committee • Share with your peers

48. Impact Statement Summary • Relevance – response - results • Issue - what has been done - impact • Impact: learning, actions or conditions • Stick with a line of logic

49. Actual Impact Statements • Evaluation data gathered from 76 of the participants revealed: • 68% learned how to read pay stubs. • 66% learned how checking accounts work. • 72% reported feeling better about their ability to handle finances as an adult. • 72% learned how to write a check correctly. 67% learned how to stub a check or keep a check register. • 70% plan to check their pay stub in the future to be sure the deductions are made correctly. • 66% plan to keep a more careful record of their check writing.

50. Actual Impact Statements • Approximately 24 beef producers worked cooperatively to market approximately 1450 feeder calves through truckload video auctions during the program year. This increased producers income over average market price by $101,500.