Planning your evaluation

1. Planning your evaluation

2. QUIZ #1 Test your evaluation planning IQ!

3. 1. How confident are you in your abilities to design a useful evaluation? (choose one) • Very confident • Moderately confident • Slightly confident • Not very confident • Not at all confident

4. 2. What do you need to think about when you decide to conduct an evaluation? Write down your ideas

5. 5 Core steps in program evaluation

6. An evaluation plan is your roadmap How do you know which way to go if you don’t know where you are going?

7. STEP 1 Engage stakeholders Who should be involved in helping design and implement the evaluation?...or, do you do evaluation alone?

8. Think about these 4-H YD stakeholders who might be your evaluation stakeholders • Youth participants • Your program partners • Parents • Teachers • Volunteers • Your funder

9. How might they be involved? What roles might they play? • Help determine what the evaluation will focus on – what you really need to know • Be part of an evaluation team • Help write the questions • Help enter data and/or do the analyses • Raise funds • Write a press releases

10. Use a logic model to describe your program Situation: INPUTS OUTPUTS OUTCOMES Activities Participation Short Medium Long-term Program investments Assumptions External factors

11. What is your purpose for evaluating We are conducting an evaluation of ______ (name of program) because _____________ in order to ___________. Example: We are conducting an evaluation of the Money Quest Program becausewe want to know to what extent youth who participate learn and use recommended money management skills in order to report program outcomes to our funder.

12. Determine use and user: Who wants to know what? Has this ever happened to you? We are a group of five people who want to evaluate our after-school program. We know evaluation is important. We want to develop a questionnaire that we will distribute to the youth who participate in the program. So we sit down to write the questionnaire. But, it becomes apparent that we each have different ideas about what we want to find out. Mark, the school administrator, wants to know if the students who attend do better academically. Marg, Dick and I, as the program staff, want to find out what the kids like and dislike so we can plan better for the next program. Gloria, our funder, is interested in knowing how many youth attend regularly, which youth they are, and whether the program serves those in greatest need.

13. What questions will the evaluation answer? Questions about: Outcomes? Changes in knowledge, skills, attitudes, opinions, behaviors, policies… Satisfaction? What the youth like/dislike? Whether they will come again? Who participated – who didn’t? What happened for different participants? What the program does/did - How the program was implemented? What activities were undertaken; which activities link to different outcomes? What was invested (inputs): who collaborated? How? What resources were used? Your teaching? Ways to improve the program?

14. What is an indicator? An indicator is the specific information, or evidence, that represents the phenomenon you are asking about. Indicatorof fire = smoke Indicator of academic achievement = passing grades

15. Evaluation design Evaluation design is your overall approach to collecting data – when and how data will be collected. Typical designs in youth programs include: • Single point in time (e.g., survey, end-of-program questionnaire) • Case study design • Pre-post program • Comparison group designs • Mixed method designs

16. STEP 3 Collecting information How will you collect the information? From whom (sources of information) Methods of data collection

17. From whom??? (Source of evaluation information) Existing information People Pictorial records and observations

18. Methods of data collection Survey Case study Interview Observation Group assessment Expert or peer reviews Portfolio reviews Testimonials Tests Photographs, videotapes, slides Diaries, journals, logs Document review and analysis

19. Case study Questionnaire Observation Focus group Interview Need to quickly and/or easily get information from people in a non-threatening way Want to talk with and listen to people either face-to-face or by telephone. Want to gather accurate information by viewing what is occurring Want to explore a topic in depth through group discussion Want to deeply and fully understand someone’s experience in the program Match method to evaluation purpose

20. STEP 4Analyzing and interpreting What sense do you make of the information collected? analyzing quantitative data analyzing qualitative data

21. Step 5Using the information How will you report, use, and learn from the findings? To whom? What? How? When?

22. Management plan Human subjects protection Timeline Responsibilities Budget

23. Human Subjects Protection Not just clinical trials or studies with children Remember, youth = vulnerable population

24. Human Subjects Protection Follow your program area advice CONTACT: Ray Schultz, Secretary of the Faculty and Academic Staff 608-262-4387 http://www.uwex.edu/hsp/

25. Remember… • “The more attention you give to planning the evaluation, the more effective it will be.” • −The Program Manager’s Guide to Evaluation, 2003

26. QUIZ!Check up How confident do you feel NOW in your ability to design a useful evaluation? • Very confident • Moderately confident • Slightly confident • Not very confident • Not at all confident

27. What things do you need to think about when you decide to conduct an evaluation? Write down the 5 steps of program evaluation – compare to your ideas you wrote down at the beginning