Planning and Focusing an Evaluation Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D.

1. Planning and Focusing an Evaluation Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D. Associate Professor and Youth Development Specialist Oregon State University 4-H Professional Development Webinar January 10, 2013

2. Webinar Agenda • Building on previous month’s topic of logic modeling we will: • Review the basic components of a logic model, using the YA4-H! Teens as Teachers program as an example • Learn the importance of identifying the purpose and stakeholders of a program evaluation • Learn the Four Level Model of educational program evaluation: • Reaction • Learning • Transfer of learning • Results • Introduce three important parts of evaluation planning: • Evaluation questions • Evaluation design • Data collection methods • Learn about IRB regulations for evaluation projects

3. YA4-H! Teens as Teachers Program We received a $50000 competitive grant to Washington State University for 18 months of program funding • Program Outline • YA4-H Teens as Teachers • Develop the YA4-H! Teens as Teachers curriculum • Provide 3-days of training to teens and adults • Participants Learn About: • Developing Youth - Adult partnerships • Forming a county YA4-H Teens as Teachers team • Learning and implementing the Choose Health: Fun, Food, and Fitness (CHFFF) curriculum This is the educational part of the program- and where we will look for short-tem outcomes • 2. Nutrition Education for Youth Ages 9-12 • Teens teach the 6 CHFFF lessons to younger youth • Younger Youth Learn: • The amount of sugar in drinks • Eating more fruits and vegetables • How to read a nutrition label • Whole grains • High fat and high sugar foods • Eating breakfast Again, this is educational content. Do you think the younger youth are the only ones who will learn? NO! The teens will learn these things as well, simply through teaching it to others!

4. YA4-H! Teens as Teachers Program We have received a $50,000 competitive grant to Washington State University for 18 months of program funding Note that these outcomes are focused on what we want to see HAPPEN! ACTION!!!!!!! • Medium Term Outcomes (Behaviors) • Younger Youth • Replace sweetened drinks with low-fat milk and water • Eat more fruits and vegetables • Eat fewer high-fat and high-sugar foods and more nutrient-rich and high-fiber foods • Eat only as often and as much as needed to satisfy hunger • Play actively 60 minutes a day • Limit screen time to two hours or less a day • Teens as Teachers • All of the above outcomes • AND • Actively promote healthy behaviors • Develop and practice teaching and leadership skills • Increase PYD • Act as a role model for younger youth

5. YA4-H! Teens as Teachers Program We have received a $50,000 competitive grant to Washington State University for 18 months of program funding • 4. Long Term Outcomes (Results) • Youth are less obese and more active • Healthier food choices are available for youth • There is a reduction in prevalence of sugar drinks and low-nutrition food • Communities are united to provide healthy spaces for youth • There is a reduction in chronic diseases related to obesity over time

6. YA4-H! Teens as Teachers Program Evaluation Inputs Outputs Outcomes What is done. Who is reached.. Teens as Teachers Curriculum Staff Youth-Adult Teams are trained Money Partners Provide training to youth-adult teams Teams return to the community and implement the CHFFF program

7. Program theory is important, but it also has to make practical sense in order for evaluations to be meaningful! • That is why logic models are so useful- they can identify critical links in the program’s theory of change. • If the links aren’t “logical” than the program may have little practical value

8. Why Do We Evaluate? • Help others understand the program (stakeholders) • Understand the need for a program • Improve the program • Improve teaching • Understand the program’s impact • Determine if the program is progressing as planned • Determine if the program is worth the cost • Meet grant reporting criteria • Meet administrative requirements

9. Why Do We Evaluate? Poll: What is the number ONE reason you evaluate your programs? Share Your Experience: What motivates you to do evaluation, and how do you feel about that motivator?

10. Ahhhh…. Stakeholders! Stakeholders will have different needs for evaluative evidence!

11. Who Cares? • People affected by the program either directly or indirectly (youth, parents, volunteers) • County boards, elected officials • Community leaders • Colleagues, volunteers, supporters, collaborators • Extension administrators • Grantors • Tenure committees • Other stakeholders It is essential to think about who cares about the evaluation results and determine how the results will be used early in the evaluation planning process.

12. Poll: Who are your most important stakeholders? (Check all that apply) Share Your Experience: Share an experience of needing difference evaluation evidence for difference stakeholders

13. The Four Level Model of Educational Program Evaluation (Kirkpatrick & Kirkpatrick, 2006) • Participant Reactionto the program- you want a favorable response to the program; people are more motivated to participate and learn if the program is a positive experience for them. • Participant Learning– the extent to which participants change attitudes, improve knowledge, and increase skills. • Participant Behavior Change (Transfer of Learning) • 4 conditions for transfer of learning to take place: • The person must have a desire to change • The person must know what to do and how to do it • The person must be in a climate that supports the change • The person must receive some type of reward for changing (intrinsic and/or extrinsic) • Participant Results –One way to guide this part of the evaluation is to ask • What is the main reason for this program? Then we ask also have to determine the links between the learning and behavior changes and these results • (yep, this is a cloaked logic model!)

14. An introduction to three important aspects of evaluation planning: • Evaluation questions • Evaluation design • Data collection methods

15. We begin with the evaluation questions, because the questions determine the design and data collection methods

16. Focusing Evaluation Questions Questions of participant reaction: Who actually participated in the program? Are there barriers to participation? Were participants satisfied with the program? Was the content of the program relevant to the participants? Questions of participant learning: Did participants learn the intended program content? Did the program change participant attitudes? Did participants leave the program with new or enhanced skills? Questions about behavior change Do participants plan to use what was learned? (potential change) Do participants have a supportive climate for implementing what was learned? Did participants actually implement new behaviors? Why or why not? Questions about Results What difference has this program made for the audience, community, and other stakeholders? (the SO WHAT question)

17. Choosing an Evaluation Design Critical to Understand! Your evaluation question determines your evaluation design and data collection methods!!! (too often I hear: I need a survey to evaluate my program )

18. Evaluation Designs 1) Post only design (X O) 2) Post only control group design E (X O) C (X O) 3) Retrospective pretest ( X O) O = “Observation” (data collection) X = “intervention” (program) E = Experimental group (program participants) C= Control group (non-participants)

19. Evaluation Designs 4) One group pretest- posttest design (O X O) 5) Control group pretest- posttest design E (O X O) C (O X O) 6) Time series design (with control group) E (O O O X O O O) C (O O O X O O O) O = “Observation” (data collection X = “intervention” (program) E= Experimental group (program participants) C= Control group (non-participants)

20. Choosing an Evaluation Data Collection Method • Some Common Methods • Mailed survey • On-line survey • Individual face to face interviews • Focus group interviews • Phone survey or interview • Observation • Archival data (records and documents) • Test (e.g. scenarios or skill/knowledge tests) Stay tuned on February 14th for our next webinar, which will focus more in depth on design and methods!

21. Poll: What data collections methods have you used? (Check all that apply) Share Your Experience: Share an experience of using a data collection method that challenged you or gave you great information

22. Oregon State University Institutional Review Board (IRB) AKA… Do I have to do that? • Research is determined by three qualities: • Systematic inquiry into a phenomena • That is designed to develop or contribute • Generalizable knowledge • Human subjects Are: • Living individuals about whom an investigator conducting research obtains: • Data through an intervention or interaction with the individual, or • Identifiable private information

23. Institutional Review Board (IRB)Okay, I have to do it… now what? • Go to the IRB Website at: http://oregonstate.edu/research/irb/ • Complete the “Does Your Study Require IRB Review? Form • Complete online ethics training modules • Complete the steps listing under “Preparing and Initial Submission” at http://oregonstate.edu/research/irb/preparing-initial-submission

24. That’s all for now! Join in next month for: Evaluation Designs and Methods Don’t forget to complete an evaluation of today’s webinar at: http://www.surveymonkey.com/s/4HEvaluationwebinar