Evaluation Designs and Methods Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D.

1. Evaluation Designs and Methods Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D. Professor and Youth Development Specialist Oregon State University 4-H Professional Development Webinar February 14, 2013

2. Webinar Agenda • Building on previous month’s topic of focusing and planning your evaluation we will: • Explore the concept of rigor in evaluation, and its particular place in Extension and 4-H evaluation • Learn the role that evaluation questions play in determining evaluation design and data collection methods • Explore common evaluation designs • Explore common and innovative evaluation methods • Learn of resources available to support evaluation efforts in 4-H and Extension

3. Elements of Rigor Braverman, M. T., & Arnold, M. E. (2008). An evaluator’s balancing act: Maintaining rigor while being responsive to multiple stakeholders. In M. T. Braverman, M. Engel, R. A. Rennekamp, & M. E. Arnold (Eds.) Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 71-86. Evaluation design Conceptualization of program constructs & outcomes Measurement strategies Timeframe of the evaluation study Program integrity Program participation and attrition Statistical analyses

4. Rigor and the 4-H Organization Who determines standards of rigor? How do decisions about evaluation methods get made? How, and to what extent, is the quality of a completed evaluation determined?

5. Post Only Design Evaluation Question Example: What skills do campers report developing at science camp? (X O) TIME O = “Observation” (data collection) X = “intervention” (program)

7. Post Only Control Group Design Evaluation Question Example: Do youth who attend 4-H summer science camp have better science skills than youth who do not attend? E (X O) C (X O) TIME O = “Observation” (data collection) X = “intervention” (program) E = Experimental group (program participants) C= Control group (non-participants)

8. What is the level of rigor? What cannot be said?

9. One Group Pre-Test/Post-Test Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning? (O X O) TIME

10. What is the level of rigor? • What cannot be said?

11. Retrospective Pre-Test Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning? ( O X O) TIME TIME

12. For each of the following items, please indicate how you felt before participating in this program, and how you feel now after participating in this program. 1 = Strongly disagree 2 = Disagree 3 = Agree 4 Strongly Agree

13. Control Group Pre-Test/Post Test Evaluation Question Example: Do youth in program develop higher levels of PYD than youth who do not participate? E(O X O) C (O --- O)

14. E LOW - Level of PYD - HIGH C TIME

15. Time Series Design with Control Group E LOW - Level of PYD - HIGH C O OOOX O OO

16. Choosing an Evaluation Data Collection Method • Some Common Methods • Archival data (records and documents) • Surveys (mailed, electronic, phone) • Interviews (phone, face to face, group) • Focus group interviews • Observation • Tests (scenarios or skill/knowledge tests)

17. Important Steps What is the key concept that must be measured in each evaluation question? Did youth participants in the YA4-H! Teens as Teachers program pilot increase their own consumption of fruits and vegetables? Who has knowledge of this potential change? Several sources may emerge: Youth, parents, leaders, teachers, friends, records, observations. What sources of data will be acceptable to stakeholders? What expertise and funding is available to make a particular method practical?

18. Existing Documents and Records • Have you ever considered meeting notes, minutes, videos, registrations, test scores, forms, records, reports as possible sources of data? • Considerably more cost effective than original data collection • Data are not affected by the act of collecting it • Programs collect lots of information that is never used, and too often we forget to look for existing data that can answer the question What existing information could answer the question of increased fruits and vegetable consumption?

19. Surveys (Mailed, Electronic, Phone) Designing Surveys: A Guide to Decisions and Procedures. Ronald Czaja and Johnny Blair (2005) Internet, Mail, and Mixed-Mode Surveys: Tailored Design Method (3rd Ed.) John Dillman, Jolene Smyth, & Leah Melani Christian (2008) How to Conduct Surveys: A Step by Step Guide Arlene Fink (2013) Survey Research Methods (4th Ed.) Floyd Fowler (2009) How could surveys help us assess increase in youth consumption of fruits and vegetables?

20. Interviews (Phone, Face to Face, Group) Designing and Conducting Your First Interview (text book) Bruce Friesen (2010) Qualitative Interviewing: The Art of Hearing Data (3rd Ed.) Hebert and Irene Rubin (2011) Focus Groups; A Practical Guide for Applied Research (2nd Ed.) Richard Krueger (1994) Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences (4th Ed.) Irving Seidman (2012) Why might interviews be a good method for collecting data about youth consumption of fruits and vegetables?

21. Direct Observation • Situations where you want direct information • May be more reliable than asking people if they are using new practices • When you are trying understand an ongoing behavior, process or unfolding situation or event • Observing camp counselors before, during and after a training program • When there is physical evidence, products or outcomes that can be readily seen • Inspecting project records, newsletters, signs • When written or other data collection procedures seem appropriate • Programs to vulnerable or underserved audiences, when language or literacy is a problem • Key Resource • Collecting Evaluation Data: Direction Observation • Ellen Taylor-Powell and Sara Steele- Booklet available on the State 4-H Website Could we use direct observations to measure youth consumption of fruits and vegetable?

22. Tests and Scenarios • Useful when assessing learning that requires specific knowledge that must be turned into action for the program to be considered successful. • Requires the ability to do a real pre –post test process • Can be very creative! Think skits and role plays • Camp counselor training • Youth leadership programs • Risk management training Can you think of how a scenario evaluation might be a useful method for assessing youth consumption of fruits and vegetables?

23. Summary Your Evaluation Question Determines Your Design Determines Your Methods Stay tuned on March 6th for our next webinar, which will focus more in depth on creating high quality questionnaires!

24. That’s all for now! Join in next month for: Creating High Quality Questionnaires Don’t forget to complete an evaluation of today’s webinar at: http://www.surveymonkey.com/s/4HEvaluationwebinar