Planning for Impact: Using Program Models to Guide Evaluation

1. Planning for Impact: Using Program Models to Guide Evaluation Arizona Cooperative Extension Conference August 6, 2019 Community Research, Evaluation and Development (CRED)

2. CRED (Community Research, Evaluation & Development) Michele Walsh Madeleine deBlois VioletaDominguez Rachel Leih DeeDee Avery John Daws Kara Haberstock Tanoue This Photo by Unknown Author is licensed under CC BY-NC-ND

3. The systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding. Michael Patton, 2008, p. 39 Building Capacity for Program Evaluation

5. What are the benefits that your program provides? Activity

6. Who does your program matter to?

8. Stakeholders ask…. So What?

9. Private (i.e., individual) value Public (i.e., societal) value Value to those not in programs, too Change in the economic, environmental & social conditions of communities Changing Expectations in Cooperative Extension • Learning outcomes of those in programs • Individual behavior change Aka, Population accountability, impacts

10. Decrease public costs related to risky behavior • Decrease public costs related to financial problems • Decrease public costs related to dependency on local, state, and federal services • Decrease public response costs to emergencies and disasters • Increase academic success for a highly qualified and productive workforce • Increase health and ability of public safety workers • Decrease health care costs • Decrease in education costs as fewer children are placed in special education Popular Public Values Adapted from Franz, 2016

11. Integrated Pest Management • Private – save $ by reducing use of pesticides • Public – improve water quality • Nutrition education • Private – increase intake of fruits and vegetables • Public – decrease health care costs • 4H STEM education • Private – explore interests in range of areas from animal sciences to aerospace • Public – increase science workforce Extension Often Provides Both Types of Value

12. Let’s revisit the benefits you identified • Classify: which represent private (individual) value and which represent public value? • What are some additional public values you think your program can bring? • What long-range societal transformation is your program striving towards? Remember, we are “improving lives, families, communities, the environment and economies in Arizona and beyond” Activity

13. Program Outcome Levels Short Term Medium Term Long Term Learning Action “Impact” awareness created behavior changed health impact knowledge gained practice adopted economic impact attitudes changed decisions made environmental impact skills developed policies changed societal impact or adopted aspirations sparked social action taken People: What they learn and do Conditions

14. Process Extension educators typically use four approaches to engaging with communities: High Content Low High For more: Franz, http://aea365.org/blog/nancy-franz-on-approaches-to-community-education-and-evaluation/ Low

15. Process Extension educators typically use four approaches to engaging with communities: High (Process and) Summative Evaluation (Impact) Learning, Behavior and Condition change Process Evaluation Content Low High Personal Evaluation (i.e., satisfaction) Organizational Process Evaluation Process evaluation Knowledge evaluation Low Adapted from Franz, 2003, 2014

16. We rarely have conclusive proof of what works, or what works best. Good data are hard to obtain and expensive. The data we have can be interpreted in multiple ways. Even reasonably good evidence can quickly become outdated. Hard Truths About Measuring Outcomes and Impacts

17. Planning for Impacts

18. We Need to Be Able to Articulate the Program Throughout the Chain to Get to Impacts Where we look for success How we achieve outcomes

19. A logic model is not an evaluation plan—it will be used for evaluation planning • The model should not just include things you can see or measure. It is meant to convey information and a visionof how the program works.  • It forms the basis of evaluation plan, where you will identify indicators of whether the outcome has been achieved. • In the beginning, be creative!

20. Healthy Eating And Physical Activity/Obesity Prevention Logic Model Example from North Central Cooperative Extension Association http://www.nccea.org/multistate-activities/program-area-logic-models/

21. Theory of Change, or Pathway, Model Source: Systems Evaluation Protocol, Cornell Office of Research on Evaluation Links activities to one or more outcomes

22. Are there any activities that are not connected to any outcomes? • Are there any outcomes that are not connected to any activities? • If yes, why do these gaps exist? • Was something simply left out of the model? • Or, is there a program activity that does not really address the program goals? • Or, does the program not include an activity that would result in that outcome? Questions to Ask About Your Completed Pathway Model

23. Outcome: Underlying change that takes place • “Families increase the variety of vegetables in their diet.” • Indicators: Can be a number of different things, and is part of the measurement strategy. • Weekly average number of different vegetables consumed at family mealtimes. • Number of different vegetables observed in the family’s grocery store purchases. Moving to Measurement

24. Table 1: 5-Tier Model Participant Specific Data • Measures of attitude • Measures of behavior • Measures of goal attainment Participant & Community Social Indicators • Health Status • Economic Well-being • Social Well-being Comparison Groups Longitudinal Data Types of Data Needed for Outcome and Impact Evaluation Captures what changes have happened for participants & links those changes to the program (Tier 4: Achieving Outcomes) Provides evidence of effectiveness and advances knowledge in the field (Tier 5: Impact) Adapted fromJacobs, F., Kaouscik, J. Williams, P., & Kates, E. (2000). Making it Count: Evaluating Family Preservation Services. Tufts University Press. Table 1: The Five-Tiered Approach: From Jacobs, F. (1988). The five-tiered approach to evaluation: Context and implementation. In H. Weiss & F. Jacobs, (Eds.), Evaluating family programs. Hawthorne, NY: Aldine de Gruyter.

25. Children participating in Family Resource Center programs enter kindergarten ready to learn, as evidenced by improved kindergarten literacy scores (school districts) • Breastfeedingrates increase over time when women are given appropriate information and access to lactation rooms, as measured by WIC reports (Arizona Department of Health Services; Inter-tribal by Council of Arizona, and their partnering tribes) Impact Studies Often Require Systems & DataPartnerships

26. Build on Established Research From Mary E. Arnold’s “The Thriving Model”. https://oregonstate.app.box.com/s/zic7atlnomdy033vemjrzb4axsgso2ra From Mary E. Arnold’s “The Thriving Model”. https://oregonstate.app.box.com/s/zic7atlnomdy033vemjrzb4axsgso2ra From My E. Arnold’s “The Thriving Model”. https://oregonstate.app.box.com/s/zic7atlnomdy033vemjrzb4axsgso2ra From Mary E. Arnold’s “The Thriving Model”. https://oregonstate.app.box.com/s/zic7atlnomdy033vemjrzb4axsgso2ra

27. Link our program data to outcomes our stakeholders care about. Infographics: Simple, visual documents connecting common measures items to outcomes from the research literature in an easy-to-read format

28. Thank You! CRED Community Research, Evaluation & Development Team Michele Walsh mwalsh@email.Arizona.edu 520-621-8739

29. Handouts provided

30. What are the activities that your program is undertaking that you expect to be an “active ingredient” in change? • Only include activities that reach people who participate or who are the target (not administrative, marketing or administrative activities) • Hint!: Try to define them so that someone unfamiliar with the program can understand the title • Identify the short-term outcomes of your program • “short-term” are the things that are logically connected to activities (changes in awareness, knowledge, attitudes, skills, opinions, aspirations, and motivations) • Hints! • Don’t limit yourself to things that can be “measured” at this point • List out anything that you think arises as a direct effect of being in the program (don’t sweat the terminology too much—just brainstorm) • Some of these may be the individual values/benefits we talked about earlier • Identify the medium-term outcomes of your programs • Hints! • Don’t limit yourself to things that can be “measured” at this point. • “medium term” are follow-on effects such as new behaviors or changes in behavior, practice, decision-making, policies, social action awareness, knowledge attitudes, skills • Some of these may be the individual values/benefits we talked about earlier • What are plausible impacts that could arise from the intermediate outcomes? • Hint! • Don’t limit yourself to things that can be “measured” at this point. • Many of these will be your public value statements from earlier • The program doesn’t exist in a vacuum, so… • Identify your assumptions • Beliefs, and expectations, about how and why a program is expected to succeed • Identify the program environment/context • External, influential factors that could affect outcomes (for example, politics, family circumstances, values, cultural factors) • You may have little or no control over them Tips for developing your logic model