Evaluating Environmental Strategies

1. Evaluating Environmental Strategies Presented by Southwest Resource Team (SWRT) August 18 – 19, 2011

2. Introductions / Logistics • Who you are? • Why you are here? • What are your expectations?

3. Workshop Objectives • Define evaluation • Identify key components of an evaluation process; • Describe Outputs and Outcomes; • Identify elements of an Outcome Performance Measure; • Identify the differences between Process and Outcome Performance Measures.

4. Discussion • What do you think when you hear the word “evaluation”? • Do you have concerns about conducting an evaluation? • What are the advantages of evaluation?

5. What is Program Evaluation? • Program is any organized action/activity implemented to achieve some result. • Evaluation is the “systematic investigation of the merit, worth, or significant of any object.” (NSF 93-125)

6. What is Program Evaluation? • A systematic collection of information to “assess how well a program is working, typically focused on achievement of program objectives.” (GAO)

7. Purposes of Evaluation • To make judgments about the program, improve program effectiveness, and/or inform decisions about future program development (Patton, 1997).

8. Types of Evaluation • Needs Evaluation: Assessment and identification of unmet needs. • Process Evaluation: Is the program being implemented as planned? • Outcome Evaluation: Is the program having the desired effects on the Target Population? • Impact Evaluation: What are the overall intended or unintended direct effects (i.e., causality) of the program?

9. Outcome Evaluation • A systematic method of collecting and analyzing data to assess the immediate and direct effects of program services on participants.

10. Types of Evaluation Data • Quantitative data are numerical. • Qualitative data are nonnumeric.

11. Activity Read the What is Evaluation questions and be prepared to share your answer with the large group. Activity

12. Why Good Evaluations are Necessary “I am an evaluator and I am here to help.”

13. “Good” Evaluations • Tier 1 (“Bunny”): Process-Based Evaluation • Tier 2 (“Beginners”): Goals-Based Evaluation • Tier 3 (“Intermediate”): Outcomes-Based Evaluation • Tier 4 (“Expert”): Impact-Based Evaluation

14. Why Good Evaluations Matter • To determine achievement toward goals, objectives, and outcomes • To improve program implementation and delivery (e.g., fidelity) • To enhance accountability to funders, community, and other stakeholders • To increase support for initiatives through community awareness

15. Why Good Evaluations Matter • To contribute to scientific base of knowledge • To inform policy decision • To permit comparisons among groups (e.g., impact of program on population with high-risk factors) • To justify the need for further funding and support • To justify expansion of the program • To find opportunities for improvement

16. Activity • Read Why Good Evaluations are Necessary, reflect, and be prepared share your answers with your partner and the large group. Activity

17. Components of the Evaluation Process

18. Steps of the Strategic Prevention Framework • Assessment of community. • Capacity of your agency/coalition. • Planning (Needs/Gaps) • Risk factors • Protective factors • Prioritizing Risk and Protective Factors. • Implementation • Involves the use of a Logic Model. • Evaluation • Monitor program implementation • Identify areas that need improvement • Highlight areas for sustainability.

19. Major Componentsof the Evaluation Process • Goals: A broad plan or idea. • Objectives: Describes what a program aims to do, achieve, or accomplish. • Outputs: Things produced (usually a physical item) by the organization with an intended desire to benefit the Target Population.

20. Major Componentsof the Evaluation Process • Performance Measures: An indicator used to determine the extent to which the project objectives are being met: • Process : Address what the grantee/program will do to effect change in Target Population; • Outcomes: Assess the immediate or direct effects the program had on participants • Impact: Level of change over time in performance or achievement that can be directly attributed to your program.

21. Goal and Objectives Goals Project Objectives Narrow plan Specific action Must be measureable Short to medium term Based on fact • Broad plan • Generic action • Generally not measureable • Longer term • Based on ideas

22. Objectives • Relevance: The Project Objective should be congruent with the overall project goals. • Applicability: The Project Objective should relate to the specific activities that are being conducted through the project. • Measurability: The Project Objective should include constructs that are conceptually measurable through data collection strategies that are feasible to implement.

23. Activity • Read Goals or Objectives and identify if it is a goal or objective Activity

24. Outputs versus Process Measures Process Measure Outputs A part of a Process Measure Usually something than can be touched End result of a task or series of tasks Does not directly affect the Target Population’s attitudes, knowledge, or behavior • Consists of one or more outputs • Usually something that can not be touched • Required to make a change in the Target Population’s attitude, knowledge, or behavior

25. Activity Read Outcomes or Outputs and identify if it is an outcome or output? Activity

26. Process Measures and Outcome Measures Process Measures Outcome Measures Assess the extent to which the objective is met Always involve some level of change in performance or achievement (e.g., attitude, knowledge, behavior) Have a baseline or benchmark • Address how the objective will be met • Address program implementation activities that involve the Target Population (i.e., constituencies) • Each Process Measure should be accompanied by at least one Outcome Measure

27. Clarifying Measures • Process Measures are not activities that will be done to administer the program • Process Measures are steps or tasks required to achieve your stated outcome. • Outcome Measures are not goals. • Outcome Measures provide clarity on expected change.

28. Outcomes • Outcomes need to be specific about what will change • Outcomes need to detail how much change is expected • Outcomes need to detail who will achieve the change • Outcomes need to detail when the change will occur • Outcomes need to detail how you will measure the change

29. Activity Read Process or Outcome Measure and identify if it is a process or outcome measure? Activity

30. Question If your goal is to “become healthier” then did you succeed when you… • Lost 10 pounds? • Reduced your waist size? • Could fit into your old clothes? • Reduced your fat intake?

31. Question If your goal is to reduce student substance abuse then did you succeed… • When students who were using drugs stop using them? • When one less student is using drugs than before? • When the percentage of students has been reduced by 1%?

32. Parts of an Outcome Measure • WHAT is going to change as a result of the program? • HOW MUCH change will occur? • WHO will achieve the change (i.e., Target Population)? • WHEN will the change occur? • MEASUREDBY what?

33. Outcome Measure Example • To reduce, compared to 2006-2007 baseline data, the percentage of middle school students (6th, 7th , and 8th) [WHO] who report past-30 day use of alcohol [WHAT] by 5%, 8%, 11%, and 14%, respectively, [HOW MUCH] in Years 1, 2, 3, and 4 [WHEN] as measured by self-reported student alcohol use on the PRIDE Survey [MEASURED BY].

34. Activity Read Outcome Measure or Process Measure and identify if it is an outcome or process measure. Activity

35. Measuring Change “You really did receive a raise.”

36. Measuring Change • Raw Number : A specific number • Ratio: One number is a multiple of another number, or of a part or parts of the other number • Rate : A calculation used to help accurately compare activities and services for groups with different population sizes. • Rate of Change: A calculation used to determine the change from one specific time to another specific time

37. Measuring Change – Raw Number • In 2000 there were 127 referrals for ‘minor in possession’ cases. • In 2009 there were 220 referrals for ‘minor in possession’ cases. • Is the raw number going up or down?

38. Measuring Change – Ratio • In 2000 there were 127 referrals for ‘minor in possession’ cases in an adolescent population of 12,000. • In 2009 there were 220 referrals for ‘minor in possession’ cases in an adolescent population of 21,000. • Is the ratio going up or down?

39. Measuring Change - Rate • In 2000 there were 127 referrals for ‘minor in possession’ cases in an adolescent population of 12,000 • In 2009 there were 220 referrals for ‘minor in possession’ cases in an adolescent population of 21,000 • Is the referral rate per adolescent going up or down?

40. Measuring Change – Rate of Change • In 2000 there were 127 referrals for ‘minor in possession’ cases in an adolescent population of 12,000 • In 2009 there were 220 referrals for ‘minor in possession’ cases in an adolescent population of 21,000 • Did the rate of referral change per student go up or down in 2009 versus 2000?

41. Measuring Change Exercise

42. Results

43. Activity Read the Measuring Change (True or False) statements and identify if it is true or false. Activity

44. Logic Models • A depiction of a program showing what the program will do and what it is to accomplish. • A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes. • The core of program planning and evaluation.

45. Why Logic Models

46. Example H E A D A C H E Getpills Takepills Feel better Situation INPUTS OUTPUTS OUTCOMES

47. Example H U N G R Y Getfood Eat food Feel better

48. Example Family Members Miles driven to park Family members learn about each other; family bonds; family has a good time Budget Set up camp RV Cook, play, talk, laugh, hike Camping Equipment OUTPUTS OUTCOMES INPUTS

49. Logical Chain of Connections INPUTS OUTPUTS OUTCOMES Activities Participation Short Medium Long Program investments What results What we invest What we do Who we reach

50. Tutoring Program Example IF THEN IF THEN IF THEN Students who were struggling academic-ally will move to the next grade level on time We invest time and money We can provide tutoring 3 hrs/week for one school year to 50 students struggling academic-ally Students struggling academic-ally get better grades Students struggling academic-ally will improve their academic skills Students struggling academic-ally can be tutored