Impact Evaluation Strategies for Public Health Programs: Enhancing Outcomes

1. State of South CarolinaDepartment of Alcohol and Other Drug Abuse ServicesPartnerships for Success - ECHO ProjectOutcome and Impact Evaluation Quarterly Meeting June 4, 2019 Al Stein-Seroussi

2. Purpose of the Session • To review concepts related to outcome and impact evaluations • To review MIS spreadsheets in related to outcome and impact evaluations • (Not discussing outputs)

3. SC ECHO Sites

4. ECHO Coalitions, Counties, and Populations 86% 14%

5. Focus on ECHO: Limitations of Site-Specific Data • Power in numbers…it is harder to observe and attribute change when looking at data from any individual site, especially for relatively low-incident events. • No identified comparison data. • Some of the data are proxies of the construct—may not be the most ideal indicator (e.g., social access).

6. Limitations of Site Specific Data At a site-specific level, you are monitoring outcomes, not attributing changes in outcomes to your interventions. Even in the aggregate, we will have challenges attributing change to the collective interventions.

7. Outcome Evaluation/Monitoring Definition • Outcome/effectiveness evaluation measures program effects in the target population by assessing the program outcomes or outcome objectives that the program is to achieve (CDC) • Outcome monitoring is the routine measurement and reporting of indicators of the results of a program’s efforts in the social domain it is accountable for approving…A prerequisite for outcome monitoring is the identification of the outcomes that can reasonably be expected to produce. (Rossi, Freeman, & Lipsey)

8. Impact Evaluation Time-Based Definition • Impact evaluation assesses program effectiveness in achieving its ultimate goals (generally beyond the study population). (CDC) • Impact evaluation has the same goal as outcome evaluation, although it is conducted later than outcome evaluation—typically years after the end of a program—to determine the lasting effect on the target population. (K4Health) • Impact: The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. (INTRAC)

9. Impact Evaluation Attribution-Based Definition • Impact is the degree to which the outcomes observed are attributable to the program activities. (Stanford Social Innovation Review) • Impact evaluation, also referred to as outcome evaluation…(Rossi et al.) Key Point: Outcome and impact language can be confusing and imprecise. That’s why it’s important to develop a clear logic model.

10. Logic Model • Establishes connections between outcomes and your strategies • Tells the story of how your strategies may be influencing outcomes

11. Statewide Logic Model

12. Statewide Logic Model Short term outcome

13. Statewide Logic Model

14. Statewide Logic Model Long term outcome

15. Statewide Logic Model

16. Statewide Logic Model Impacts

17. Statewide Logic Model

18. Rx: Retail Access Count of drugs (pills) dispensed Count of drugs (prescriptions) dispensed Count of recipients of Rx drugs

19. Rx Retail Access: Quantity of Pills Dispensed

20. Rx Social Access Number of prescriptions received from others Amount of unused prescriptions in circulation Use of safe disposal/safe storage

21. Rx Social Access: Number of People that Received Deterra

22. Rx Perceptions of Risk • Risk of harm from using Rx drugs not prescribed to them and improper use of drugs prescribed to them • Data Sources: • YRBS (annual, but statewide) • CTC (biennial) • PIRE community survey (pre-post)

23. Rx Consumption • Use of Rx drugs without a Rx • Improper use of prescribed drugs • Data Sources: • YRBS (annual, but statewide) • CTC (biennial) • PIRE community survey (pre-post)

24. Rx Consequences Drug-related overdose deaths Drug-related hospital ED visits

25. Rx Consequences:Drug Overdose Deaths

26. Rx Drug Story Cannot make causal attributions but can link strategies with outcomes you are monitoring. Story should be accurately reflected and should provide insights into your strategies. • Community is implementing: • Prescriber education (provide numbers) • Consumer education about, and tools for, safe storage and disposal (provide numbers) • Media campaigns about risks associated with nonmedical use of Rx drugs (provide #s) • Community is experiencing: • Reductions in number of pills prescribed (with exceptions) • Increases in number of people with safe disposal kits • Reductions in prescription drug-related deaths (increases in heroin and fentanyl)

27. Statewide Logic Model

28. Impaired Driving: Perceptions of Risk • Risk of being injured in an crash • Risk of injuring someone in a crash • Risk of arrest/citation • Data Sources: • YRBS (annual, but statewide) • CTC (biennial) • PIRE community survey (pre-post) • Arrest/citation rates

29. Impaired Driving Perceptions: Arrests/Citations per 100 Cars

30. Impaired Driving Consumption • Drinking and driving • Driving with someone who drank • Data Sources: • YRBS (annual, but statewide) • CTC (biennial) • PIRE community survey (pre-post)

31. Impaired Driving Consequences DUI crashes (by age group) Percent crashes that are DUI (by age) DUI fatalities Rate of DUI fatalities Single vehicle nighttime crashes

32. Impaired Driving Consequences:DUI Crashes Under 21

33. Impaired Driving Consequences:DUI Crashes Total

34. Impaired Driving Story Cannot make causal attributions but can link strategies with outcomes you are monitoring. Story should be accurately reflected and should provide insights into your strategies. • Community is implementing: • Saturation patrols (provide numbers) • Safety checkpoints (provide numbers)Prescriber education • Routine enforcement (provide numbers) • Media campaigns about risks associated with impaired driving (provide numbers) • Community is experiencing: • Increases in the rates of citations/arrests • Decreases in DUI crashes