FROM GOOD INTENTIONS TO EFFECTIVE INTERVENTIONS: A PRIMER ON EVIDENCE-BASED PREVENTION PROGRAMS

1. FROM GOOD INTENTIONS TO EFFECTIVE INTERVENTIONS:A PRIMER ON EVIDENCE-BASED PREVENTION PROGRAMS May 18, 2006 Stephen SmallUniversity of Wisconsin-Extension and University of Wisconsin-Madison

2. What Works, Wisconsin Full report and additional information available at: www.oja.state.wi.us/jj

3. Overview of today’s presentation • What are evidence-based programs? • What does it mean for a program to be “cost-effective”? • What are some principles of effective programs? • How can we use this knowledge to improve existing programs?

4. The Three Little Pigs The real story

5. What are evidence-based programs? A new class of programs that: • Use scientific method: Program has been carefully implemented and evaluated using rigorous scientific methods including a control or comparison group and quality measures • Have been replicated:the findings have been repeated in a number of studies • Have generalizable findings:the findings have been found to apply to other settings and audiences • Have a strong grounding in current scientific theory • Meet rigorous review standards: Have evaluation findings that have been subjected to critical peer review and published in respected scientific journals • Have been “certified” as evidence-based:Respectedfederal agency or research organization has reviewed evidence

6. Levels of Effectiveness CENTER FOR THE APPLICATION OF PREVENTION TECHNOLOGIES (CAPT), Minnesota Institute of Public Health

7. What evidence-based is NOT… It’s not the Christopher Columbus approach to programming: • Step #1: When he started, he didn’t know where he was going… • Step #2: When arrived, he didn’t know where he was… • Step #3: When finished, he didn’t know where he had been.

8. Terminology Research-based content  Evidence-based programs

9. History of evidence-based programs • Medical practice and health care • Public health • Social work • Clinical and school psychology • Prevention science • Other fields

10. Classifications of Evidence-Based Prevention Programs • Universal: interventions directed at the general public or an entire population (e.g., all new parents)

11. Classifications of Evidence-Based Prevention Programs • Selective: interventions directed at a subgroup of a population at risk of developing a problem but not yet exhibiting any difficulties(e.g., parents who are isolated or experiencing high levels of stress)

12. Classifications of Evidence-Based Prevention Programs • Indicated: interventions targeted at high risk individuals who show some signs or symptoms of a problem (e.g., parents who who are showing warning signs of abusive behavior)

13. Terminology • Efficacy versus Effectiveness

14. Number of evidence-based programs 45 40 35 30 25 20 15 10 5 1950s 1960s 1970s 1980s 1990s

15. Market Penetration of Evidence-Based, Prevention Programs 1% Jensen, 2003

16. Evidence-based program registries • Number of registries is growing • Most focus on a particular area of interest • There is a great deal of overlap between evidence-based registries

22. Benchmark:   Increase the percentage of children who do not experience physical, psychological or emotional abuse • Proven Programs • Child-Parent CentersNurse Family Partnership • Promising Programs • Child Sexual Abuse Prevention: Teacher Training WorkshopProject TRUSTSafe Child ProgramWho Do You Tell?

23. Common program labels • Exemplary Programs (Department of Education) • Effective Programs (CDC, NIDA, DHHS) • Model Programs (SAMHSA, OJJDP Blueprints, Surgeon General) • Proven Programs (Promising Practices Network) • What Works(Child Trends)

24. Why the interest in evidence-based programs? • Body of scientific evidence has reached a critical mass • Public accountability • Efficiency (don’t need to reinvent the wheel) • Increases the likelihood that programs will have the impact that they were designed to produce • Evidence helps sell the program to funders, stakeholders and potential audiences • Data may be available to estimate cost effectiveness

25. Adolescent Cigarette Use Over Time Number of cigarettes per month Spoth, R.L.; Redmond, C.; and Shin, C. Randomized trial of brief family interventions for general populations: Adolescent substance use outcomes 4 years following baseline. Journal of Consulting and Clinical Psychology 69(4):627-642, 2001.

26. Sample evidence from an evidence-based program Aggressive and hostile behaviors Aggression and Hostility Index SOURCE: Spoth, R., Redmond, C., & Shin, C. (2000) Reducing adolescents' aggressive and hostile behaviors: Randomized trial effects of a brief family intervention four years past baseline. Archives of Pediatrics and Adolescent Medicine 154, 1248-1257

28. Cost-benefit analysis Assessment of program impact taking costs into account “Evaluation of alternatives according to their costs and benefits when each is measured in monetary terms.”Levin & McEwan (2002) Program economic-benefit indicators: 1. Net economic return is Benefits – Costs 2. Benefit-cost ratio is Benefits / Costs

29. Making sense of cost-benefit analysis: An illustration Two children born in similar circumstances Johnny Ricky

30. Both live in environments characterized by risk factors such as: • Poverty • Family history of criminal behavior • Family history of maltreatment or neglect • Single parent household • High crime neighborhood • Low parental educational attainment

31. At age 3 Johnny is cared for at home by his older sibling and aunt Ricky is enrolled in the Chicago Child-Parent Center Johnny Ricky

32. The program provides educational enrichment for Ricky

33. Parent education for Ricky’s mother

34. Home visits, health screening, and other services

35. $0 - $5,364 per year for 2 years Public costs for Johnny & Ricky in the preschool years Johnny Ricky NOTE: All dollar amounts are converted to 2006 dollars, based on present-value calculations in 1998 dollars published in Reynolds, Temple, Robertson, & Mann (2002). Age 21 Cost-Benefit Analysis of the Title I Chicago Child-Parent Centers. Educational Evaluation and Policy Analysis. 24(4), 267-303.

36. Johnny is enrolled in special education classes Ricky succeeds at school At age 9 - $9,497 per yearfor 4 years (Above and beyond normal instruction costs) $0 (Only normal instruction costs) Johnny Ricky

37. Johnny is enrolled in special education classes Ricky succeeds at school At age 9 24.6% of comparison children were enrolled in special education for at least one year, compared to only 14.4% of the preschool participants. Johnny Ricky

38. Johnny is found to be the victim of child abuse Ricky’s family has no contact with the child welfare system At age 10 - $10,861 (Average cost for child welfare services with a substantiated report) $0 Johnny Ricky

39. Johnny is found to be the victim of child abuse Ricky’s family has no contact with the child welfare system At age 10 10.3% of comparison children were found to be victims of maltreatment between the ages of 4 and 17, compared to only 5% of the preschool participants. Johnny Ricky

40. At age 14 Johnny is arrested Ricky stays out of trouble - $16,690 (Average juvenile justice system expenditure for a court petition) $0 Johnny Ricky

41. At age 14 Johnny is arrested Ricky stays out of trouble 25.1% of comparison children had at least one petition to juvenile court by age 18, compared to only 16.9% of the program participants. Johnny Ricky

42. At age 18 Johnny doesn’t graduate from high school and is often unemployed Ricky graduates from high school and enrolls in college $0 - $4,039 per yearfor 4 years (Average taxpayer share of cost for tuition at a Chicago university) Johnny Ricky

43. At age 18 Johnny doesn’t graduate from high school and is often unemployed Ricky graduates from high school and enrolls in college 61.9% of preschool participants had graduated from high school by age 21, compared to only 51.4% of the comparison group. At age 21, 47% of program participants were attending college. Johnny Ricky

44. Entering adulthood Johnny is in and out of jail for petty crimes Ricky graduates college and gets a good job - $40,195 (Average criminal justice system expenditure over an adult criminal career) + $78,838 (Increased tax revenue based on lifetime earnings of $223,303 greater than a non-graduate) Johnny Ricky

45. Entering adulthood Johnny is in and out of jail for petty crimes Ricky graduates college and gets a good job These projections for adult outcomes were based on the age-21 follow-up and national statistics. Researchers continue to track the preschool participants and comparison group and will refine these estimates based on reports of actual outcomes throughout their adult years. Johnny Ricky

46. Other differences in Johnny and Ricky’s adult lives Johnny has poor health Ricky is more likely to be healthy Johnny relies on public systems for living expenses and health care Ricky supports himself and his family, pays taxes, and stays out of trouble Ricky is more likely to be involved in raising his own children Based on findings from the Perry Preschool Age 40 follow-up and projections from the Chicago Child-Parent Study at Age 21

47. Overall costs and benefits to society Chicago Child-Parent Center Special education - $37,988 - $10,728 Child welfare system College tuition - $10,861 - $16,156 Juvenile delinquency Increased tax revenue - $16,690 + $78,838 Adult crime - $40,195 TOTAL: - $105,734 TOTAL: + $51,954 Johnny Ricky

48. Cost-benefit analysis for Chicago Child-Parent Centers • Based on actual differences at age 21 between group that attended preschool program (n=989) and comparison group that did not (n=550) • The study also took into account other economic factors, such as costs to crime victims and benefits to individual program participants • The study looks at various combinations of preschool participation (1 year or 2 years), school-age program participation, or no program • Overall, the program was found to have benefits of $10.15 for every dollar spent

49. Cost-benefit analysis summary • Cost-benefit analysis builds on a rigorous evaluation of program impacts over time • The analysis takes into account: • Costs of program participation • Costs and monetary benefits – for individuals and for society – of various outcomes • Percentage of individuals in each group who experience those outcomes • Average impact of the program across individuals

50. Key CBA results from primary prevention programs Program Costs Benefits B – C B / C Chi Child-Parent $7,755 $78,732 $70,977 $10.15 Perry Preschool $16,648 $145,414 $128,766 $8.74 NurseFamilyPart $7,324 $37,041 $29,717 $5.06 Strengthening $851 $6,656 $5,805 $7.82 Families10-14 (2004 dollars)