Evaluation of the Fighting Back Initiative by Kay E. Sherwood

1. Evaluation of the Fighting Back Initiativeby Kay E. Sherwood Presented by Maddie Velez

2. What is Fighting Back? A community-based drug abuse prevention program

3. Why study this Initiative? Shows the importance of taking context into evaluations Raises questions about how community interventions are conceptualized and evaluated Provides a warning about the manageability of large-scale, comprehensive evaluations.

4. Background: Duration and Scale Contribute to Complexity • A 12 year initiative • Original stakeholders differed greatly from the stakeholders involved 12 years later • Reduction of intervention sites went from 15 to 5 • First evaluation team replaced after 2 years. • High staff / leadership turnover • Original key leader retires • Few examples of credible, successful evaluations that truly measured the interventions impact

5. The Foundation Takes on Substance abuse • Robert Wood Johnson heads up the Foundation’s first efforts in the area of substance abuse. • First grant was made to Vanderbilt University for $26.4 million in 1988 • Foundation explores addressing the national problems of substance abuse and dependence

6. Continued… • July 1988…the goal became “by pulling together into a single unified effort, communities can begin to solve the pressing problem of drug and alcohol abuse.” • The expectation…”to reduce the demand for illegal drugs and alcohol in the funded communities.” • Project STAR and ALERT • Poly abuse - combination of mental health problems and substance abuse occurring

7. New Leadership: Kathryn Edmundson • New evaluation agenda: Could you organize to create political will for change at the local level and get it to add up to a national-level movement? • An element of racism and elitism in the law enforcement • Expected outcomes

8. Evaluation I: Lost time, Money, and Credibility • 1990-1994 • The first evaluation team replaced, 4 years, $4.6 million, and a baseline • Division between stakeholders missed changes • Augment between the 2nd evaluation team and foundation staff regarding lack of baseline data.

9. A 1996 Watershed • Become unified with an emphasis on prevention, early intervention, treatment, and aftercare. • NPO (National Program Office) moved to Boston University School of Public Health. • NPO joined another foundation funded program called “Joined Together”, with new director David Rosenbloom. • Board of Trustees makes a recommendation to give the program Fighting Back more time. • Preliminary analysis indicates data that during mid-implementation Fighting Back had no effect.

10. A National Program Office Change • Fighting Back reduces # of sites eligible for new funding. • Measure most substance abuse within the communities to be able to do something measurable at community level. • Increasing treatment and treatment capacity an important goal.

11. 1994-2000 Evaluation II • Consensus 2nd evaluation team does an credible job with difficult circumstances. • 1st Evaluators spend $4.6 million dollars with little to show for it. • Fighting Back Program and evaluation staff is moving forward w/out replacement dollars.

12. Relying on Survey Data • Phone surveys throughout the community. • Management Information Systems (MIS). • Ethnographic Studies. • Community Indicators • Four Research questions were identified by the 2nd evaluation team. • Strong correlations between strategies and outcomes. • Community Indicators • School survey data difficult to use.

13. The Price of Relying on Survey Data • 1996 residue of distrust • Saxe’s research team became known as the “national evaluation • Community has been seen as the “human subject” • National evaluation offer no alternative to outcomes perspective • High emotions surrounding analysis emerged accusations • Bickman claims bias evaluations; Eval. Team are required to point out potential problems in the interventions

14. The Evaluation’s Ability to Explain • Evaluation illustrate all central problems for evaluation • Saxe wanted to undertake a more extensive implementation analysis, foundation unwilling to pay for it • Fighting Back site activities revised after an initial publication in 1997 • Knickman claims the foundation had the wrong goals; He felt that there was a need for shorter-term goals

15. Measuring and Interpreting Outcomes • Key disagreements remain a piece of the national evaluation that focuses on the use of household survey data • 3 waves of surveys- 1995, 1997, 1999 • Jellinek described early thinking on the evaluation • Presentation of Results - A second area of disagreement

16. The Continuing Debate and the Foundation’s Takeaway • Knickman and Morris presented a summary of the Fighting Back experience to the foundation’s board in 4/’04 • Knickman focused on the fundamentals of complexity and the lessons about realistic scale for expected outcomes • Teams were formed • Substance abuse- D.A.R.E. and treatment reform