The Standardized Program Evaluation Protocol ~SPEP~

1. Division of Adult Correction and Juvenile Justice The Standardized Program Evaluation Protocol~SPEP~ Understanding the Tool and Program Improvement Implications

2. Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices • First State-wide community-based alternatives (CBA) ~ late 1970s • First “wraparound” model in the US via Willie M. Program~ 1979 class action lawsuit • 1998 NC Juvenile Justice Reform Act ~ Second state to mandate evidence-based services

3. Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices • First state to fully adopt the OJJDP Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders • First legislated statewide, county-level Juvenile Crime Prevention Councils (also first statewide use of a risk factor assessment beginning in the late 1990’s) • Most effective statewide use of a juvenile offender risk and needs assessment instrument along with a disposition matrix to reduce confinement (Nov, 2001)

4. Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices • First state to begin implementation of a state-wide gang prevention and intervention initiative (Community-Based Youth Gang Violence Prevention Project) 2009 (Use of Recovery Act funds) (THE first U. S. experiment to fund statewide Comprehensive Gang Model programming!)

5. Historical Context of SPEP in NC • 1998 NC Juvenile Justice Reform Act required a statewide evaluation of programs funded through the JCPC • The Standardized Program Evaluation Protocol ~SPEP~ was developed specifically for this purpose in NC. 2006 • First statewide Level II services based on SPEP primary service types (2010)

6. Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr. Mark Lipsey, 2002)

7. The SPEP Process Key Steps to SPEP Score

8. IDENTIFY SERVICES SPEP SCORING PROCESS MATCHAGAINST RESEARCH-BASED CATEGORIES DATA(DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE SPEP SCORE EVALUATE/PROGRAM IMPROVEMENT

9. IDENTIFY/ MATCH SERVICES DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE)

10. Getting Started……. Classification of Program Intervention

11. SPEP 101 Definitions • Structure or Format: Setting within which the program services are delivered • Example: Residential, YDC, Day Treatment • Program: Services that are delivered within the context of a structure or format. • Service: The treatment ingredient of the program. Services target key criminogenic risk factors and treatment needs and are direct interventions with the juvenile or with others (peers or family) in order to alter juvenile behavior that leads to reduced recidivism.

12. Identification of “Philosophies” toward Altering Juvenile Behavior • Program Services Fall into 3 Broad Categories: • External Control Techniques • Behavioral change through instilling discipline, fear, or detection of bad behavior in the absence of treatment • Therapeutic Techniques • Behavioral change through improved skills, relationships, insight. • These are the only services for which SPEP ratings are completed. • Other :Services for which there is insufficient research to estimate the effects on recidivism

13. Program Types Sorted by General Approach: Average Recidivism Effect Discipline Deterrence Surveillance Restorative Skill building** Counseling* Multiple services -10 -5 0 5 10 15 % Recidivism Reduction from .50 Baseline Therapeutic approaches Control approaches Control approaches

14. Further Sorting by Intervention Type within, e.g., Counseling Approaches

15. Further Sorting by Intervention Type within, e.g., Skill-building Approaches

16. SPEP 101 Definitions • Primary Service: • Main focus of the program • Most time/effort spent on this • Recognized as the dominant theme • Other program elements support its effectiveness

17. SPEP 101 Definitions • Supplemental Service: • Other distinct service • Less time/effort spent on this • Serves to reinforce or complement the primary service • Note: Supplemental service can be deemed as a “qualifying” one in order to receive additional points on the SPEP tool OR points may be automatic based on lack of evidence in the research base to couple a supplemental service with the identified primary service.

18. WHAT IS THE PROGRAM’S PRIMARY SERVICE? IN WHICH OF THE 5 SERVICE GROUPS DOES THE SERVICE FALL? IS THERE A QUALIFYING SUPPLEMENTAL SERVICE ?

19. Things to Consider……… • JCPCs will need to fund structures or formats. (Teen Court, Structured Day Programs) • It is important that an analysis of key therapeutic element (s) within the structures be explored. • JCPCs need to fund Therapeutic Approaches/Interventions • JCPCs need to understand the SPEP process in order to understand SPEP scores and their implications for program improvement.

20. DATA Examining the Service Quality, Amount of Service (Duration and Contact Hours), and Risk Level

21. Data Used for SPEP Scores • Quality of Service: • SPEP Quality of Services Checklist~ Completed with the Area Consultant • Examines the following: • Written program protocol ~ program manual; written protocol that describes how the intended services is delivered • Staff training /staff retention • Monitoring of Program effectiveness, protocol and corrective action to correct deviations

22. SPEP Quality of Services Checklist Score

23. Data Used for SPEP Scores • Amount of Service: (NCALLIES) • Duration of Service • Target number of weeks specified for each service type • % of youth who received targeted weeks of service • Maximum number of points ~10 • Contact Hours • Target number of hours specified for each service type • % of youth who received targeted hours of service • Maximum number of points ~10

24. Data generated from NC ALLIES

25. Data Used for SPEP Scores • Risk level of Youth Served: • Program’s use of a validated risk assessment instrument/tool to provide juvenile risk level data on each juvenile receiving services • North Carolina’s Assessment of Juvenile Risk of Future Offending~ Validated tool currently being used by Juvenile Court Services. This risk assessment tool has been selected for use by program providers. The department will train providers on its use. • NC ALLIES will be modified with the addition of a Risk Level field. Risk Level Scores from referrals from Juvenile Court would be entered as well as risk level scores generated by the program. • Future modifications will include incorporating the risk assessment tool into NCALLIES.

26. Data Generated via a Validated Risk Assessment Tool and entered into NC ALLIES

27. The SPEP SCORE • Examines how a specific program is performing compared to the effective practice for that service type in the research • Provides opportunities for program improvement for programs that fall short of the evidence-based effective practice profiles.

28. Systematic Correction to Optimize Recidivism reduction Effectively

29. Scoring = Improvement Opportunity Initial Score Understanding the Score Implications for Improvement Program Improvement Plan Subsequent Scoring Repeat

30. Program Improvement • Program Improvement Plans address the areas needing improvement as identified through the program’s SPEP score. • Timeframes for program improvement will be established • Monitoring of the status of program improvement will be incorporated into the JCPC monitoring/planning processes and Consultant Program Monitoring • JCPCs would continue to evaluate the effectiveness of their funded programs through general oversight and supporting program improvement plans.

31. Division of Adult Correction and Juvenile Justice Questions? Ronald Tillman Central Area Consultant Ronald.tillman@ncdps.gov 919-323-6845