Planning for Performance Measurement: Development of the OJJDP Performance Measurement System

1. Planning for Performance Measurement: Development of the OJJDP Performance Measurement System Presenters: Greg Thompson, OJJDP Dennis Mondoro, OJJDP Heidi Hsia, OJJDP

2. Day 1 Objectives • Enhance planning for performance measurement (PM) at the State level • Review development of the Performance measurement system • Review logic models and measures • Incorporate PM into the RFP • Understand reporting guidelines

3. Day 2 Objectives • Overcoming challenges in implementing performance measurement and being able to train subgrantees to: • Identify data sources • Share information • Monitor and track effectively Understand GMS reporting system and how to do performance reporting on GMS

4. The Federal Climate • Governmentwide move toward accountability • Government Performance and Results Act • President’s Management Agenda • Office of Management and Budget • Program Assessment Rating Tool • Paperwork Reduction Act of 1995 Programs need to show effectiveness to justify funding.

5. Funding and Information Flows Congress $ $ $ OJJDP Information State Subgrantee

6. Information Example Congress OJJDP State State Subgrantee Subgrantee Subgrantee Subgrantee

7. Development Methodology 1. Categorize Formula Grants Program Areas: • System Improvement • Core Requirements • Prevention • Intervention 2. Develop logic models. 3. Define common indicators. 4. Apply prevention and selected intervention models to Title V Analysis possible by or across Program Area

8. Sources for Indicator Development • 1. Federal guidelines and best practices • 2. Literature from the field • 3. Examples from other agencies at the Federal, State, and local levels • JJ Specialist/Title V Coordinator input • Written feedback • Conference calls

9. Planning for Performance Measurement: Reviewing the Logic Models and Program Areas Presenters: Marcia Cohen, DSG, Inc. Heidi Hsia, OJJDP

10. What Is Performance Measurement? It is directly related to program goals and objectives. It measures progress quantitatively. It is not exhaustive. It provides a “temperature” reading—it may not tell you everything you want to know but provides a quick and reliable gauge of selected results.

11. Measurement vs. Evaluation Impact evaluations are broader and assess the overall or net effects—intended or unintended—of the program as a whole.* Impact evaluation Scope Evaluation Outcome evaluations investigate whether the program causes demonstrable effects on specifically defined target outcomes.* Outcome evaluation Process evaluations investigate the process of delivering the program, focusing primarily on inputs, activities, and outputs.* Process evaluation Performance Measurement Program Monitoring Time * Evaluation definitions excerpt from: Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://trochim.human.cornell.edu/kb/index.htm> (version current as of Aug. 02, 2000).

12. Outputs Versus Outcomes • Outputs are products of program implementation/activities. • Outcomesare benefits or changes as a result of the program. There are two types of outcomes: • Short-term: The first benefits or changes experienced and the ones most closely related to program outputs. • Long-term: Link a program’s short-term and long-term outcomes. Often they are changes in practice, policy, decision-making, or behavior.

13. Outputs Versus Outcomes (cont’d.) With regard to the Formula Grants and Title V performance measures, to summarize: • Outputs are at the micro level and reflect program-level activity. • Outcomes are at the macro level and, when aggregated, will reflect Federal outcomes. • A good performance measurement system should be results oriented and focus on desired outcomes, less on outputs.

14. Outcome Measure Definitions Short-term: Occurs during the program or by the end of the program. Long-term: Occurs 6 months to 1 year after program completion.

15. Logic Model: Template A graphic representation that clearly lays out the logical relationships between the problem to be addressed, program activities, outputs, and outcomes. Outcomes Problem Activities Outputs • Short term • Long term

16. Overall Formula Grants and Title V Programs Logic Model Activities Outputs Short Term Long Term Goals Objectives Juvenile Justice System Improvement Program Areas: 10, 19, 23, 31, 33 Increased system capacity Improved planning and development Improved program quality Improved monitoring of compliance Implement processes Improved policies, procedures, operations, staffing, service delivery Improved juvenile justice systems Increased compliance with Core Requirements To improve juvenile justice systems by increasing compliance with the Core Requirements and increasing availability and types of prevention and intervention programs To support both State and local prevention and intervention efforts and juvenile justice systems improvements Improved youth outcomes Core Requirements Program Areas: 6, 8, 10, 17, 28 Reduced recidivism Increased accountability Implementing Title V programs for keeping at-risk youth from offending or first-time, nonserious youth out of the JJ system. Program Areas: 3, 4, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 25, 26, 27, 32, 34 Implementing Formula Grants prevention and intervention programs Program Areas: 1, 2, 3, 4, 5, 7, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 25, 26, 27, 29, 30, 32, 33, 34 Key = System-level indicator = Program-level indicator OUTCOMES Problems Juvenile Delinquency Outcome measures are for illustrative purposes only and are not comprehensive. To see a comprehensive list of outcomes, refer to the individual program area logic models.

17. Title V Program Areas Title V programs are for keeping at-risk youth from offending or first-time, nonserious youth out of the JJ system. Title V has 18 Prevention and Early Intervention Program Areas: 3, 4, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 25, 26, 27, 32, 34

18. Formula Grants Program Areas Three Formula Grants Program Area categories: Prevention and Intervention Program Areas: 1, 2, 3, 4, 5, 7, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 25, 26, 27, 29, 30, 32, 33, 34 JJ System ImprovementProgram Areas: 10, 19, 23, 31, 33 Core RequirementsProgram Areas: 6, 8, 10, 17, 28

19. Planning for Performance Measurement: Reviewing the Performance Measures Presenters: Gregory Thompson, OJJDP Heidi Hsia, OJJDP Marcia Cohen, DSG, Inc.

20. Reporting Requirements • Output:All mandatory (bold) and two non-mandatory measures. • Outcome:All mandatory (bold)andtwo non-mandatory measures (may be either short- term or long-term).* *Except Core Requirements and JJ System Improvement require only one non-mandatory measure.

21. Key to Logic Models and Grid Charts Bold = Mandatory measure * = Mandatory only for intervention programs ** = Mandatory only for prevention programs + = Mandatory if applicable

22. Output Objective Categories • Six types of output objectives: • Increase organizational/system capacity •   Improve planning and development • Improve program activities • Improve program efficiency • Improve the monitoring of compliance

23. Outcome Objective Categories • Nine types of outcome objectives: • Reduce delinquency • Increase accountability • Improve system effectiveness • Increase prosocial behaviors • Increase program support • Increase compliance with Core Requirements • Reduce disproportionate minority contact • Improve school climate • Improve the management of the State Juvenile • Justice and Delinquency Prevention • program

24. Prevention and Intervention Program Areas

25. Mandatory Output Measures for Prevention and Intervention Programs • Title V or FG $ awarded for services: • The amount of Formula Grants and Title V funds in whole dollars that are awarded for delinquency prevention services during the reporting period. Program records are the preferred data source.

26. Mandatory Output Measures for Prevention and Intervention Programs (cont’d) • Number of program youths served: • An unduplicated count of the number of youths served by the program during the reporting period. Definition of the number of youth served for a reporting period is the number of program youth carried over from previous reporting period, plus new admissions during the reporting period.  In calculating the 3-year summary, the total number of youth served is the number of participants carried over from the year previous to the first fiscal year, plus all new admissions during the 3 reporting fiscal years. Program records are the preferred data source.

27. Mandatory Outcome Measures for Prevention** Programs • Number and percent of program youths exhibiting desired change in targeted behaviors: • Substance use • School attendance • Antisocial behavior • Family relationships • Pregnancies Note: Double ** denotes mandatory for prevention programs

28. Mandatory Outcome Measures for Intervention* Programs • Number and percent of program youths who offend or reoffend: • The number of program youths who were rearrested or seen at juvenile court for a new delinquent offense. Appropriate for any youth-serving program. Note: Single * denotes mandatory for intervention programs

29. Mandatory Outcome Measures for Prevention and Intervention Programs • Number and percent of youth completing program requirements: • The number and percent of program youth who have successfully fulfilled all program obligations and requirements. Program requirements will vary by program but should be a predefined list of requirements or obligations that clients must meet prior to program completion. Program records are the preferred data source.

30. Example: Probation Program Area

31. Sample Logic Model: PA 24—Probation OUTCOME MEASURES ACTIVITIES OUTPUT MEASURES SHORT-TERM LONG-TERM SUBPROBLEM(S) #/% OF PROGRAM YOUTH WHO OFFEND OR REOFFEND #/% OF PROGRAM YOUTH WHO OFFEND OR REOFFEND FG $ AWARDED FOR SERVICES # of planning activities conducted # of MOUs developed # of program slots available # of FTEs funded with FG $ #/% of program staff trained # of hours of program staff training provided Use of best practice model (Y/N) # of program materials developed # of program/agency policies or procedures, created, amended, or rescinded INTERVENTION PROGRAM (Deficiency or lack of adequate probation programs) Conduct Planning Activities and Implement Program(s) #/% of program youth committed to correctional facility #/% of program youth committed to correctional facility #/% of program youth charged with formal probation violation Objectives To support both State and local intervention efforts and JJ system improvements #/% OF PROGRAM YOUTH EXHIBITING DESIRED CHANGE IN TARGETED BEHAVIORS (E.G., SUBSTANCE USE, SCHOOL ATTENDANCE, ANTISOCIAL BEHAVIOR, FAMILY RELATIONSHIPS, SOCIAL COMPETENCIES) # OF PROGRAM YOUTH SERVED Average length of stay in program # of service hours completed Monitor Program(s) #/% OF PROGRAM YOUTH COMPLETING PROGRAM REQUIREMENTS #/% of program staff with increased knowledge of program area • Long-Term Outcome Objectives • Reduce delinquency • Increase accountability • Short-Term Outcome Objectives • Reduce delinquency • Improve prosocial behaviors • Increase accountability • Increase program support • Output Objectives • Increase organizational capacity • Improve planning & development • Improve program quality • Improve program activities • Improve program efficiency Key = System-level indicator = Individual-level indicator = Objectives Bold= Mandatory measure PROBLEM Delinquency Goals To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs Performance measures should report on those activities funded by Title II (Formula Grants) funds. Outcome Measure Definitions Short Term:Occurs during the program or by the end of the program Long Term:Occurs 6 months to 1 year after program completion

32. Sample Grid Chart: PA 24—Probation Output Performance Measures PROGRAM AREA 24. Probation OUTPUT PERFORMANCE MEASURES

33. Sample Grid Chart: PA 24—Probation Outcome Performance Measures

34. Example: Delinquency Prevention Program Area

35. Sample Logic Model: PA 09—Delinquency Prevention OUTCOME MEASURES ACTIVITIES OUTPUT MEASURES SHORT-TERM LONG-TERM PROBLEM SUBPROBLEM(S) FG $ AWARDED FOR SERVICES # of MOUs developed # of planning activities conducted # of program slots available # of FTEs funded with FG or TV $ #/% of program staff trained # of hours of program staff training provided Use of best practice model (Y/N) # of program materials developed # of program/agency policies or procedures, created, amended, or rescinded #/% OF PROGRAM YOUTH EXHIBITING DESIRED CHANGE INTARGETED BEHAVIORS (E.G., SUBSTANCE USE, SCHOOL ATTENDANCE, ANTISOCIAL BEHAVIOR, FAMILY RELATIONSHIPS, PREGNANCIES) #/% OF PROGRAM YOUTH EXHIBITING DESIRED CHANGE INTARGETED BEHAVIORS (E.G., SUBSTANCE USE, SCHOOL ATTENDANCE, ANTISOCIAL BEHAVIOR, FAMILY RELATIONSHIPS, PREGNANCIES) Delinquency PREVENTION PROGRAM (Deficiency or lack of adequate delinquency prevention programs) Conduct Planning Activities and Implement Program(s) Goals Objectives To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs #/% OF PROGRAM YOUTH COMPLETING PROGRAM REQUIREMENTS To support both State and local intervention efforts and JJ system improvements #/% of youth satisfied with program Monitor Program(s) # OF PROGRAM YOUTH SERVED #/% of parents served Average length of stay in program # of service hours completed #/% of program families satisfied with program Performance measures should report on those activities funded by Title II (Formula Grants) funds. #/% of program staff with increased knowledge of program area Outcome Measure Definitions Short Term:Occurs during the program or by the end of the program Long Term:Occurs 6 months to 1 year after program completion • Long-Term Outcome Objectives • Improve prosocial behaviors • Short-Term Outcome Objectives • Improve prosocial behaviors • Increase program support • Increase accountability • Output Objectives • Increase organizational capacity • Improve planning & development • Improve program quality • Improve program activities • Improve program efficiency Key = System-level indicator = Individual-level indicator = Objectives Bold= Mandatory measure

36. Sample Grid Chart: PA 09—Delinquency Prevention Output Performance Measures

37. Sample Grid Chart: PA 09—Delinquency Prevention Outcomes Performance Measures

38. Core Requirements Program Areas

39. Mandatory Output Measures forCore Requirements Programs • FG funds awarded: • The amount of Formula Grants money in whole dollars that is awarded for the separation requirement during the reporting period. Program records are the preferred data source.

40. Mandatory Output Measures for Core Requirements Programs (cont’d) • Number of programs implemented: • The number of programs created and/or implemented during the reporting period. Program records are the preferred data source.

41. Mandatory Outcome Measure for Core Requirements Programs Change in the number of violations The change in the number of violations of the [sight and sound] requirement from the previous reporting period compared with the current reporting period. Sight and sound compliance is determined according to the definition in the OJJDP Guidance Manual for Monitoring Facilities Under the JJDP Act of 2002. The Annual Compliance Monitoring Report is the preferred data source.

42. Example: Separation Program Area

43. Sample Logic Model: PA 28—Separation ACTIVITIES OUTPUT MEASURES OUTCOME MEASURES SUBPROBLEM(S) CORE REQUIREMENT (Ensuring/monitoring compliance with the Separation Core Requirement) Conduct Planning and Oversight Activities Implement Processes and/or Programs FG $ AWARDED FOR SEPARATION # of MOUs developed Needs assessment completed (Y/N) #/% OF PROGRAMS IMPLEMENTED # of transportation plans developed # of shelter beds contracted #/% of staff trained # of hours of staff training provided # of materials developed # of program policies/procedures created, amended, or rescinded CHANGE IN THE NUMBER OF VIOLATIONS OF SIGHT AND SOUND SEPARATIONS #/% of staff with increased knowledge of program area (e.g., separation regulations) Objectives To support both State and local prevention and intervention efforts and JJ system improvements Monitor Processes and/or Programs #/% of site visits conducted # of facilities receiving TA • Output Objectives • Improve monitoring of compliance • Increase organizational capacity • Improve planning & development • Outcome Objectives • Increase compliance with core requirements • Increase program support Key = System-level indicator = Individual-level indicator = Objectives Bold= Mandatory measure PROBLEM Delinquency Goals To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs. Performance measures should report on those activities funded by Title II (Formula Grants) funds. Outcome Measure Definitions Reported Annually.

44. Sample Grid Chart: PA 28—Separation Output Performance Measures

45. Sample Grid Chart: PA 28—Separation Outcome Performance Measures

46. Example: Compliance Monitoring Program Area

47. Sample Logic Model: PA 06—Compliance Monitoring ACTIVITIES OUTPUT MEASURES OUTCOME MEASURES SUBPROBLEM(S) CORE REQUIREMENT (Ensuring/monitoring compliance with Section 223 (a)(14) of the JJDP Act of 2002) FUNDS ALLOCATED TO ADHERE TO SECTION 223 (a)(14) OF THE JJDP ACT OF 2002 # of activities that address compliance with Section 223 (a)(14) of the JJDP Act of 2002 # of MOUs developed #/% of program staff trained # of hours of program staff training provided # of program materials developed # of program policies/procedures created, amended, or rescinded # of facilities receiving technical assistance SUBMISSION OF COMPLETE ANNUAL MONITORING REPORT TO OJJDP Conduct Planning and Oversight Activities #/% of staff with increased knowledge of program area (e.g., compliance monitoring) Objectives To support both State and local prevention and intervention efforts and JJ system improvements • Output Objectives • Improve monitoring of compliance • Increase organizational capacity • Improve planning & development • Outcome Objectives • Improve the management of the State JJDP Program • Increase program support Key = System-level indicator = Individual-level indicator = Objectives Bold= Mandatory measure PROBLEM Delinquency Goals To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs Performance measures should report on those activities funded by Title II (Formula Grants) funds. Outcome Measure Definitions Reported Annually.

48. Sample Grid Chart: PA 06—Compliance Monitoring Output Performance Measures

49. Sample Grid Chart: PA 06—Compliance Monitoring Outcome Performance Measures

50. Sample Logic Model:PA 10 – Disproportionate Minority Contact OUTCOME MEASURES SUBPROBLEM(S) ACTIVITIES OUTPUT MEASURES Short-Term Long-Term CORE REQUIREMENT (Ensuring compliance with the DMC Core Requirement) FG OR TV $ AWARDED FOR DMC AT THE STATE AND LOCAL LEVELS # of FTEs funded with FG or TV $ # of activities conducted + # OF PROGRAMS IMPLEMENTED #/% of program staff trained # of hours of program staff training provided # of program materials developed # of program/agency policies or procedures, created, amended, or rescinded # of objective decision-making tools developed +# OF STATE AGENCIES WITH IMPROVED DATA COLLECTION SYSTEMS + % OF CONTACT POINTS REPORTING REDUCTION IN DISPROPORTIONALITY AT STATE LEVEL Conduct Planning Activities & Implement Program(s) and Processes #/% of recommendations implemented from assessment studies INTERVENTION PROGRAM (Deficiency or lack of adequate intervention to reduce DMC) + % OF CONTACT POINTS REPORTING REDUCTION IN DISPROPORTIONALITY AT LOCAL LEVEL Number of minority staff hired PREVENTION PROGRAM (Deficiency or lack of adequate prevention programs to reduce DMC) + * #/% OF PROGRAM YOUTH WHO OFFEND OR REOFFEND + * #/% OF PROGRAM YOUTH WHO OFFEND OR REOFFEND + ** #/% OF PROGRAM YOUTH EXHIBITING DESIRED CHANGE INTARGETED BEHAVIORS (E.G., SUBSTANCE USE, SCHOOL ATTENDANCE, FAMILY RELATIONSHIPS, AND ANTISOCIAL BEHAVIOR) JUVENILEJUSTICE SYSTEM IMPROVEMENT (Lack of effective activities associated with planning and administration of the Formula Grants Program) # of local agencies reporting improved data collection systems Monitor Program(s) and/or Programs + # OF PROGRAM YOUTH SERVED Average length of stay in program # of service hours completed #/% of site/TA visits conducted #/% of state agencies reporting improved data collection systems + #/% OF PROGRAM YOUTH COMPLETING PROGRAM REQUIREMENTS #/% of program families satisfied with program Objectives To support both State and local prevention and intervention efforts and JJ system improvements #/% of program youth satisfied with program #/% of program staff with increased knowledge of program area • Outcome Objectives • Reduce DMC • Improve system effectiveness • Reduce delinquency • Output Objectives • Improve monitoring of compliance • Increase system capacity • Improve program quality • Improve planning & development • Outcome Objectives • Reduce DMC • Improve system effectiveness • Improve prosocial behaviors • Reduce delinquency • Increase program support PROBLEM Delinquency Goals To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs Performance Measures should report on those activities funded by Title II (Formula Grants) funds. Outcome Measure Definitions Short-Term:Occurs during the program or by the end of the program Long-Term:Occurs 6 months to 1 year after program completion Key = system-level indicator = individual-level indicator = objectives *Bold = mandatory measure + = mandatory only if applicable (if not applicable, choose a different measure) * = mandatory for intervention programs only ** = mandatory for prevention programs only