Performance Measurement Tools for Justice Information Technology Projects

1. Performance Measurement Tools for Justice Information Technology Projects Illinois/ Cook County Focus Group Experience

2. Using “scenario” approach to reach agreement and define performance • Bring stakeholders together to develop a criminal justice scenario • Reach consensus on the desired state of integration • Define the current state of integration (baseline) • Quantify gap between current state and desired state • Define desired outcomes • Develop objectives and performance measures

3. IIJIS Strategic Plan: Existing performance measures 2.1.4. Number of mechanisms identified to reduce paper-based processes 5.2.4. Number of stakeholders adopting functional standards promoting interoperability by September 2003 5.2.5. Number of stakeholder agencies recognized through the certification program 6.1.3. Percent of recommended infrastructure solutions implemented 6.3.3. Year 1: Number of users utilizing the resource center 6.3.4. Year 2: Percent increase of users utilizing the resource center

4. IIJIS Strategic Plan: Existing performance measures 6.4.5. Year 2: Percent increase of stakeholders adopting enterprise-wide disaster recovery plans 6.4.6. Year 1: Number of stakeholders performing disaster recovery tests 6.4.7. Year 2: Percent increase of stakeholders performing disaster recovery tests. 7.3.1. Number of research projects on biometric technological solutions completed by September 2004 7.3.2. Number of research projects on costs and benefits of biometrics completed by September 2004

5. Illinois/Cook County Focus Group on Performance Measures • Sixth in a series of facilitated workgroups by CSLJ to refine & validate PMs • All participants had contributed to development of state and/or Cook County strategic plans • Participants were provided information on PM Project and the PM toolkit • Participants applied toolkit to IIJIS Strategic Plan; then answered questions and shared their reactions with CSLJ staff

6. PARTICIPANT CONCLUSIONS: Performance measures cannot be objectively selected; stakeholders must reach consensus • Stakeholders, depending on their role within the justice enterprise, are likely to have strongly divergent views • Higher-level goals and outcomes will usually be easier to agree upon than specific performance measures • Performance measure selection, however, is not entirely subjective, and some measures are likely to be agreed upon by all

7. PARTICIPANT CONCLUSIONS: It is nearly impossible to definitively establish causal linkages to public safety outcomes • Outcomes become more diffuse the further out you get from actual processes • Improved information sharing is only one of a number of factors affecting public safety • A single performance measure may be inadequate to indicate that a system is working (or not) • A carefully selected family of measures is preferable, and should be thoroughly tested before being deployed.

8. PARTICIPANT CONCLUSIONS: Public sector outcome measures are driven by different factors than for the private sector • Greater integration success by private industry due to clearer lines of accountability • While the private sector is driven by the bottom line, government is driven by the competing needs of many agencies with differing missions and purposes; lines of accountability are therefore more diffuse. • Government is accountable not only to the governor, legislature, and public officials, but to the public as well • Concern that the public believes that the justice system works as it is portrayed on television creates another type of accountability

9. PARTICIPANT CONCLUSION: Performance measures derived from strategic planning are different from those derived from tactical planning • Strategic plans are often written at a a very high level in order to convey the conceptual framework (particularly in state-level plans) • Strategic performance measures often address pre-implementation “process” activities, thus producing a preponderance of “outputs” rather than “outcomes” • Two-step planning process could make logic model/theory of change more straightforward

10. Performance measurement caveats • Most people (including your employees and consultants) can learn to make the measures come out the way they think you want them to, while exerting a minimum of effort in actually improving a process • Always question the measures you’ve defined, keeping in mind that the people applying them could find ways of boosting the measures without really improving anything • Test each measure to determine if it operates as expected. Does it always go one way when things get better and the other when things get worse?

11. Performance measurement musts • Early and often, measure and evaluate progress toward the goals and objectives that have been defined by the governing body • Continually verify that defined measures actually correspond with the achievement of goals and objectives • Resist overreaching : measures of long-term, global outcomes tend to be unreliable. • Performance measurement is complex and requires significant expertise

12. Strategic Goal 3: Identify and recommend cost-effective biometric identification applications Objective 3.1: By September 2004, research, identify, and recommend technological applications that support biometrics for rapid identification. Objective 3.2: By September 2004, research, identify, and evaluate the costs and benefits of biometric identification applications. Outcomes: • Increased knowledge of biometric technologies • Improved cost-effective biometric identification solutions Performance Measures: • Number of research projects on biometric technological solutions completed by September 2004 • Number of research projects on costs and benefits of biometrics completed by September 2004 • Number of research reports presented to the IIJIS Governing Body

13. Revised Performance measures for Strategic Goal #3 Assumed Causal Chain as Inputs, Outputs, and Outcomes Staff work time conducting research on biometric solutions Input Increased knowledge of biometric technologies Output Increased knowledge of cost-effective biometric solutions for courtroom identification of defendants Output Eliminate problem of sentencing wrong individual Short-term Outcome More accurate reporting of court dispositions to criminal history repository Short-term Outcome More complete criminal history records Intermediate Outcome More informed justice decision-making Intermediate Outcome Enhanced Public Safety Final Outcome

14. Why do I have to worry about all this? • Projects have been de-funded to lack of performance measure data • Need to justify capital expenditures • Need for public accountability • It’s part of professional “tool kit”