NGA Justice Information Sharing Regional Training Achieving Implementation:

1. NGA Justice Information Sharing Regional Training Achieving Implementation: Measuring Progress and Outcomes Performance Measurement Tools for Information Technology Projects Funded by the Bureau of Justice Assistance Grant # 2002LD-BX K002 August 8, 2003 Denise L. Baer, Ph.D. Consultant Center for Society, Law and Justice University of New Orleans

2. CSLJ PROJECT PURPOSES • Support DOJ Accountability • 1993 Government Results & Performance Act (GPRA) • 1996 Information Technology Management Reform Act (ITMRA) • The President’s Management Agenda • OMB’s Program Assessment Rating Tool (PART) • 2. Support DOJ IT Initiative Which Seeks To • Promote implementation of integrated IT systems • Facilitate the exchange of appropriate information • Build consensus on models, approaches, and standards. • 3. Provide Reporting Tools to States and Local Units of Government That • Are easy to use & reduce Federal reporting burdens • Mesh with State and local-level evaluation and policy needs • Support performance-based management & benchmarking

3. PROJECT METHODS • 12 Month Project to Build Tools • Field-driven Approach • Work with Project Leaders/Real Projects – 26 NGA Projects • Build Consensus on What to Measure • Incorporate Existing Data & Measures • Address Field-Driven Criteria and Needs • Identify Meta Data Issues • Project Staff • Dr. Peter Scharf, CSLJ Executive Director • Dr. Denise Baer, Project Consultant • Dr. Michael Geerken, Project Consultant • Dr. Heidi Unter, CSLJ Program Coordinator

4. CSLJ PM PROJECT TOOLS • Business Case for PM • BRIEFING PAPER – Benefits/Case for Performance Measurement • 2. ”Plug & Play” Resources for States • Inventory of Performance Measures • Logic Models for Criminal History • 3. Facilitation Tools • Steps in Facilitating a Logic Mode/Theory of Change Team • Exercise on Developing Performance Measures • Exercise on Identifying a Business Process for Justice Integration • Performance Path Measurement Matrix Tool & Worksheet • 4. Capacity-Building • Definition of Attributes of Justice Performance Measurement • Identification of Analytical and Methodological Issues

5. CUSTOMERS Access and Participation Customer Service Customer Satisfaction Public Trust BUSINESS PROCESS Short, Intermediate & Long-term Outcomes Knowledge Management & Value Added Outcomes ORGANIZATIONAL Enterprise System Change Agency System Change Training Human Resources FINANCIAL Cycle Time Efficiency Cost Benefit & Avoidance Cost Effectiveness INVENTORY OF PERFORMANCE MEASURES MISSION VISION STRATEGY PROJECT IMPLEMENTATION MEASURES IT Project Milestones Project Management IT SYSTEM OUTPUT MEASURES Data & Report Outputs Number Served Usability of System Data Quality Quality Assurance Functional Requirements Compliance Availability of System Resources System Capacity IT SYSTEM CAPACITY MEASURES INPUTS→PROCESS MEASURES→O U T P U T M E A S U R E S→ O U T C O M E M E A S U R E S

6. LESSONS LEARNED • Demand is High for PM Tools • Need for Tools and Training Paramount • Performance Measurement Increasingly Required • Tight Budgets & Competing Demands • 2. Integration Projects Not Where We Thought • Some Legislature Imposed Measures • Existing Measures -- Only Simple Measures • Most are NOT Outcome Oriented • Many Justice Professionals Unfamiliar with PM • 3. Justice Arena Unique Challenges for PM • Lack Advantages of Health Care Field • State/Local Role and Split-Sovereignty

7. PUBLIC PERFORMANCE MEASURES • Six Key Characteristics • Target End Outcomes • Population-Based (i.e., geographic defined, accountable entities) • Drawn from Routinely Collected Data • Track Accountability for Managing as well as Planning and Evaluation • Should be Derived From Core/Priority Strategies • Produces Intended Result When Tied to • Measurement of a Stable, Managed “Business Process”

8. USE LOGIC MODEL APPROACH STATE STRATEGIC PLAN FRAMEWORK INPUTS ACTIVITIES OUTPUTS OUTCOMES INPUTS are your resources, while OUTPUTS are the product of an activity OUTCOMES are not what the program or initiative itself does, but the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities. The STRATEGIC PLAN and THEORY OF CHANGE links these together

9. DEFINING PERFORMANCE MEASURES STRATEGIC GOAL PERFORMANCE GOAL (Objective) MEASURES A broad, general statement describing a desired outcome A target performance level expressed as tangible, measurable objective A particular value or characteristic to measure results. SAMPLE EXERCISE: Identifying Performance Measures

10. DEFINING INTEGRATION OUTCOMES Catch 22 Catch 22 Hard to Calculate Benefits Until AFTER Processes Automated Hard to Prove IT Alone Drove Outcomes ULTIMATE GOAL Enterprise-Wide Justice Integration Information that is: More Accurate More Timely More Complete Less Expensive BETTER JUSTICE OUTCOMES Public Safety Accountability Less Recidivism Reduced Crime MATURE SYSTEMS MORE BENEFITS More Efficient More Effective Better Evaluation Data Better Targeting of Programs & Services AUTOMATED PROCESSES Cross-System Integration Reduced Cycle Time Outcome-Based Performance By Creating To Result in To Produce …but what are the PROCESSES that define integration?

11. META-DATA ISSUE 1 ….From the Perspective of Process Management

12. META-DATA ISSUE 2 …..From the Perspective of Research Metrics IMPLEMENTING MANAGING & MONITORING BEFORE AFTER ? ROI Return on Investment Business Case Cost-Benefit Analysis Program Evaluation Performance Measurement is a Metric that Links All 3

13. META-DATA ISSUE CHALLENGES Integration is a KEY ENABLER PM is Measurement of a PROCESS PM is a Hybrid: (NIST Baldrige Criteria) Performance Measurement …Identifying the PROCESSES that define integration is CRITICAL for PM

14. DEFINING A BUSINESS PROCESS • Start, End and Purpose • Crosses Functional Units/Org Boundaries • Five Levels – ad hoc to optimized • Produces Normative Outcome (value) • Event or Activity Driven • Defined Processes – Defined Inputs & Resources • Subprocess – smaller value chains • Processes are often Invisible Until Mapped or Defined Strategically • Manage Through Measurement • Efficiency Reduces CYCLE TIME • PROCESS REENGINEERING Dramatic Improvements SAMPLE EXERCISE: What is the Business Process for Integration?

15. IS IT THE STAGE OF INTEGRATION ? Information Sharing Spin-Offs Cross-Agency & Inter-Governmental Electronic Document & Critical Information Sharing & Reporting STAGE 2 Virtual Electronic Document Sharing STAGE 1 Pencil & Paper Information Flow STAGE 3 Agency-Wide Virtual Data Warehouse Building Block System STAGE 4 Cross-Agency Justice Integration Business Rules Spin-Offs Cross-Agency & Inter-Governmental Establishment of Rules, Processes And Methods For Sharing Information ULTIMATE GOAL Enterprise- Wide Justice Integration

16. MATURE PROJECTS ARE MORE CAPABLE JUSTICE OUTCOMES ENTERPRISE-WIDE INTEGRATION 2+ SYSTEMS SHARED PRACTICE DISCRETE PROCESS INTEGRATION AS-NEEDED CRITICAL INFORMATION SHARING IT SYSTEM PERFORMANCE LONGTERM INTERMEDIATE MATURE INTEGRATION PROJECTS SHORT-TERM CRITICAL THRESHOLD SYSTEM CAPACITY BUILDING EXPANDING ELECTRONIC ACCESS ABILITY to ACHIEVE LONGTERM OUTCOMES IMMATURE INTEGRATION PROJECTS AD HOC INFORMATION SHARING PROCESS MEASURES PAPER AND PENCIL SYSTEMS

17. IS IT THE EXCHANGE OF INFORMATION ? OPEN CASE FILE COURT DISPOSITION SEAL RECORDS INCIDENT ARREST CUSTODY INCARCERATION 1 C O R R E C T I O N S INVESTIGATION DETENTION 2 PRE-DISPOSITION SUPERVISION 3 PRE-DISPOSITION COURT 4 POST-DISPOSITION COURT 5 POST-DISPOSITION SUPERVISION 6 7 INCARCERATION 8 POST-SUPERVISION VISUAL MODELING OF JIEM INFORMATION EXCHANGE PROCESSES (SEARCH)

18. DEPENDS – ON WHAT TYPE OF PROCESS 3 TYPES – Which is Integration? • A Management Process • 2. A Support Process • 3. A Core Mission Process

19. CORE MISSION PROCESSES Expected Organizational Outcomes Development and Alignment diverge/converge goals, objectives and measures Mission Statement Strategic Goal Vital Few Performance Goals Communication, Collection, Analysis, Use Vital Few Measures Management And Staff Performance Expectations

20. PERFORMANCE MEASUREMENT SYSTEMS • Key Checklist Questions • Is the program aligned with public policy purposes? • Is the program managed to maximize achievement of these purposes and goals? • Are reasonable performance targets and indicators identified that track progress towards achieving these purposes and goals? • Are the inferences about data indicators justified? • Are the performance indicators tracked and reported on a regular basis? • Are the performance data used to revise policy and • program goals and management? SAMPLE TOOL: Draft Checklist for Developing Performance Measures

21. META-DATA TOOL • Importance/Priority of Outcome Area • a. High priority for maximizing public order and safety of persons • or populations. • b. Financially important • c. Demonstrated variation in services and/or potential for improvement • 2. Usefulness in improving Public Safety or Order • a. Based on established scientific or practice recommendations • b. Potentially actionable by the user • c. Meaningful and interpretable to the user • 3.Measure Design • a. Well defined specifications • b. Documented reliability • c. Documented validity • d. Allowance for risk SAMPLE TOOL: Desirable Attributes for Justice Performance Measurement

22. META-DATA DILEMMA FOUR DIFFERENT LANGUAGES CRITICAL TO PERFORMANCE MEASUREMENT • Policy Strategies • Program Strategies • Management Strategies • Accountability Strategies

23. FACILITATION TOOLS • Invest in Performance Measurement • 2. Form a Team That Includes all 4 Languages • Identify the Business Process • Specify the Logic Model using the Theory of Change Method • Decide the Audience / Level of the Measures • Keep in Mind that the Model Should be Plausible, Doable and Testable • Aim for Compound Measures Using Attributes • List These Strategies, Activities, and Outcomes on the Performance Path Matrix Tool and Worksheet • Revise as Appropriate

24. FACILITATION TOOLS • Assumptions & Factors to Consider: • There will be different TYPES of Logic Models (e.g., access vs. capacity-building vs. discrete outcomes) • Within these types, there may be different LOGIC MODEL GOALS. • Each will need to be mapped to Program COMPONENTS, PROCESS MEASURES AND OUTPUTS. • Each Logic Model will have Stages of Time-Sequenced Initial/Short-term, Intermediate and Longterm OUTCOMES. • For each Stage, there will be appropriate EVENTS and MEASURES (Indicators) SAMPLE TOOL: Steps in Facilitating a PM Logic Model/ Theory of Change Team

25. DEVELOPING A THEORY OF CHANGE • Process for Surfacing & • Articulating a Theory of Change • Use Focus Group Format • Start with Longterm Outcomes • Work Backward Towards Initial Activities • Map Required Existing Resources • Reconcile Multiple Theories of Change • Adapted from J.P. Connell, J.L. A.C. Kubish, L.B. Schorr & C.H. Weiss, 1995, “New Approaches to Evaluating Community Initiatives,” Aspen Institute.

26. S O C I A L I N D I C A T O R S ACCOUNTABILITY STRATEGIES System Change Measure Outcome Measure Outcome Measure Outcome Measure Outcome Measure POLICY STRATEGIES Resources Authority Programs Outcome Outcome Outcome Final Outcome CRIMINAL JUSTICE PROGRAM STRATEGIES Start-Up Activities Activities Activities Activities Activities Formative Measure Output Measure Output Measure Output Measure Output Measure MANAGEMENT STRATEGIES W O R K L O A D I N D I C A T O R S SAMPLE TOOL: PERFORMANCE PATH MEASUREMENT MATRIX MODEL

27. PLEASE EVALUATE THE TOOLS Performance Measurement Tools for IT Projects www.cslj.net CONTACT: Dr. Peter Scharf Executive Director Center for Society, Law and Justice University of New Orleans 3330 North Causeway Boulevard, Suite 413 Metairie, LA 70002 (504) 849-8021 Dr. Denise L. Baer Project Lead (301) 493-8996 src_dlbaer@hotmail.com dbaer@gwu.edu