Presenter: Irwin Goverman Partner, Business IT Strategy and CIO Services

1. Perspectives on IT Performance Management – Stewardship of the IT Investment and FunctionJune 29, 2004 CIO advisory SERVICES = + CIO Services Presenter: Irwin Goverman Partner, Business IT Strategy and CIO Services

2. Today’s Objectives • Discuss Changes and Challenges in Measuring and Proving IT Performance • Discuss Special Challenges Facing the CIO • Discuss Some Approaches to Performance Measurement • Identify Relevant Tools and Methods

3. Remember the “Good Old Days”? Performance Measurement Used to Have a Different Meaning

4. What is Performance? What is Performance? Growth SUCCESS EFFECTIVENESS ACHIEVEMENT Performance Improvement PRODUCTIVITY BONUS LEADER HEALTHY EFFICIENCY Many words and concepts are associated with the concept of Performance.

5. What is Performance Management? What is Performance? • The fulfillment of a claim, promise or request. • Webster’s • PERFORMANCE: A condition or results that we can achieve • through actions over which we do have control. • OUTCOME: A desired state that we wish to achieve, but over • which we do not have direct control. • Hammer & Company

6. What is Performance Management? A Definition of Performance Measurement Performance Measurement is theprocess of collecting, analyzing, evaluating, and communicating informationrelative to an organization's performance and results. Effective Performance Measurement provides information for the planning and controlof organizations, and is a means of ensuring that the direction, efforts and results of an organization are in alignment.

7. What is Performance Management? BUSINESS PLANNING S T R A T E G I C D I R E C T I O N INFORMATION FOR DECISION MAKING AND TAKING ACTION PERFORMANCE MANAGEMENT • Activity Based Management • Priority Setting • Impact Analysis PERFORMANCE MEASUREMENT Performance Management is the integration of the results of performance measurement into management processes so that the results can influence the decisions made by the organization.

8. Why Create an I.S. Scorecard? Why Worry About Performance Measurement? • To demonstrate the value added by the I.S. Organization • To guide the creation of I.S. strategic plans which link into operational plans • To establish a balanced set of measures to determine I.S. effectiveness • To communicate and motivate I.S. performance in key areas • To establish a framework for I.S. Management reporting • To justify getting funding for: • New projects • “LODO” – Lights On, Doors Open • Infrastructure improvments • Raise the average tenure of CIOs

9. Public Sector Has Special Challenges Public Servants have to balance multiple stakeholder demands • Statutory obligations • Electoral commitments • Avoiding embarrassments • Demonstrate achievements • “Doing more with less” • Sustainable Economy • Quality of Life • Safety Net Taxpayers/Voters Legislature • Timely and responsive service • Accurate communications • Accessibility • Equity and dignity • Job security and compensation • Workload and morale • Satisfaction • Values Customers Employees

10. The Four Phases... Performance Management is a lot more than the selection of measures… I II III IV What is Performance? Measuring Performance Optimizing Performance Monitoring and Managing Performance Typically, IT organizations progress through fourphases towards performance management...

11. The Four Phases... The four phases are common to all organizations and represent the journey from awareness and performance measurement to the management of performance … I II III IV What is Performance? Measuring Performance Optimizing Performance Monitoring and Managing Performance ... most are caught either in Phase I or in Phase II

12. The Bad Part of Being Stuck... • Frustration over measures that are hard to interpret and use • Unreasonable amount of effort expended in data collection • Long delays between events and availability of measurements • Little correlation between business goals and IT performance measures • Strong dissatisfaction with high cost of measurement and reviews • Heavy emphasis on financial and volume measures unrelated to operational realities • Inordinate amount of terminological profusion, conceptual complexity and variations across companies • Too much measurement, too little used, and too late For example, being caught in Phase II for too long will most likely result in a cynical attitude towards the topic within the organization…

13. A Potential Solution – The Balanced Scorecard • Known as the Kaplan Framework. At the source of the Performance Measurement trend in the last few years. • Approach developed by Robert Kaplan and David Norton in the United States. • Started as an article in the Harvard Business Review back in 1992. • Book published in 1996 - “Translating Strategy into Action - The Balanced Scorecard”. • “Sold” as an alternative to the traditional financial accounting model which focuses only on financial performance measures. • The Balanced Scorecard: • complements financial measures of past performance with measures of the drivers of future performance • Objectives and measures of the scorecard are derived from the organization’s vision and strategy.

14. Balanced Scorecard The Balanced Scorecard Model has four quadrants linked to strategy and vision. FINANCIAL “To succeed financially, how should we appear to our shareholders?” CUSTOMER INTERNAL BUSINESS PROCESS VISION AND STRATEGY “To achieve our vision, how should we appear to our customers?” “To satisfy our shareholders and customers, what business processes must we excel at?” LEARNING AND GROWTH “To achieve our vision, how will we sustain our ability to change and improve?”

15. Balanced Scorecard The Balanced Scorecard performance measures are linked to strategy and vision. Targets Initiatives Objectives Measures EXAMPLE: CUSTOMER

16. Balanced Scorecard It Also Provides A Way To Facilitate “Taking the Journey Together” CLARIFYING AND TRANSLATING THE VISION AND STRATEGY • Clarifying the Vision • Gaining Consensus COMMUNCATING AND LINKING STRATEGIC FEEDBACK AND LEARNING • Articulating the shared vision • Supplying strategic feedback • Facilitating strategy review and learning • Communicating and Educating • Setting goals • Linking rewards to performance measures PLANNING AND TARGET SETTING • Setting Targets • Aligning strategic initiatives • Allocating resources • Establishing milestones

17. What is The Goal of the Balanced Scorecard? It’s a Way of Running IT like a “Business within the Business” Strategy Marketplace Operations Business People/Capabilities Effective IT that Supports the Business IT Strategy Applications and Services Technology, Architecture & Infrastructure Information Technology People & Process

18. Alignment is a key component of the Scorecard Knowing What Kind of Value You Are Being Asked For is Important Strategic Partner BusinessPartner Outward to Business Are the views consistent? Orientation Commodity ServiceProvider ITEntrepreneur Inward to IT Innovation Fast Follower Leading Edge

19. Commodity Service Provider Strategic Partner IT is for business growth Budgets are driven by business strategy IT is inseparable from the business IT is seen as an investment to manage IT managers are business problem solvers • IT is for efficiency • Budgets are driven by external benchmarks • IT is separable from the business • IT is seen as an expense to control • IT managers are technical experts The “Customers” Views of I.T. Differing Value Across the Continuum

20. A Strategy B Process C D Roles Tools and Technology E Performance Measures IT View of Itself – It Helps to Know “Where You Are” Stage I Nonexistent Stage II Developed Stage III Communicated Stage IV Integrated Stage V Monitored D I M E N S I O N S

21. Cost of I.S. Services Inputs Outputs Value Concept of Value Beauty Will Always Be in the Eye of the Beholder

22. How is Value Measured? • Operational Measures (Dinosaur focus) • Financial Measures (traditional focus – cost side only) • Information Economics (traditional focus – cost per unit) • Economic Value Added (EVA) (contribution to business metrics) • Balanced Scorecard (overall approach)

23. Aligning the Scorecard

24. Alignment Leads to Acceptance of Measures • Agree on what is important • Agree on the role of IS in what’s important • Agree on the objectives and measurements • Agree on how things will be measured • Develop a feedback and refinement mechanism

25. Financial Objectives Objectives Objectives Objectives Objectives Measures Measures Measures Measures Measures Initiatives Initiatives Initiatives Initiatives Initiatives “To succeed financially, how should we appear to our shareholders?” Targets Targets Targets Targets Targets Customer Internal Business Process “To achieve our vision, how should we appear to our customers?” “To satisfy our shareholders and customers, what business processes must we excel at?” Learning and Growth “To achieve our vision, how will we sustain our ability to change and improve?” Traditional Balanced Scorecard So, Now to the IT Balanced “Scorecard” Vision and Strategy

26. Select/ • Evaluate • I.S. Mgmt • Reporting • Customize • Accountability • Develop • Key Measures • I.S. • Alignment • Evolve • Evaluate Basic Approach

27. Stakeholder Driven Potential Performance Measures Performance measures can also be established by looking at each stakeholder’s needs through the reporting tools. Stakeholders Reporting Tools Subjects of Interest Sample measures Client and Customers • Client satisfaction • System availability • Time for services • Reliability of services • Performance Reports • Service Level Agreements • Service delivery • Client satisfaction Senior Managers • Performance against budget • Cost per unit output • Volume of work • Staff satisfaction • Issues • Management Dashboard • IT efficiency • Issue identification • Financial IT Managers • IT Balanced Scorecard • Performance Reports • IT efficiency • Financial • Staff • Process management • Work per unit labor • Performance against budget • Staff morale • Skills gap • vlume of work IT Staff • Performance Reports • IT efficiency • Service delivery • Client satisfaction • System availability • Time for services • Reliability of services

28. Model Driven Potential Performance Measures Performance measures can also be established by using the functions of an IT Service Delivery Model, deriving potential indicators and measures from IT functions. IT Service Delivery Model

29. Performance Measures Measures can be developed in a brain storm session using a variety of creativity techniques such as starting with a list of thoughts. Some examples: • Taxpayer satisfaction • Employer satisfaction • Client satisfaction against benchmarks • Program partner satisfaction ratings • State/federal recognition • Agency progression milestones • Public fairness perception • Pubic awareness • Publication understandability • Readability ratings of publications • Complaints and problem resolution • Objections and appeal counts • Accessibility • Calls answered/abandoned • “One-stop shopping” availability • Speed of refund • Internal client satisfaction • Employee turnover • Employee overtime hours • Employee retention • Employee satisfaction survey scores • Union relations • Employee communications • Staffing speed • Training hours • Internal support hours • Management as a % of total staff • Processing times • Processing error rates • Downtime • Perceived support tool adequacy • Service mix • Revenue earned / dollar spent • Savings/cost reductions • Project milestone status • % electronic returns

30. Documenting the Measures The selected performance measures are then documented using a template. A measurement template, containing the following items, is used for each of the core measures: • name of indicator • operating definition • detailed formula (as required) • reference to goal, objective or deliverable which the measure supports • significance (implication if the indicator increases or decreases) • data source • data supplier/owner (contact name(s)) • frequency of reporting • target/standard • links to other measures • issues

31. Measure Template A selected performance measures is documented using the template. The score assigned by the client representation on a scale of 1 to 5 in response to the question: “How satisfied are you with the IT service provided your organization by BRMS?” Overall Client Satisfaction The number is assigned as follows: 1 - Highly Dissatisfied, 2 - Somewhat Dissatisfied, 3 - Neither Satisfied nor Dissatisfied, 4 - Satisfied, 5 - Highly Satisfied. The resulting metric is an average of all returns The measures is directly related to the strategic objective of client satisfaction The decrease of the score indicates that clients are less satisfied in the services provided them by BRMS Producedquarterly The measure is derived from returns of a quarterly survey of client satisfaction The measure is produced by the Office of Integration of BRMS who are responsible for conducting the quarterly client satisfaction survey The target for the measure is a score of 4.0, a score below 3.5 triggers investigation by BRMS management The measure is provided at the same time as six other client satisfaction measures from the survey including: help desk satisfaction, IT staff satisfaction, knowledge of IT initiatives, perceived value from IT Issues: Any individual score below 3 is examined for rationale and remediation. Scores of 5 are also examined to identify effective measures and pass on congratulations of staff involved.

32. Proposed Measures - Volumetric The following set of measures are recommended for tracking work volumes. Measure Type Frequency Definition Manual Business Transactions Drivers License Renewal Monthly Number of renewals processed (all) Vehicle Renewal Monthly Number of renewals processed (all) Voter Registrations Monthly Number of renewals processed within branches Appeals Scheduled Monthly Number of appeals schedule Alternative Channel Transactions Web Site Hits Monthly Number of hits recorded on home page Web Vehicle Renewal Monthly Number of renewals processed on the web Mail Renewals Monthly Number of renewals processed by the Renewal by Mail unit of BDVR IVR Requests Monthly Number of calls received by all call-in centers and handled by IVR equipment Office Automation Transactions E-Mails Monthly Number of e-mails handled by departmental mail servers Volume of files mailed Monthly Volume of mail transactions in Gigabytes GroupWise Schedules Monthly Number of scheduling requests managed by GroupWise

33. Proposed Measures - Volumetric (cont’d) Measure Frequency Type Definition IT Process Transactions Help Desk Calls Monthly Number of calls logged to the help desk Service Requests Monthly Number of service requests logged Program Modifications Monthly Number of programs modified and recorded in change management systems Projects Managed Monthly Number of programs currently managed as of last day of month Staff Transactions New Hires Monthly Number of new hires Departures Monthly Number of staff departing for all reasons Reprimands Monthly Number of staff reprimanded or receiving disciplinary measures Commendations/Awards Monthly Number of staff receiving commendation or rewards Financial Expenditure Monthly All monies spent during the month Capital Acquisitions Quarterly Volume and value of capital computer acquisitions during the quarter Contracts Quarterly Volume and value of contracts entered into that quarter Transfers outside "client" Quarterly Value of any IT funds transfers outside of the Department either funds in or funds out

34. Proposed Measures - Outcomes Measure Type Frequency Definition Service Availability Mainframe Uptime Monthly Percentage of working hours mainframe was available Network Outages Monthly Number of branch hours of network outage (number of branches affected * duration) Branch Application Outages Monthly Number of branch (B-38) applications taken out of service unscheduled for fixing Branch Equipment Outage Monthly Number of replacements of branch equipment requested Service Effectiveness Unscheduled Batch Runs Monthly Number of batch updates performed due to urgent need unmet by current schedule Batch Re-runs Monthly Number of batches re-run due to processing error Client Delays Monthly Number of times a client is told to wait that are attributed to IT failing IT Staff Effectiveness Client Staff Satisfaction Quarterly Indication from survey of client satisfaction with IT staff capabilities Staff Complaints Monthly Number of complaints from clients regarding staff effectiveness

35. Proposed Measures - IT Processes The following set of measures are recommended for managing IT processes. Measure Frequency Definition Type Technology Compliance to Architecture Monthly Percentage of infrastructure compliant to evolving standards (estimate with explanation for changes) Number of Protocols Monthly Number of network protocols supported throughout "client" Number of Operating Systems Monthly Number of operating variants supported (e.g., HP-UNIX, Solaris, NT, etc.) Number of Users of Old Software Versions Monthly Number of staff known to be using a superceded version of software such as WordPerfect 5.1 Unsupported Workstations Monthly Number of workstations that have passed their supported use (e.g., B-38s, 486 PCs) Management Late, Over-budget Projects Monthly Number of projects behind schedule on time or cost Documentation Monthly Percentage of documentation for existing programs (estimate with explanation for changes) Approvals Monthly Number of projects that have a phase awaiting approval either from BRMS managers or clients Processes Time to Respond - Service Request Monthly Average time to acknowledge a service request with a considered response Time to Respond - help Desk Monthly Average time to acknowledge a help desk call with a considered response Time to Resolve Help Desk call Monthly Average time to address a help desk call to the client’s satisfaction Help Desk Survey Response Quarterly Percentage of help desk callers who fill in a satisfaction survey Help Desk Call Tracking Monthly Percentage of Help Desk calls formally tracked (estimate)

36. Linking the Scorecard Back to Other Measures The measures developed must support four tools: IT Balanced Scorecard, Management Dashboard, Performance Reports and Service Level Agreements. The SLAs help to define the measures required to support the Performance Reports. measures from both the Performance Reports and IT Balanced Scorecard are used for the Management Dashboard. • IT Balanced Scorecard: provide indicators of five types: • financial • process • organization • technology • client service Performance Monitoring Service Level Agreements: defined by end state type indicators - the result of IT Processes. SLAs require measures on service delivery, client satisfaction Management Dashboard: provides overview,high level measures that will be a rollup of those in Performance Reports and the IT Balanced Scorecard Performance Reports including Customer Surveys: requires process efficiency indicators and client service indicators. SLA will define minimum set of indicators

37. STRATEGIC INVESTMENTS OPERATIONAL INVESTMENTS INNOVATION GROWTH Strategy Business Marketplace Operations SHAREHOLDER VALUE People/Capabilities Initiatives MAINTENANCE PRODUCTIVITY Solutions (Applications & Services) Technology, People & Infrastructure IT Results and Returned Benefits HealthPlan Value Map Summary - Process and Deliverables Current Initiatives Are We Tied to the Business Priorities? Statement of Value and Benefits IT Capacity and Mix Allocation of Value by Business Area and Priority Original Governance Decisions and Benefit Statements Where is the Value Delivered? Analysis of Value (e.g. strategic vs. financial, mandated vs. discretionary Interdependencies

38. Appendix – Value Map . The foundation of the Value Map is “Shareholder Value”, the primary components of which include Revenue, Operating Margin and Asset efficiency At the lowest level, the Value Map illustrates the business levers, or discrete activities, that can help clients meet their ROI objectives The Value Map breaks down into high and low level business objectives The Value Map includes a matrix to explore how business activities can potentially impact operating drivers, both positively and negatively

39. Questions