Measuring ROI in the Public Sector

1. Measuring ROI in the Public Sector Jack J. Phillips, Ph.D. March 9, 2004

2. Agenda • Review current status of ROI implementation • Explore the myths and mysteries of ROI • Identify the key steps involved in the ROI methodology • Examine the key issues involved in ROI implementation in the public sector

3. Paradigm Shift for a Results-Based Approach Activity Based No business need for the program Results-Based Approach Program linked to specific business needs

4. Paradigm Shift for a Results-Based Approach Activity Based No assessment of performance issues Results-Based Approach Assessment of performance effectiveness

5. Paradigm Shift for a Results-Based Approach Activity Based No specific measurable objectives Results-Based Approach Specific objectives for behavior and business impact

6. Paradigm Shift for a Results-Based Approach Activity Based No effort to prepare program participants to achieve results Results-Based Approach Results expectation communicated to participants

7. Paradigm Shift for a Results-Based Approach Activity Based No effort to prepare the work environment to support transfer Results-Based Approach Environment prepared to support transfer

8. Paradigm Shift for a Results-Based Approach Activity Based No efforts to build partnerships with key managers Results-Based Approach Partnerships established with key managers and clients

9. Paradigm Shift for a Results-Based Approach Activity Based No measurement of results or benefit-cost analysis Results-Based Approach Measurement of results and benefit-cost analysis

10. Paradigm Shift for a Results-Based Approach Activity Based Planning and reporting is input focused Results-Based Approach Planning and reporting is output focused

11. Key Measurement Issues • Time? • Costs? • Data Collection? • Management Support? • Skills

12. Why ROI? • ROI is the ultimate level of evaluation • ROI is a concept familiar to most managers • ROI has a rich history of application • Many top executives are requiring ROI information The ROI issue cannot be ignored!

13. The ROI Process Generates six types of data • Reaction, Satisfaction, and Planned Action • Learning • Application and Implementation • Business Impact • Return on Investment • Intangible Measures • ….and includes a technique to isolate the effect of the program or solution.

14. ROI by the Numbers • Process refined over a 20-year period • Thousands of impact studies conducted each year • 100 case studies published • 2,000 individuals certified • 15 books developed to support the process • 500 member professional network • ROI Process adopted by hundreds of organizations in 38 countries

15. Payoff • Aligns programs to business needs • Show contributions of selected programs • Earn respect of senior management/ administrators • Build staff morale • Justify/defend budgets

16. Payoff • Improve support for human resources/ learning/development • Enhance design and implementation processes • Identify inefficient programs that need to be redesigned or eliminated • Identify successful programs that can be expanded

17. Associate Relations Programs Career Development Programs Competency Systems Diversity Programs E-Learning Executive Coaching Executive Education Gain Sharing Programs Global Leadership Organization Development Orientation Systems Recruiting Strategies Safety & Health Programs Self-Directed Teams Skill-Based/Knowledge-Based Compensation Technology Implementation Total Quality Management Wellness/Fitness Initiatives ROI Applications

18. Who is Using the ROI Process?Private Sector Organizations • Allstate Insurance • Amazon.com • Apple Computer • AT&T • Bristol-Myers Squibb • Coca Cola • Dell Computers • Deloitte & Touche • Delta Airlines • DHL Worldwide Express • Federal Express • General Motors • Georgia Pacific • Hewlett Packard • Hilton Hotels • Home Depot • Intel • Lockheed Martin • Motorola • NCR • Nextel • Olive Garden Restaurant • PricewaterhouseCoopers • Shell Oil • Singapore Airlines • TD Bank • Verizon Communications • Wachovia Bank

19. Who is Using the ROI Process?Government Organizations • US Department of Defense • US Internal Revenue Service • US National Security Agency • US Department of Veteran Affairs • NASA • State of New York • State of Mississippi • State of Texas • City of Richmond, Virginia • Government of New Zealand • Government of Singapore • Government of Italy • Government of Australia • Provincial Government of Ontario, Canada

20. ROI Quiz

21. Appropriate Interventions • Performance Improvement • Training and Learning Solutions • Organization Development • HR Programs • Change Initiatives • Technology Implementation

22. Public Sector Issues • Absence of revenues and profits • Absence of hard data • Whose ROI? • Government services are essential and, therefore, may not need this level of evaluation • Restricted range of options to correct problems

23. ROI: The Pieces of the Puzzle An Evaluation Framework Case Applications and Practice Implementation Operating Standards and Philosophy A Process Model

24. Evaluation Levels

25. Characteristics of Evaluation Levels Chain of Value of Customer Frequency Difficulty of Impact Information Focus of Use Assessment Satisfaction Lowest Consumer Frequent Easy Learning Application Impact ROI Highest Client Infrequent Difficult Customers Consumers: The customers who are actively involved in the process. Client: The customers who fund, support, and approve the project

26. Linking Assessment with Evaluation Needs Program Assessment Objectives Evaluation Business Impact Business Needs Objectives Impact 4 4 Job Performance Application Application Needs Objectives 3 3 Skills/Knowledge Learning Learning Needs Objectives 2 2 Preferences Satisfaction Reaction Objectives 1 1

27. Matching Evaluation Levels with Objectives

28. THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution Evaluation Planning Data Collection Level 1:Reaction, Satisfaction, and Planned Actions Level 3: Application/ Implementation Develop Objectives of Solution (s) Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data After Solution Implementation Level 2: Learning Level 4: Business Impact

29. Tabulate Costs of Solution Data Analysis Reporting Isolate the Effects Convert Data to Monetary Value Calculate the Return on Investment Generate Impact Study Level 5: ROI Identify Intangible Measures Intangible Measures

30. Collecting Level 3          Level 4      • Follow-Up Surveys • Follow-Up Questionnaires • Observation On the Job • Interviews with Participants • Follow-Up Focus Groups • Program Assignments • Action Planning • Performance Contracting • Program Follow-Up Session • Performance Monitoring

31. Isolating • Use of a control group arrangement • Trend line analysis of performance data • Use of forecasting methods of performance data • Participant’s estimate of program impact (percent) • Supervisor’s estimate of program impact (percent) • Management’s estimate of program impact (percent) • Use of previous studies • Subordinate’s report of other factors • Calculating/Estimating the impact of other factors • Use of customer input

32. Converting • Converting output to contribution - standard value • Converting the cost of quality - standard value • Converting employee’s time • Using historical costs • Using internal and external experts • Using data from external databases • Linking with other measures • Using participants’ estimates • Using supervisors’ and managers’ estimates • Using staff estimates

33. Tabulating • Needs Assessment (Prorated) • Development Costs (Prorated) • Program Materials • Instructor/Facilitator Costs • Facilities Costs • Travel/Lodging/Meals • Participant Salaries and Benefits • Administrative/Overhead Costs • Evaluation Costs

34. Intangible Benefits • Increased Job Satisfaction • Increased Organizational Commitment • Improved Teamwork • Improved Customer Service • Reduced Complaints • Reduced Conflicts • Reduced Stress

35. Return on Investment Benefits / Program Benefits Costs = Ratio Program Costs Net Program Benefits ROI = Program Costs

36. Calculating ROI BCR = = 3.0 ROI = = ???% $240,000 $80,000 Total Benefits - Program Costs Program Costs

37. Calculating ROI BCR = = 3.0 ROI = = 200% $240,000 $80,000 $240,000 - $80,000 $80,000

38. ROI Target Options • Set the value as with other investments, e.g. 15% • Set slightly above other investments, e.g. 25% • Set at break even - 0% • Set at client expectations

39. When properly implemented, high ROI values can be achieved with programs on: • Leadership • Team Building • Management Development • Supervisor Training • Sales Training . . . 100% to 700% ROI is not uncommon

40. ROI Best Practices • Implemented as a process improvement tool not a performance evaluation tool • Impact studies are conducted involving 5-10% of the programs • A variety of data collection methods are used • The effects of learning/development are isolated • Business impact data are converted

41. ROI Best Practices • ROI evaluation targets are developed • The ROI methodology generates a micro level scorecard • Data are being integrated to create a macro scorecard • Implemented for about 3-5% of the budget • Routine ROI forecasting • Used as a tool to strengthen/improve

42. Building a Scorecard • Provides macro-level perspective of success • Serves as a brief report versus detailed study • Shows connection of training’s contribution to business objectives • Integrates various types of data • Demonstrates alignment between programs, strategic objectives, and operating goals

43. Micro Level Scorecard Macro Level Scorecard 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3

44. Seven Categories of Data • Indicators • Reaction and Planned Action • Learning • Application • Business Impact • ROI • Intangibles

45. Scorecard ReportingFirst University

46. Key Implementation Actions • Determine/establish responsibilities • Develop skills/knowledge with ROI • Develop transition/implementation plan • Conduct ROI studies • Prepare/revise evaluation policy/procedures/ guidelines • Train/brief managers on the ROI Process • Communicate progress/results

47. Cost-Saving Approaches • Plan for evaluation early in the process • Build evaluation into the process • Share the responsibilities • for evaluation • Require participants to • conduct major steps • Use short-cut methods for major steps • Use sampling to select the most appropriate programs for ROI analysis

48. Cost-Saving Approaches • Use estimates in the collection and analysis of data • Develop internal capability to implement the ROI process • Streamline the reporting process • Utilize web-based software to reduce time

49. Recommended Targets 100% Level 1 - Reaction 60% Level 2 - Learning 30% Level 3 - Application 10% Level 4 – Business Impact 5% Level 5 - ROI

50. Criteria • Expected life cycle of the program • The importance of the program in meeting the organization’s goals • Cost of the program • Visibility of the program • The size of the target audience • Extent of management interest