Strategic Approach to Connecting Performance and Training Inputs

1. Establishing a strategic approach to determine the measures that reliably connect performance and training inputs Bob MegensBMO Financial Group Institute For LearningSTEP: May 3, 2007

2. Why focus on measuring the impact of learning on performance? BMO has set up an Our Way Model as a way of achieving the goal of a high performance culture. Tracking performance is an essential component of BMO’s enterprise strategy of becoming a high performing culture. Harvard Business Review Special Double issue – July/August 2005 The High Performance Organization BMO Financial Group Our Way Model • Editor Thomas A. Stewart’s • Key Things for a High Performance Organization • Leadership • Teams • Organizational Design • Learning What are the outputs that will be used to valid the achievement of high performance? To what degree of accuracy are we able to measure the contribution of these inputs to the achievement of the outputs being measured?

3. The Conference Board of Canada Learning and Development Outlook 2007 (May), Ch. 6, Learning Evaluation “The lack of detailed evaluation may explain [training, learning and development] professionals’ inability to obtain higher training budgets: they simply do not have the analytical basis to make the business case for increasing budgets. Although many [training, learning and development] professionals expect their budgets to increase, their evaluation efforts suggest that they are not fully engaged in demonstrating the impact of their current training investments. This is likely to reduce spending: as evaluation efforts decrease, budgets might be expected to decrease.”

4. Lexicons and Paradigms: Kirkpatrick or time for a change? High-Impact Learning Measurement, Bersin and Associates, Nov. 2006 While many organizations make a valiant effort to measure specific high-value programs, most organizations do not have a repeatable, high-value measurement process for measuring training impact. We believe that one of the reasons for this problem is the limitations of the existing well-known measurement “models,” primarily the Kirkpatrick Model – which describes four levels for measurement: learner satisfaction, learning, job impact and business impact. A fifth “level” (ROI or return on investment) was added through the work of Jack Phillips4. While Donald Kirkpatrick does not necessarily believe that ROI is the fifth level, it is often recognized as the ultimate measure of training impact. While these models are widely understood and have done much to help training managers understand some of the possibilities of training measurement, we believe that they tend to limit an organization’s thinking and make the measurement process difficult to implement… While we do not want to confuse the industry with yet another model, as a result of this research we found it necessary to develop one of our own.

5. 3 Key Criteria for linking performance and training There were 3 criteria (which stood out from the research) that needed to be in place to determine the impact of learning on performance. • 1. Alignment of learning with enterprise priorities • 2. Conditions that maximize the learning investment • Business performance measurement and reporting systems in place to identify the metrics that are to be used to isolate the business performance impact of learning Systematic approach Concerted monitoring Consistent process

6. Current Approach Consistent collection and reporting on Level 1 evaluation. While there is collection of data at higher levels, evaluation and reporting are less consistent and not systematic. Level 4:- ad hoc collection - late in the process- no management reporting Level 3: - follow up survey- no management reporting- employee survey (Learning Index) reported annually Level 4 Level 3 Level 2: - dependent upon initiative- no management reporting. Level 2 Level 1: - evaluation consistently applied - quarterly reports. Level 1 Approach Amount of activity What is measured and management reporting

7. 2 dimensions for capturing the impact of learning on performance Establish processes that identify opportunities to measure the impact of learning at an enterprise level and consistently capture the impact of learning for specific initiatives and programs • What are the opportunities for the collection of non-survey data? • If survey data is collected, who would provide the most valid data? • Increase Customer Service metrics for learning at enterprise level • - establish a link with Market Research to formalize a process to align with Market Research’s initiative in linking customer survey results to business results • - develop customer service metrics/questions that target specific learning and development outcomes for integration into customer surveys

8. Proposed Approach The proposed approach places more emphasis on identifying robust and appropriate metrics for determining the impact of learning on performance (Level 4) at the beginning of the engagement process , making the agreement to collect these metrics explicit components of learning initiatives. Level 4: (financial, non-financial and survey data)- focus on strategically relevant metrics (no more than 10% of all learning initiatives)- leverage business performance measurement systems and coaching tools to isolate or associate the impact of learning- survey executives and managers to assess the impact of learning on business - establish reporting that feeds into tracking systems Level 4 Level 3 Level 3: (financial, non-financial and survey data)- employee performance metrics - survey executives and managers to assess the impact of learning on employee performance- index of employee survey performance questions - index of performance questions from follow-up evaluations BENCHMARKING Level 2 Level 1 Level 2: dependent upon initiative Level 1: evaluation system reviewed and reported quarterly. Approach Amount of activity What is measured and management reporting

9. 3 approaches for measuring the impact of learning on performance Developing more rigorous metrics to demonstrate the impact of training on performance requires a declaration of priority, sustained by a focused and managed approach. • Track and aggregate (where possible) metrics for the impact of learning as a performance measure for the learning function • Develop the expertise for client engagement, analysis and collection of measurement data within the learning function • develop skills of individual project manager/course owners • establish a centre of competency for evaluation to work with projects/learning initiatives • Leverage performance measurement systems used by the business • learning function cannot create systems to measure performance within the business • must leverage the financial and non-financial data collected by the business • begin at the scoping stage of the project, start with Level 4 metrics and seek agreement with business client(s) on measures used • document measurement decisions into the project charter • Collect metrics that isolate or associate the impact of learning on business performance for relevant strategic initiatives • validate alignment with learning objectives • select initiatives that best demonstrate the business value of learning (up to 10% of all learning initiatives) • focus on sales and service, and leadership and performance management • Monitor the conditions that maximize the impact of learning on business performance by monitoring them 1. Attention and Priority 2. Process and Focus 3. Monitor Conditions

10. Conditions directly affecting the impact of learning on performance Research indicates that specific conditions need to be in place to realize the investment in learning as a sustainable performance lift. Business Conditions Required to Maximize Learning Investment • Leadership Engagement (Funding and Support, e.g. Diagnostic) • Appropriate target audience properly prepared for learning • Follow up Coaching • Conditions for Business Performance • Reward and Recognition • Appropriate Measurement • Hiring Strategies • Business process improvement and technology Potential to maximize learning’s impact on business performance. Monitor to ensure presence. Assumed to be in place. Raised only if they present barriers to performance. • Conditions taken from Phillips, Jack. Return on Investment in Training and Performance Improvement Programs, 2nd Ed, 2003.

11. Metrics that measure business and employee performance Measuring the impact of learning on business performance will leverage business performance measurement and reporting systems. The following table lists metrics at the business and employee performance levels that can be used to isolate the performance impact of learning. These metrics are particularly effective when applied to sales and service, especially in the area of customer service.

12. Deriving verifiable and specificmetrics Research shows that deriving verifiable level 4 metrics to determine the performance impact of learning using specific figures is only viable and recommended about 10% of the time. The following criteria can be used by the learning function and the business sponsors to determine when it is more appropriate to pursue robust data as opposed to a more associative level of evaluation for particular learning initiatives: - Business leader engagement - Highly strategic initiative - Large audience - Greater allocation of resources - Comprehensive and specific business - Appropriate baseline measurements performance measurement systems Level 4 evaluations require greater engagement of the business and dedication of resources over an agreed upon time period. Level 4 Business Engagement Dedication of Resources Level 3 Level 1 and 2 Evaluations will remain in high use due to the relative low dedication of resources and the ease of implementation immediately after the learning is completed Level 2 Level 1 • from Phillips, Jack. Return on Investment in Training and Performance Improvement Programs, 2nd Ed, 2003.

13. Tracking impact on performance Metrics on the impact of learning on business performanceshould be used as an input into tracking of the performance of the learning function.