Learning Analytics

1. Learning Analytics

2. How are we doing in leadership training?

3. Training’s ImpactGood News &Bad News

4. First the Good News 1. Top Training 100 companies Training investment = 4% of payroll (Industry Average 2%) 92% Evaluate business impact to Level 4 (Industry Average 11%) (Training magazine, March 2003)

5. Good News (…continued) 2. ASTD 3-year Study (576 Companies) Impact of Training Investment on Total Shareholder Return (TSR) IMPACT ON TOTAL SHAREHOLDER RETURN (TSR) FIRMS IN TOP HALF OF RANKING: - TSR 86% higher than firms in bottom half of ranking - TSR 45% higher than S&P 500 firms

6. Now the Bad News… Most of your training investment is wasted

7. International Perspectives on Learning and PerformanceASTD, Robinson & Robinson 1996 “On average, less than 30 % percent of what people learn (in training) actually gets used on the job.”

8. Transfer of TrainingBroad and Newstrom, 1992 "Most of the investment in organizational training...is wasted because most of the knowledge and skills gained...(well over 80 percent by some estimates) is not fully applied by those employees on the job.”

9. Baldwin & Ford, 1988Reconfirmed by Ford & Weinstein, 1997 “American industries annually spend more than $100 billion on training…not more than 10% of the expenditures actually result in transfer to the job."

10. Why? Did learner understand management’s expectations? Was it the training? Was it the instructor? Lack of support on the job? Lack of incentives to apply new skills?

11. We don’t know what we don’t know. EVALUATION TRENDS Level 1 > 80% Level 2 < 50% Level 3 9 – 17% Level 4 7% Level 5 5-10% 2002 ASTD State of the Industry

12. Why don’t organizations evaluate more? “Its not a lack of interest or importance —but rather a lack of tools and expertise.” The Time is Now Josh Bersin, 2003

13. Training Analytics Part of the trend to Enterprise-wide HR technologies

14. Learning Management Systems • Distribute elearning courseware • Register employees (ILT & elearning) • Track - progress - course completion - scores - employee learning path - employee skill gaps

15. Top Reasons for LMS Purchase 2001 Managing Classroom Logistics 2002 Migration to eLearning

16. Survey, July 2003 9,000 Respondents Why did you purchase your Learning Management System (LMS)? % % % % Learning & Training Innovations (LTI) &- Bersin & Associates

17. Training Analytics

18. Training Dashboard

19. All Courses

20. Leverage TechnologyData Collection • Data is collect evaluation data over the Web or via email • Stakeholders get realtime, online access to meaningful reports • Cost of learning measurement becomes pennies per learner • Time is saved in collection, processing and reporting to allow for diagnostics and continuous improvement • Use baseline measures from the system to determine where more detailed data analysis and learning measurement is needed

21. The Evaluation Flow Data Collection Data Storage Data Processing Data Reporting • Post Event Survey • Follow Up Survey • Manager Survey • Phillips ROI Input • Templates • Paper, web, email • LMS Integration’ • ‘Hosted’ solution • Database creation • Central storage • Data security • Raw data access • Aggregating of data • Filtering of data • Convert to metrics • Standard reports • Push email alerts • Phillips ROI report • Custom query • Custom reports • Analytical tools • Because data collection, processing and reporting is automated, you have more time to analyze the data and turn it into actionable business intelligence to: • Improve your learning programs • 2) Demonstrate value to your stakeholders

22. Level 1 Did they like it? Level 2 Did they learn? Level 3 Do they use it? Level 4 What were the results? Level 5 What’s the ROI? Data can be Collected at all Levels

23. Data Collection Tools • Post Test • Follow up • Lykert Scale • Learner Comments

24. Learning Levels Scorecard Benchmark across the key performance indicators on your evaluation forms that link back to the 5 levels of learning measurement. Compare the actual performance to goals and to internal or external benchmarks so you properly monitor using a balanced scorecard approach.

25. Performance to Goals

26. Slice & Dice View Reaction, Learning, Application, Business Results, ROI By Training Program By Internal Client By Training Provider By Location or Region By Learning Event

27. Drill Down by Course, Vendor, Instructor, Location…

28. Which Vendors Provide Value?

29. Who Passed the Test?

30. Which Customers are Getting the Best Value?

31. Is Training Aligned to Business goals?

32. What Impact did the course have on the job?

33. What do managers think? Managers can provide feedback on how the training is impacting their employees.

34. Learner Comments by Course/Event

35. Learner Comments by Instructor

36. Phillips ROI Tool Set

37. What is ROI on My Training Investment? The scorecard is a management tool to showcase value to key stakeholders. This report shows you each level of learning and the Key Performance Indicators within each level. At level five you actually produce quantitative ROI metrics including: Benefit to Cost Ratio Payback Period ROI%

38. Track Critical Measures • Ensure Regulatory Compliance • Professional Certifications • Safety Training

39. The Real Value Proposition Fact-based, data driven insight into the impact of your learning investments on business goals.

40. “You can’t manage what you don’t measure” Peter Drucker

41. “If you’re not keeping score, it’s just practice.” DPR Construction (Ranked # 60) Training’s Top 100 For More Information: abailey@learning-designs.com www.learning-designs.com (905) 823-3367