Measuring the Productivity of Technical Training Systems

1. Session: TH303 Level: 201 Measuring the Productivity of Technical Training Systems • Establish standards for measuring the effectiveness and efficiency of technical training system; • Determine the productivity of a technical training system; • Ascertain when and how to take corrective action to improve the productivity of technical training systems; and • Generate data to contrast the productivity of two or more technical training systems.

2. Americans Value The Three Ps Performance ProfitabilityProductivity

3. Kirkpatrick’s Four Levels of Evaluation 1. Reaction – Did the participants like the program? 2. Learning – Did the participants learn anything? 3. Behavior – Did the participants behave differently? 4. Results – Did the change in behavior benefit the organization?

4. Profitability Worth – Cost = Profit

5. Net Program Benefits ROI (%) = X 100 Program Costs Return on Investment

6. Do Any of These Techniques Allow You to Determine the Productivity of an Instructional System? Not Really

7. Definitions 1. a. The quality or fact of being productive; capacity to produce; b. Ecol. The rate of production of biomass. c. Econ. The rate of output per unit of input, used esp. in measuring capital growth, and in assessing the effective use of labour, materials, and equipment. Oxford English Dictionary (1989).

8. Output Productivity = Input Example Units Produced Labor Hours Used 1000 250 4 = = Units per hour Productivity Formula

9. Confused?

10. The Chief Aim “The chief aim of instructional design is to improve employee performance to increase organization efficiency and effectiveness. For this reason, instructional designers should be able to define such important terms such as performance, efficiency, and effectiveness.” (p.4) Rothwell, Willaim J. and Kazanas, H.C. (1998)

11. What is Performance? “Performance is perhaps best understood as the achievement of results, the outcomes (ends) to which purposeful activities are directed. It is not not synonymous with behavior…” (p.5) Rothwell, Willaim J. and Kazanas, H.C. (1998)

12. What is a Behavior? A behavior is any observable activity.

13. Behavior vs. Performance A performance is any activity measured against a standard(s). • Distance • Time

14. Efficiency and Effectiveness “Traditionally, two aspects of performance have been considered – efficiency and effectiveness. These two terms have no universally accepted definitions.” (p.5) Rothwell, Willaim J. and Kazanas, H.C. (1998)

15. Efficiency “But efficiency is usually understood to mean the ratio between the resources needed to achieve results (inputs) and the value of results (outputs). Some have said that the central question of efficiency can be posed simply: Are we doing things right? In this question, the phrase “doing things right” means “without unnecessary expenditure of time, money or effort.” (p.5) Rothwell, Willaim J. and Kazanas, H.C. (1998)

16. Effectiveness “Effectiveness, on the other hand, usually means the match between results achieved and those needed or desired. Its central question is this: Are we doing the right things? In this question, the phrase “right things” typically means “what others expect or need from the organization, group, or individual.” (p.5) Rothwell, Willaim J. and Kazanas, H.C. (1998)

17. Things Right and Right Things “Effectiveness is the foundation of success – efficiency is a minimum condition for survival after success has been achieved. Efficiency is concerned with doing things right. Effectiveness is doing the right things.” (p.45) Drucker, Peter F. (1973)

18. Effectiveness Dimension Productivity Model Qualitative Measure Concern for Quality Asks: How Well? Resource + Activity = Results

19. Wright Stuff How did the Wright brothers know when they had built an airplane?

20. Efficiency Dimension Productivity Model Resource + Activity = Results Asks: How Much? Concern for Quantity Quantitative Measure

21. On the Road Again What might have Mr. Ford wanted after building a car that would carry him from Detroit to Lansing using 40 gallons of gas?

22. Effectiveness Dimension Efficiency Dimension Productivity Model Qualitative Measure Concern for Quality Asks: How Well? Resource + Activity = Results Productivity Asks: How Much? Concern for Quantity Quantitative Measure

23. Effectiveness Dimension Efficiency Dimension Productivity Model Qualitative Measure Concern for Quality Asks: How Well? Output Resource + Activity = Results Productivity Input Asks: How Much? Concern for Quantity Quantitative Measure

24. Effectiveness Dimension Efficiency Dimension Productivity Model Curriculum DevelopmentAsks: What should be taught? Resource + Activity = Results Productivity Instructional StrategyAsks: By what means should the curriculum be taught?

25. High Slope of Productivity Effectiveness Low Low Efficiency High Graphing the Data B D A C

26. 100% 95% Effectiveness Average Competency Level Achieved 90% 85% $125 $100 $75 $150 Delivery Cost per Participant Efficiency Graphing the Data Tech. TrainingSystem

27. Factors that Influence the Effectivenessof Technical Training Systems(Curriculum Development) • Needs Assessment • Assessment of Learners • Analysis of the Work Setting • Job, Task, or Content Analysis • Statement of Performance Objectives • Sequence of Performance Objectives • Performance Measurement (test items)

28. Item Analysis Based upon criterion referenced test questions

29. Factors that Influence the Efficiencyof Technical Training Systems(Instructional Strategy) • Instructional Techniques • Design of Instructional Materials • Instructional Resources - Money - Time - Instructional Staff - Required equipment / infrastructure

30. 100% 95% Effectiveness Average Competency Level Achieved 90% 85% $125 $100 $75 $150 Delivery Cost per Participant Efficiency Comparing Two or More Systems Tech. TrainingSystem B Technical TrainingSystem A

31. Where and When to Use theProductivity Model Use it throughout the technical training design process - Analysis - Design - Development - Implementation - Evaluation

32. Where and When to Use theProductivity Model In conjunction with the various levels identified by Kirkpatrick. 1. Reaction – Did the participants like the program? 2. Learning – Did the participants learn anything? 3. Behavior – Did the participants behave differently? 4. Results – Did the change in behavior benefit the organization?

33. Net Program Benefits ROI (%) = X 100 Program Costs Where and When to Use theProductivity Model In conjunction with the Return on Investment calculation.

34. References Davies, Ivor K. (1981). Instructional Techniques. New York: McGraw-Hill Book Company. Drucker, Peter F. (1974). Management Tasks, Responsibilities, Practices. New York: Harper & Row Publishers. Gilbert, Thomas F. (1978). Human Competence. New York: McGraw-Hill Book Company. Kirkpatrick, Donald (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.  Phillips, Jack J. (1997). Handbook of Training Evaluation and Measurement Methods, 3rd Edition. Houston: Gulf Publishing Company. Rothwell, William J. and Kazanas, H. C. (1998). Mastering the Instructional Design Process, 2nd Edition. San Francisco: Jossey-Bass Publishers.

35. Contact Information Philip McGee, Ed.D. Technology and HRD G-01 Tillman Hall Clemson, South Carolina 29634 Tel: (864) 656-3644 E-Mail: pmcgee@clemson.edu