Effective Training: Strategies, Systems and Practices, 2 nd Edition

1. Effective Training: Strategies, Systems and Practices, 2nd Edition Chapter Eight Evaluation of Training

2. Session Overview • Define evaluation • Describe why many organizations don’t evaluate training • Explain who is interested in evaluation data • Describe the various levels of training evaluation Chapter 8

3. Evaluation Defined • Evaluation: judgment of training programs worth or value • Formative Evaluation (process) examines how the training was designed, developed, and carried out • Summative Evaluation (outcome) determines if training accomplished its objectives Chapter 8

4. Resistance to Training Evaluation A few reasons why training managers don’t evaluate training: • Nothing to evaluate • No one really cares • Evaluation is a threat to my job Chapter 8

5. Types of Evaluation Data The following are types of evaluation Data that should be collected: • Reaction Questionnaire • Learning • Job Behavior • Organizational Results Chapter 8

6. Who Is Interested Chapter 8

7. Kirkpatrick Level 1: Reaction Questionnaire Reaction Questionnaires are: “Used to determine what the trainees thought about the training.”(p. 353) Affective Questionnaire: - measures general feelings about training Utility Questionnaire: - measures the trainee's belief about the value of the training Chapter 8

8. Reaction Questionnaire: Types of Questions The following is obtained from a reaction questionnaire: • Facilities issues • Trainer issues • Perceived value of training • Relevance of training (example: pp. 355-356) Chapter 8

9. Timing of Reaction Questionnaire The following times can be used to administer a reaction questionnaire? • During training (after each segment) • Immediately following training • One or two months after training Chapter 8

10. 1.  Determine what you want to find out (consider training objectives). 2.  Develop a written set of questions to obtain the information. 3.  Develop a scale to quantify respondents’ data. 4.  Make forms anonymous so participants will feel free to respond honestly. 5.  Ask for information useful in determining differences in reactions by subgroups (e.g., young vs. old; minority vs. non-minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment. Steps in Developing a Reaction Questionnaire (1 of 2) Chapter 8

11. Steps in Developing a Reaction Questionnaire (2 of 2) 6. Allow space for “Additional Comments” in order to allow participants the opportunity to mention things you might not have considered. 7. Decide the best time to give the questionnaire to get the information you want. a. If right after training, have someone other than the instructor administer and pick up the information. b. If some time later, develop a mechanism for obtaining a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time). Chapter 8

12. Kirkpatrick Level 2: Learning Assessing Learning: • Declarative knowledge – paper/pencil multiple choice test • Procedural knowledge – paired comparison method (p. 359; 8-4) • Determining relationships among a number of concepts • Strategic knowledge – Probed Protocol Analysis • Define a problem • Explain step-by-step how to solve problem Chapter 8

13. Kirkpatrick Level 3: Job Behavior Job Behavior: “Measuring if training actually transferred to the job” Chapter 8

14. Kirkpatrick Level 3: Job Behavior Data collection approaches: • Behavior data • Interviews • Direct observation • Archival records of performance • Scripted situation - a process for assessing transfer of training to the job by providing rater with a number of responses to rate trainee (p. 363) • Attitudes – attitudinal surveys Chapter 8

15. Kirkpatrick Level 4: Organizational Results Organizational Results: “This measurement would be used to determine any change after training was complete” (p. 364) • Did grievances go down? • Did scrap rates diminish? Chapter 8

16. Kirkpatrick Level 4: Organizational Results Data collection: • Consistent tracking of organizational performance should take place at intervals before, throughout, & after training. • Scrap rate after training – scrap rate before training = results Chapter 8

17. External environment of the organization: Economy, regulations, suppliers, etc Internal environment of the organization: Policies, procedures, systems Employee performance, KSAs, and needs Organizational Results Job Behavior Transfer of training Motivational forces in the job setting Opportunity to applying training on the job KSAs Learning Trainee readiness for the course Trainee motivation to learn Design, materials, and content Trainer(s) behaviors Reactions Perceived match between trainee Exceptions and what training provided Evaluation: Kirkpatrick Chapter 8