The Kirkpatrick Model learning and behavior change

1. The Kirkpatrick Model learning and behavior change Richard Lambert, Ph.D.

2. Review • Barriers to innovation • Career stage • Stress • Administrative support • Determine the level of evidence needed • Mini-process evaluation – What happened? • Reaction – Did they like it?

3. The Levels • The four levels from Kirkpatrick’s model • Level I – Reaction • How well did the attendees like the training? • Level II – Learning • What concepts, facts, or skills were learned? • Level III – Behavior • What changes in behavior or attitude did the attendees make once back on the job? • Level IV – Results • What were the tangible impacts of the training program on the organization?

4. The Learning Level • Prerequisites for learning • Motivated participants • Content matches a real need • Time on Task in the sessions • Favorable reaction from participants • Previous knowledge and experience • These necessary conditions can help facilitate learning, but are not sufficient to insure that learning has taken place.

5. The Learning Level • The following steps can help determine if the trainees learned anything during the training. • What were the instructional objectives for the session? • Were all of the objectives covered in the training? • Tapes, transcripts, observations.

6. The Learning Level • Were the objectives covered at various levels of the “taxonomy”? • Memory • (recall, summarize, reconstruct) • Analysis • (analyze, compare, contrast, extract the meaning of, criticize, evaluate, infer)

7. The Learning Level • Creativity • (design, come up with, discover, devise, create, imagine, suppose) • Synthesis • (summarize, conceptualize) • Practical Application • (use/utilize, apply, implement, convince, problem solve)

8. The Learning Level • Have the participants experienced the content at the different levels? • Hold staff development to the same standards we set for the instruction of students. • Evaluate whether principles of adult learning were included in the staff development.

9. Evaluation Design Issues • Is a control or comparison group available? • Can random assignment be used? • Can training be delivered in phases? • Delayed treatment group as the control? • Was the primary focus of the training acquisition of knowledge, skill development, or attitude change?

10. The Learning Level • If the focus is on knowledge, can a pre and post test be used that will measure gains in the knowledge areas involved? • Do measures of knowledge already exist for the given content area? Are they tied to the specific instructional objectives of the training?

11. Skill Development • Can performance assessment be built into the training session? • For some tasks there is no substitute for having every participant demonstrate mastery. • Criterion-referenced versus norm-referenced testing.

12. The Learning Level • If attitudes are the focus, can a pre and post attitude survey be used? • For each of these assessment tasks, the phrase “Garbage in – garbage out” applies. • What do I mean?

13. Behavior Change • If the training: • Included meaningful content and on-task behavior • Generated positive reactions from the attendees • And the participants acquired specific knowledge, skills, or attitudinal benefits • There is still no guarantee that the training will result in behavior changes on the job.

14. Behavior Change • First, conduct a mini-context evaluation to answer the following questions: • Do the trainees want to change their work habits? Do they see a need to change? • Has the training changed their attitudes toward changing work behavior?

15. Behavior Change • Do participants have an understanding of their own strengths and weaknesses? • Do they receive meaningful, regular feedback about their performance? Are they encouraged to be reflective? • Does the work environment welcome change? • Value innovation? • Permit discussion of weaknesses?

16. Behavior Change • Is there ongoing support for change in the work environment? • Mentoring • Role models / Demonstration classrooms • Administrative support • The necessary resources • “Train the trainer” models

17. Behavior Change • The mini-context evaluation helps the evaluator determine if that there is a reasonable chance for the training to result in application. • If so, the focus can shift to gathering evidence of behavior change.

18. Observational Measures • Consider the use of multiple informants, each of whom are asked to rate the behavior of the trainee: • The trainee. • The trainee’s peers. • The trainee’s supervisor. • The trainee’s subordinates. • Outside observers. • Ratings or observational information should be collected as pre and post measures.

19. Additional Considerations • Use a control group whenever possible. There may be factors that are independent of the training that are impacting behavior in the organization. • Use of a control group can help indicate what behavioral changes might have reasonably taken place, even without the training.

20. Additional Considerations • Schedule the post data collection after a reasonable time has elapsed since the training was completed. • Determine how much time the trainees need to fully implement the content they learned.

21. Conclusions • This level of detail and specificity is used in industry. • Training can be critical in industrial and institutional applications. • Safety issues in manufacturing plants • Airline industry • Health care • Military

22. Conclusions • We can do a better job evaluating staff development in education. • In general, does staff development in education meet its objectives? • When does it “work” best?