Evaluating Teacher Training changing classroom practices

1. Evaluating Teacher Training changing classroom practices Richard Lambert, Ph.D.

2. Magic bullets and everyday practice • In educational evaluation we try to determine the value of the magic bullet. • In the end what we really end up doing is investigating the interaction between the magic bullet and the teachers who were asked to change their practices.

3. Magic bullets and everyday practice • We discussed this question as a part of Process Evaluation • Implementation Fidelity • Follow-up technical assistance • Administrative buy-in • Nothing happens without the teacher • What process must take place for magic bullets to make it into everyday classroom practice?

4. Magic bullets and everyday practice • Teachers have to see a need for change • Teachers have to believe the magic bullets will meet the need • Teachers have to be “free” to change • Resources have to be available • Time to plan • Training has to be effective • What is effective training?

5. Effective Training • Targeted to real needs • Engaging / Challenging • Prompts new thinking • Modeling • Training with peers / time to process • Time to experiment / try out methods • Learning takes place / new knowledge • Leads to change in practice

6. Barriers to Change • Administrative support / expectations for implementation • Technical assistance and follow-up • Interactions between the new and innovative strategies from training and what was already in place • Implementation fidelity • Resources for sustainability

7. The Kirkpatrick Model • The first task involved in evaluating a training or professional development program, is to design a process for documenting what actually takes place during the training. • The evaluator needs to conduct a mini-process evaluation about the training itself.

8. The Mini-Process Evaluation • Design an evaluation process to document the following: • Who attended? • What content was delivered? • What mix of activities and teaching techniques did the trainer use?

9. The Mini-Process Evaluation • What were the physical characteristics of the space in which the training took place? • How long was the actual contact time between the attendees and the trainer, considering breaks and time-on-task issues? • How engaged were the attendees?

10. Process Evaluation Strategies • Video or audio taping of the training • The tapes can then be translated into transcripts or other records • The tapes and / or transcripts can be coded • Attendance records • Observers / Checklists • Interviews

11. Next Steps • Evidence for the effectiveness of Training can take many forms • Simple short term satisfaction • Causal inferences about the effects of training • High levels of organizational change and long term impact.

12. Next Steps • The next task is to determine the level or levels of evaluation evidence that the stakeholders are interested in, and that the resources are sufficient to address. • Kirkpatrick’s four levels of evaluation can serve as a guide in this process.

13. The Kirkpatrick Model • Reaction • Learning • Behavior • Results • The goal is to make INFORMED judgments about the merit or value of training.

14. The Levels • The four levels from Kirkpatrick’s model • Level I – Reaction • How well did the attendees like the training? • Level II – Learning • What concepts, facts, or skills were learned? • Level III – Behavior • What changes in behavior or attitude did the attendees make once back on the job? • Level IV – Results • What were the tangible impacts of the training program on the organization?

15. The Levels • Each of these four levels of evaluation is associated with specific evaluation goals and strategies. • With each increase in level comes an increased level of difficulty for the evaluator.

16. The Levels • It may be relatively easy to document process and gauge participant reaction to a training program. • It may be very difficult to establish causal links between a training program and organizational changes.

17. The Levels • A set of specific steps, in the form of questions and activities, will be outlined in the coming weeks to further illustrate the nature of each level. • What evaluation activities / strategies match the levels? • Reaction • Learning • Behavior • Results