Instructional Design, Transfer of Training & Program Evaluation

1. Instructional Design, Transfer of Training & Program Evaluation Barbara A. Bichelmeyer, Ph.D Associate Professor Department of Instructional Systems Technology Indiana University Middle East Technical University Ankara, Turkey 21 June 2005

2. A sampling of my time in Turkey • Dolmabahce Palace • Taksim Square • Galata Tower • Bogazici University tour • Basilica Cisterns • Haghia Sophia • Blue Mosque • Topkapi Palace • Grand Bazaar & Spice Bazaar • Whirling Dervishes • First National Assembly • Anatolian Civilization Museum • Old Castle view • METU tour

3. What I have learned so far… • Turkey is 6,500 years old • Crossroad of East and West • 36 sultans during the Ottoman Empire • Attaturk revolutionized everything • Turkey has changed constantly to survive and grow • Technology is changing everything here and everywhere • People from different countries are more alike than they are different • Just like countries, people must change in order to survive and grow

4. My Assumptions About You • A wide-range of experiences as educators • Working in educational institutions • Expertise in IDT and related disciplines of education • Various levels of academic experience • You could be giving a lecture to me…

5. 20 years experience:- Teacher- Instructional designer- Performance consultant- Professor- Evaluator Work Settings:- Secondary school- Higher Education - Corporate- Government- Military- Non-profit- Health Care What Experience Do I Have to Offer?

6. What Can I Possibly Tell You? • Technology has changed the world so that the amount of information is now so great that people can never again “know all that is known” • People must develop skills that facilitate intentional change and intentional learning • Educators must teach not only their topic, they must also teach students how to learn • Educators need to know about how peopleexperience the processes of change and growth

7. Purpose of this Session • Identify basic assumptions behind initiatives for human change and transfer of learning, • Identify transfer as the goal of instruction, • Describe elements that impact transfer of learning, • Explore relationship between instruction and change initiatives, • Explain implications for instruction/program evaluation, • Provide IDT professionals with a more complete skill set for conducting and evaluating instructional and change initiatives.

8. Assumption #1:Basic Progression of Human Performance Influences(IC, S/K, M/I, E) Actions Accomplish-ments OrganizationGoals Influences = Programmatic support for organization initiatives Actions = Activities that leads to accomplishments Accomplishments = What individuals produce to support org goals Organization Goals = The value-added of the organization

9. Assumption #2:The Performance Equation P = IC * M/I * S/K * * Env Inherent Capability = individuals’ inherent abilities and dispositions Skills/Knowledge = facilitate individuals’ knowledge of what/how to do Motivation/Incentive = support individuals’ desires to do Environmental supports = provide individuals with resources to do

10. Assumption #3:Basic Progression of Instructional Design Systematic instructional design involves reverse engineering(beginning with the end in mind): Influences(IC, S/K, M/I, E) Actions Accomplish-ments OrganizationGoals 1. What is the Goal we are trying to achieve? 2. What accomplishments must be produced in order to achieve the goal? 3. What behaviors must performers engage in? 4. What influences are required to support performers (IC, S/K, M/I, Env)?

11. Goal and Purpose of Instruction • Instruction is an appropriate intervention if (and only if) a Skill / Knowledge deficit is present in performers • The purpose of instruction is to increase overt and covert capability in learners so they can do things they were never before able to do (as opposed to information, the purpose of which is to reduce uncertainty) • The goal is instruction is to ensure that performers’ new capabilities transfer to the “real world”

12. Influences(S/K, M/I, E) Actions Accomplish-ments BusinessGoals Why does transfer of learning matter? We are interested in transfer of learning because: - it impacts human performance - it impacts effectiveness of organizations

13. Two Types of Learning Transfer 1) Transfer to long-term memory 2) Transfer to “real world” performance • Can’t have #2 if you don’t have #1 • This is why pre-test and post-test are so important

14. Impact of Instructional Design on Transfer Elements that systematically produce transfer of learning: • Goals - what do you want to transfer? • Performance Aids - tools that support transfer • Tests - how you will measure transfer • Application & Feedback - how you will facilitate transfer • Examples - how you will model for transfer • Instruction - how you will prepare learner for transfer • Leader material - how to set up instruction that transfers

15. Impact of Application on Transfer • Transfer to LTM = Quantity of opportunities for application • Transfer to “Real World” = Quality of opportunities for application

16. Major Blocks to Transfer • Lack of consideration of Performance Equation • Instruction that is not designed systematically • Not enough opportunities for application and feedback • Low levels of simulation in application • Minimal variance of problems in application • Forgetting (time between learning and performance) • Lack of E and M/I support in real world

17. Systemic Support for Transfer • Back to Performance equation:IC + S/K + Env + M/I • Plan and Follow up:Before - During - After • Lack of systematic support minimizes possibility of transfer • Instructional designers and program managers must work together for integrated support

18. Kirkpatrick & Evaluation of Training • Level 4: Long-Term Results / ROI • Level 3: OTJ Behavior • Level 2: Participant Learning • Level 1: Participant Reaction

19. Kirkpatrick’s 4 Levels of Evaluation • A sequence of ways to evaluate instructional programs • Each level is important • None of the levels should be bypassed to get to a level that is considered more important • As you move from one level to the next, the process becomes more difficult and time-consuming • Each level provides more valuable information

20. Links between Program Design, Instruction & Program Evaluation Program Evaluation: Program Design: Business Goal Results / ROI Accomplishments Real World Behavior Behaviors Learning Influences Reaction Instruction

21. Where Does Transfer Begin? • Level 4: Long-Term Results / ROI • Level 3: Real World Behavior • Level 2: Participant Learning • Level 1: Participant Reaction (Kirkpatrick’s four levels of training evaluation)

22. What To Know about Instruction 1) Purpose of instruction is to increase human capability that helps to achieve goals of program/organization 2) Instruction is but one influence on human performance 3) Instruction is an investment in human capital – most expensive investment 4) Instruction is only valuable if it transfers to performance context 5) Transfer is dependent on well-designed instruction 6) Transfer requires performance supports after instruction 7) Instructional staff must work with program management to provide appropriate supports for transfer 8) Evaluators should study relationship between instruction and program management

23. Implications for Evaluation • The better the instructional design, the easier it is to do instructional evaluation and program evaluation • Success of instruction is measured in Level 2 • Instruction can never take complete credit for “real world” performance or ROI • This doesn’t mean that we can not evaluate whether our goals have been achieved

24. Seminal Sources • Joseph H. Harless (1993), An Ounce of Analysis • Donald L. Kirkpatrick (1994), Evaluating Training Programs • Robert Mager (1997), The New Mager Six-Pack • Thomas Gilbert (1968), Human Competence

25. Contact Information Barbara A. Bichelmeyer, Ph.D. Associate Professor Department of Instructional Systems Technology Indiana University 201 North Rose Ave., #2226 Bloomington, IN 47405-1006 812-856-8468 bic@indiana.edu