Improving and Assessing Instructional Effectiveness

1. Improving and Assessing Instructional Effectiveness Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Invited Lecture to IUB Medical Sciences Faculty October 14, 2002

2. Overview • Confounding of instructional effectiveness by other factors • Types of knowledge about education • Type 2, 3 and 4 examples • Recommendations Improving and Assessing Instructional Effectiveness -- Ted Frick

3. The Logic of Effectiveness Improving and Assessing Instructional Effectiveness -- Ted Frick

4. Confounding Factors • Learning occurs without intentional instruction. • Learning occurs despite poor instruction. • We’re not the only teachers a student has. • Results of instruction may occur LONG afterwards. Improving and Assessing Instructional Effectiveness -- Ted Frick

5. Conflation of Research Methods and Outcomes • In addition to the confounding problem in determining effectiveness of instruction, there has been a lot of debate in education about inquiry methods – e.g., quantitative vs. qualitative. • Continuing problem of bridging research and practice in education Improving and Assessing Instructional Effectiveness -- Ted Frick

6. Consider this Debate Imagine for a moment three carpenters arguing about which tool is best. • Quanta: "Hammers and nails are clearly superior." • Qualia: "I disagree.  Screwdrivers and screws are much more effective." • Performa: "You're both wrong.  Saws are best for cutting wood." • Quanta: "Who said anything about cutting wood?  I thought we were talking about fastening wood together." • Qualia: "Right. Who needs saws?" • Performa: "I do. I need to cut this board in half." Improving and Assessing Instructional Effectiveness -- Ted Frick

7. Premise The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful. Improving and Assessing Instructional Effectiveness -- Ted Frick

8. Types of Educology: Knowledge about Education • Methodology of Theory Building (Elizabeth Steiner, 1988) (link) • Non-axiological knowledge – what is • Axiological knowledge – pertaining to values: • Instrumental value: good for – what is effective • Intrinsic value: good in itself – what is worthwhile • The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles (Lee Cronbach, et al., 1972) Improving and Assessing Instructional Effectiveness -- Ted Frick

9. Six Types of Knowledge (Frick) See Frick: R690 Syllabus for further details Improving and Assessing Instructional Effectiveness -- Ted Frick

10. Six Types of Knowledge (Frick) Example: Type 1 Knowledge Claim ‘Elizabeth Steiner is an educational philosopher.’ Improving and Assessing Instructional Effectiveness -- Ted Frick

11. Six Types of Knowledge (Frick) Example:Type 2 Knowledge Claim ‘Students who spend more time engaged successfully in arithmetic and reading tasks score more highly on achievement tests in those content areas.’ (Fisher, et al., 1976) Improving and Assessing Instructional Effectiveness -- Ted Frick

12. Six Types of Knowledge (Frick) Example:Type 3 Knowledge Claim ‘The DISTAR program for teaching arithmetic and reading to elementary students works.’ Improving and Assessing Instructional Effectiveness -- Ted Frick

13. Six Types of Knowledge (Frick) Example:Type 4 Knowledge Claim ‘Instruction is most effective when it is problem-based, activates prior learner knowledge, demonstrates what is to be learned, provides opportunities for guided practice, and encourages integration with everyday life.’ (Merrill, 2001) Improving and Assessing Instructional Effectiveness -- Ted Frick

14. Six Types of Knowledge (Frick) Example:Type 5 Knowledge Claim ‘The corporal punishment policy in Houston schools is a bad policy.’ Improving and Assessing Instructional Effectiveness -- Ted Frick

15. Six Types of Knowledge (Frick) Example:Type 6 Knowledge Claim ‘Teachers and students should respect each other.’ Improving and Assessing Instructional Effectiveness -- Ted Frick

16. Do not confuse ends and means The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful. • Outcome of research is knowledge. • Inquiry methods are a means to that end. • Criteria for evaluating adequacy of research methodology are NOT the same for each knowledge type – e.g., • criteria for statistical inference to a broad population do not apply to Types 1, 3, and 5; • effectiveness not of concern in Types 1, 2, 5, and 6, but main concern in 3 and 4. Improving and Assessing Instructional Effectiveness -- Ted Frick

17. For today… Focus on a few examples of Types 2, 3 and 4 knowledge of education • Type 2: Academic learning time research • Type 3 outcomes: Program/product evaluation – Kirkpatrick’s 4 levels (1959) • Type 4 process: Effective Web Instruction – Frick & Boling (2002) • Type 4: Principles of 5-Star Instruction – David Merrill (2001) Improving and Assessing Instructional Effectiveness -- Ted Frick

18. Six Types of Knowledge (Frick) Type 2 Example: Relationship of academic learning time and student achievement Improving and Assessing Instructional Effectiveness -- Ted Frick

19. Type 2: Academic Learning Time ALT is positively correlated with academic achievement in the same content areas (Fisher, et al., 1976; Rieth & Frick, 1983; Berliner; 1985). Improving and Assessing Instructional Effectiveness -- Ted Frick

20. Six Types of Knowledge (Frick) Type 3: Evaluation of a particular instructional program or product Improving and Assessing Instructional Effectiveness -- Ted Frick

21. Type 3: Program/Product Evaluation – Kirkpatrick’s Levels Techniques for Evaluating Training Programs -- Donald Kirkpatrick (1959) • Reaction (satisfaction) • Learning (achievement) • Behavior (transfer to real context) • Results (impact on organization/context) More on Kirkpatrick’s levels from Encyclopedia of Educational Technology Improving and Assessing Instructional Effectiveness -- Ted Frick

22. Six Types of Knowledge (Frick) Type 4 Example: how to create effective instructional products – design theory Improving and Assessing Instructional Effectiveness -- Ted Frick

23. Type 4: Effective Web Instruction: An Inquiry-Based Process – Frick & Boling More Improving and Assessing Instructional Effectiveness -- Ted Frick

24. Type 4: First Principles of Instruction – David Merrill Integration Activation Problem Application Demonstration Improving and Assessing Instructional Effectiveness -- Ted Frick

25. Type 4: First Principles of Instruction (cont’d) “Learning is facilitated when: • Learners are engaged in solving real-world problems. • Existing knowledge is activated as a foundation for new knowledge. • New knowledge is demonstrated to the learner. • New knowledge is applied by the learner. • New knowledge is integrated into the learner’s world.” (Merrill, 2001, p. 2) Does your instruction rate 5 stars? A rating scale Improving and Assessing Instructional Effectiveness -- Ted Frick

26. Six Types of Knowledge (Frick) Type 3: Indiana University physician education program Improving and Assessing Instructional Effectiveness -- Ted Frick

27. Recommendations In physician education, focus on Type 3 knowledge: • Apply Merrill’s 5 principles as criteria for design. See video (requires RealPlayer). • Use Kirkpatrick’s 4 levels of evaluation for your programs at IU. Improving and Assessing Instructional Effectiveness -- Ted Frick

28. Questions? • This presentation is available at: http://www.indiana.edu/~tedfrick/improveinstruction.ppt • Contact: frick@indiana.edu Improving and Assessing Instructional Effectiveness -- Ted Frick