Tuteurs m étacognitifs : Supporter la métacognition par la reflexion

1. Tuteurs métacognitifs : Supporter la métacognition par la reflexion Roger Nkambou

2. What is a “Cognitive Model”? • A simulation of human thinking & resulting behavior • Usually used to explain or predict data on human behavior • Like error rates or solution time • Usually implemented as a computer program that can behave like humans • Often using AI knowledge representations like semantic nets, frames, schema, production rules

3. What are Cognitive Models used for? • Output of basic research • Explain results of psychology experiments • Guide design of software systems • Have cognitive model “use” the system • Model predicts people’s time & errors(VanLehn) • Redesign system to reduce time or errors • Can derive predictions without full implementation (e.g., Ethan) • As a component in an intelligent system • Player in a game or training simulation • Part of expert system or intelligent tutor

4. What is an “Intelligent Tutoring System” (ITS)? • A kind of educational software • Uses artificial intelligence techniques to • Provide human tutor-like behavior • Be more flexible, diagnostic & adaptive • Write more general code to get more capabilities with less effort • Components of an ITS: • Interface or problem solving environment, domain knowledge, student model, pedagogical (tutoring) knowledge

5. Reflective thinking & tutoring meta-cognition Cognitive Modeling and Intelligent Tutoring Systems Ken Koedinger Vincent Aleven

6. Overview • ACT-R background & declarative transfer • Two studies of tutoring meta-cognition • Future: 3rd generation tutors

7. Different Learning Goals From: e-Learning and the Science of Instruction : Proven Guidelines for Consumers and Designers of Multimedia Learning by Ruth Colvin Clark & Richard E. Mayer, 2002.

8. Corresponding Instructional Approaches

9. ACT-R’s declarative-procedural distinction • Declarative knowledge • Includes facts, procedures that people can describe • Stores inputs of perception & includes visual memory • Procedural knowledge • Performance knowledge, cannot be verbalized • Procedural k “runs on hardware” • Efficient • Declarative k is interpreted by procedural k • Can be flexibly adapted • But requires associated interpretive procedural k

10. Calculus Study in Declarative Transfer chapter of Singley & Anderson • What’s the difference between operator selection & operator application? • What are the four training conditions in the study? What’s the same in all 4? • During test (day 2) the interface is like which training condition? • Is there transfer from operator … • application to selection? • selection to application?

11. Declarative Transfer Summary • Declarative k is basis for transfer b/t different uses of same knowledge • May be short-lived & sometimes overshadowed by extended practice • Need to search for source of analogy • Can be problematic (Gick & Holyoak) • Requires world knowledge & can serve well as a learning & transfer mechanism even as young as 3 yrs old (Brown & Kane)

12. Overview • ACT-R background & declarative transfer • Two studies of tutoring meta-cognition • Future: 3rd generation tutors

13. Meta-Cognition 1: Encourage Active Declarative Processing Through Self-Explanation Aleven, V. & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2)

14. Problem: Shallow knowledge acquisition • Variations on shallow knowledge • Over-general procedural knowledge • right for wrong reason • No declarative k -- cannot explain, transfer • Geometry example • “Looks-equal” production rule • If the goal is to find angle A and it looks equal to angle B and angle B is D degreesThen conclude that angle A is D degrees

15. Example of Shallow Reasoning

16. Hypothesized Solution • Active processing of declarative knowledge of problem-solving principles leads to: • Better detection of relevant features behind correct inference • Provides dual code for enhanced memory • Less error-prone implicit procedural learning • Instructional manipulation: • Ss explain steps using principles & get feedback on explanations

17. Problem solving answers Explanation by reference Explanation Condition

18. Problem Solving Condition

19. SE Study 1 Method • Between subjects comparison: • Problem Solving vs. Explanation • Run in a Geometry class at local HS • Participants • 41 high school geometry students total • 24 Ss provided complete data, pre-test, tutor, & post-test • About 7 hours of instruction • Ss done when they satisfy tutor’s mastery criteria on problem solving skills

20. Hypothesis • Requiring students to explain steps results in deeper understanding: • Less shallow procedural knowledge • More general declarative knowledge • Consequences: • Better reason giving • Near transfer as good or better • Better far transfer

21. Pre/Post Test Items • Problem-solving items • Answer - Finding unknown quantities • Items associated with deeper understanding • Reason - Explain answers by citing geometry rule • Not Enough Info - Transfer items where students are asked to judge if there is enough information to find quantities, and the answer is “No”.

22. Assessing transfer: “Not Enough Info” item

23. Assessing transfer: Incorrect over-generalization

24. SE Study 1 Results .9 .8 .7 .6 Condition Answer Only .5 Reason % Correct .4 .3 .2 .1 0 Not Enough Info Items AnswerItems Reason Items

25. Possible confounding factors in study 1: time & S prior ability Neither difference is statistically significant but ... Hard to rule out alternative explanation: Explanation condition had more time & higher prior ability

26. Self-Explanation Study 2 Motivation • Replicate the results of Study 1, while controlling for time on task

27. SE Study 2 Method • Between subjects comparison: • Problem Solving vs. Explanation • Run in a Geometry class at local HS • Participants • 53 students total • 41 provided complete data • 7 hours of instruction • Time fixed, so all students spent the same time

28. No time differences in Study 2 Differences between conditions cannot be attributed to differences in time on task

29. SE Study 2 Results

30. Different instruction => different kinds of knowledge acquisition • Shallow (over-general procedural) • Right answers for wrong reason, wrong answers when pressed • Procedural • Right answers with correct knowledge • Efficient, fluent, but inflexible • Declarative • Principles interpreted & reflectively applied • Flexible, but slow & may fail in high cognitive load situations

31. Extra Practice in Problem Solving => More Shallow Learning 1 .9 Condition Problem Solving .8 Explanation .7 % Correct .6 .5 .4 .3 Hard to guess items Easy to guess items

32. Problem Solving group jumps to incorrect conclusions Explanation group shows more control, reflects on sufficiency of knowledge Shallow Procedural Knowledge vs. “Frontal” Control Commission errors / total errors

33. Student Performance During Instruction Problem solving group appears better at end of tutoring. But, not better on post-test! Shallow procedural knowledge acquisition => lack of transfer

34. Estimating Acquisition of Different Knowledge Types

35. Predicts Performance on Different Test Items ...

36. Implications • When Ss explain they learn more & learn with greater understanding: • better explanations of answers • better on harder-to-guess test items • better on transfer questions • Possible to achieve benefits of self-explanation with simple manipulation • Future work: system with which students can explain in their own words

37. Meta-Cognition 2: Supporting Error Detection & Self-Correction • PhD student Santosh Mathan

38. Benefits of Immediate Feedback • Supports efficient skill acquisition • Eliminates floundering • LISP Tutor study • Faster learning • Same post-test

39. Criticisms of Immediate Feedback • Qualitative Basis • Human tutors may wait (Merrill, 1995) • But, just because humans do it ... • Empirical basis • Benefits of delayed feedback in motor learning • Schmidt et al., 1988 • Some cognitive studies • Transfer (Lee, 1992) • Retention (Schooler & Anderson, 1985)

40. Recasting Delayed vs. Immediate Feedback Debate • Debate cast in terms of latency • Alternative: What is the “model of desired performance”? • Expert Model • immediate error correction • emphasizes generative skills • Intelligent Novice Model • allows errors, guides students through error detection & correction • emphasizes generative & evaluative skills

41. Domain of study • Cell referencing in Excel spreadsheet programming • “Glass ceiling” in natural spreadsheet use & skill acquisition

42. Expert Feedback

43. Expert Feedback

44. Intelligent Novice Feedback

45. Intelligent Novice Feedback

46. Intelligent Novice Feedback

47. Participants • 48 participants recruited from a temporary employment agency • All had general computer experience • No Excel experience

48. Instruction, transfer & retention testing 90 min Day 1 Declarative Procedural Post Test Pre Test 50 min Procedural Post Test Day 2 . . . . 8 days later 30 min Day 3 Transfer Pre Test Procedural Post Test

49. Kinds of Pre & Post Tests • Prior experience tests • Computer experience questionnaire • Algebra word problems • Excel coding test • Excel concept test • Transfer coding task • More complex with novel demands

50. Results • Students using intelligent novice model tutor significantly outperformed students using expert-model tutor on all measures • Coding • Concepts • Retention • Transfer