Lecture 4 Classic Intelligent Learning Tools - continued

1. Lecture 4Classic Intelligent Learning Tools - continued Adaptive Learning Environments

2. 4. Knowledge Modelling:Qualitative Models(Glass Box) Qualitative models

3. Qualitative Models • Concern with reasoning in qualitative terms about the causal structure of world. • Allow reasoning about dynamic processes. • Not necessarily same as cognitive fidelity but does aim for it. Qualitative models

4. QUEST: White and Frederiksen (1986,87) • The domain is electrical circuits. • Internal representation uses causal calculus: basically component-oriented, but incorporates some higher-level concepts guiding evaluation of component states. • Progressions of mental models: model changes as the students’ understanding of the domain progresses • “modelling possible evolutions in students’ reasoning about electrical circuits as they come to understand more and more about circuit behaviour” • Focus on using model progressions: helps students learn: • a. to predict and explain circuit behaviour, and • b. to troubleshoot by locating opens and shorts to ground in series-parallel circuits • The more advanced the student’s understanding, the higher level the model Qualitative models

5. QUEST: a circuit amenable to zero-order qualitative reasoning (White & Frederiksen, 1986) • “In order for the bulb to light, there must be a voltage drop across it. There is a device in parallel with the bulb, the switch. Two devices in parallel have the same voltage across them. Voltage drop is directly proportional to resistance: If there is no resistance, there can be no voltage. Since the switch has no resistance, there is no voltage drop across the switch. Thus, there is no voltage drop across the light, so the light will be off.” Qualitative models

6. De Kleer’s ENVISION Theory, 1983 • Mechanistic Mental Models (= causal/qualitative) • Reasoning about physical devices • Causal model of buzzer = envisionment: • links components with respect to behaviour; • describes device in terms of component states, changes in states and consequences for other components • Model can be run on specific inputs to yield predictions. • Envision theory attempts to provide modelling framework for reproducing causality from structure • Models expert/scientist's knowledge Qualitative models

7. Construction of a causal model for a buzzer: (adapted by Wenger, 1987, from de Kleer and Brown, 1983) Qualitative models

8. De Kleer and Brown, 1984 (figures from Wenger, 1987) Qualitative models

9. Qualitative Process Theory - Forbus, 1984 • Reasoning about processes • Attempts to provide a language for encoding causality as perceived by people (more like “naive physics”) • Qualitative Process Theory: description of a process of heat transfer, (from Wenger, 1987, after Forbus, 1984) • Lot of later work in Qualitative Reasoning followed from these, and also got used as the basis for other tutoring systems Qualitative models

10. Process heat-flow • Individuals: • source: an object, Has-Quantity(source, heat) • destination: an object, Has-Quantity(destination, heat) • path: a Heat-Path,Has-Connection(path,source,destination) • Preconditions • Heat-Aligned(path) • Quantity Conditions • A [temperature (source)] > A [temperature (destination)] • Relations • Let flow-rate be a quantity: A [flow-rate] > ZERO • flow-rateaQ+(temperature(source)-temperature(destination)) • [temperature (source)aQ+ heat(source)] • [temperature (destination)aQ+ heat(destination)] • Influences: • I- (heat(source), A[flow-rate]) • I+ (heat(destination), A[flow-rate]) Qualitative models

11. 5. Towards Cognitive Models:Anderson’s Lisp Tutor Adaptive Learning Environments

12. Anderson’s Lisp Tutor Based on Anderson's ACT theory Cognitive Model: cognitive functions as production rules. - Instruction for skills in problem solving context: declarative knowledge can be converted into useful productions. - Student makes successive approximations to progressively form and refine productions. Knowledge Compilation: Basic learning mechanism in two forms 1. Proceduralization: general piece of knowledge converted into a specific production to apply to a special class of cases 2.Rule Composition: a few rules used in sequence to achieve a goal are collapsed into a single rule combining their effects Adaptive Learning Environments

13. Lisp Tutor, contd Tutors communicates with students in terms of task = designing a computer program • Tutor has 325 rules based on analysis of novice protocols (i.e. correspond to conceptual units novice can remember). • Has 'ideal model’ rules and 'buggy' ones. Advanced rules turned of in early stages; limited to expertise of ideal student at level for lesson. Tutor always relates explanations to current situation and problem solving goals. [not same as GUIDON ones - only trigger explanations, do not embody strategy] Adaptive Learning Environments

14. Student is held close to 'correct path' • problem solver generates all steps the student might take, according to rules • compared to student's step • rule that matches is selected as interpretation of action • if cannot give model-driven interpretation of student's step, suggest best next one from ideal model Every keystroke monitored - intervenes when perceives meaningful error Tutorial rules associated with ideal and buggy rules provide various levels of explanation Note: did not support approach to programming of debugging as student was never allowed to go wrong More recent work at CMU/Pittsburgh, Carnegie Learning, LearnLab developed from this and other early tutors Adaptive Learning Environments

15. 6. Student Modelling:BUGGY, DEBUGGY, IDEBUGGY Adaptive Learning Environments

16. Goals of Student Modelling • Find out what the student knows, believes, can do • Look for evidence that user fails to exploit some knowledge • Look for inconsistent beliefs, differences between student and domain models • If teacher believes students has different beliefs or skills: • make a list (bug catalogue) and match to it • reason about what student would have to believe in order to exhibit behaviour indicating this • Component of an ITS that represents student's current state of knowledge = STUDENT MODEL • Process of inferring the Student Model = DIAGNOSIS Adaptive Learning Environments

17. Classification and Examples • OVERLAY - student's knowledge as subset of expert/system's knowledge of domain • e.g. GUIDON WUSOR II • PREDICTIVE - run models and see which best fits the users input • e.g. BUGGY/DEBUGGY/IDEBUGGY • ANALYTIC - analyse user's input and see which features has and try and account for structure • e.g. WEST Adaptive Learning Environments

18. Predictive Models: BUGGY • Diagnostic skills in addition and subtraction. • 1. BUGGY selects a faulty algorithm • BUGGY: Welcome to Buggy.I have chosen a bug. Here is an example of the bug. • 17 • + 5 • 13 • Now you give me problems to determine the bug. • 2. the user sets some problems • User: 18 43 • + 6 + 79 • 3. BUGGY gives the answers (by faulty algorithm) • BUGGY: 15 23 • 4. user diagnoses the problem • User: Got the bug! Adaptive Learning Environments

19. BUGGY contd. • BUGGY: Please describe the bug. • User: Student adds the number to be carried into the last column worked in. • 5. BUGGY sets test, user predicts answers • BUGGY: Here are some problems to test your theory about the bug.What is: • 21 • + 39 • User: 51 • BUGGY: That’s not the bug I have. Using my bug: • 21 • + 39 • 15 • Try giving me some more examples. Adaptive Learning Environments

20. BUGGY contd. • User: 51 99 68 • + 1707 + 99 + 9 • BUGGY: 21 36 23 • User: Got the bug! • BUGGY: Please describe the bug you think I have. • User: The student is adding all the digits together. • BUGGY: Here are some problems to test your theory about the bug. What is: • 33 1091 8 28 90 • + 99 + 60 + 34 + 70 + 6 • User: 24 17 15 17 15 • BUGGY: Very good. My description of the bug is: • The student always sums up all the digits with no regard to columns. Adaptive Learning Environments

21. Representation • Knowledge of arithmetic procedures and possible flaws encoded as a `procedural net'. • each node represented as single component skill: • eg ‘write digit in the answer position’ • ‘find the next column’ or • ‘get top number of the column’ • links between nodes show how single compound skill decomposed into lower level steps: • eg ‘subtract this column’ • Buggy variants represented explicitly • Network is easily executable Adaptive Learning Environments

22. BUGGY: Drawbacks • Misconception represented as a single node: • can contain any code as its executable part • single out bugs, but can only explain misconception by means of demonstration or canned text attached to node • Representation supposes that: • - user is merely executing some kind of algorithm, in a context-free way • basic misconceptions represented in same way as basic components of the skill • depends on decomposition of skill to level where single bug is a separate procedure • Only procedural knowledge? • System builder has to do all representational work Adaptive Learning Environments

23. DEBUGGY: • Works off-line, tries to do the diagnosis • 1. Accepts a series of problems and answers: • 2. Conducts search through network for flawed version of procedure which would explain the whole input: • 3. All basic misconceptions accounting for any part of data included in initial hypothesis set • - set reduced by eliminating bugs for which there is only a little supporting evidence • 4. All combinations of remaining bugs considered • - combinations which explain more of data than individual components do retained as possibilities • 5. Process repeated until clear winner or until complexity of compound bugs too large Adaptive Learning Environments

24. IDEBUGGY: • Interactive version of DEBUGGY • - works incrementally • at each step new problems can be suggested which might reduce the hypothesis space, or might test for more obscure bugs. • Basic approach influential in student modelling in particular Adaptive Learning Environments

25. 7. Student Modelling:West (Brown and Burton, 1979) Adaptive Learning Environments

26. Adaptive Learning Environments

27. WEST (Brown and Burton 1979) 1. Give student ‘random’ numbers; watches the user play • 2. Get the user and procedural expert's expression Issues: - mathematical skills: addition, use of parentheses - game-specific skills: use of shortcuts, landing on your opponent, strategies such as always aiming for towns - general game skills: learning from opponent, exploring space of strategies permitted by the game. • 3. Update record of strengths with respect to each issue; 4. Look for (indirect) evidence that student lacks issues - form differential model of those not used - determine those issues where student is weak; • - find better moves which illustrate any of issues; • - consider intervening to show a better example. • Intervenes only when good evidence of a weakness The user can ask for a hint Adaptive Learning Environments

28. Issues For each issue, there is a: RECOGNISER checks to see if: • issue features user move, • necessary in that expression, • necessary in the optimal move. [coach has an expert player built into it] EVALUATOR determines if student seems weak in issue, by failing to use it to good effect. [shares blame equally among all issues that might be implicated in the failure to find the best move] WEST decide when to intervene and what to say • controlled by general principles (not explicit) • Adopts a cautious attitude • If user is employing a different strategy from expert player WEST detects this - considers whether another (pre-wired) strategy would give a better fit Adaptive Learning Environments

29. West: general principles • Only give advice on an issue on which the student is clearly weak. • To illustrate an issue, use an example which is very obviously better than what the student did. • After giving advice, let the student take his turn again so he can apply the advice at once - but do not force him to retake his turn. • If the student is about to lose, interrupt and tutor him with moves that will stop him losing. • Never tutor on two consecutive moves. • Do no tutoring until the student has had a chance to discover the game for himself. • Congratulate as well as criticise. • Always have the computer play an optimal game. • If the student asks for help, provide several levels of hints. • If the student is losing consistently, adjust the level of play.[make spinners non-random and give the computer poor numbers]. • Be forgiving abut potentially careless errors, but be explicit about them too. Adaptive Learning Environments

30. References(also see course webpage): • Brown, J.S. and R.R.Burton, (1978) Diagnostic models for procedural bugs in basic mathematical skills, Cognitive Science, 2, pp.155-192 • Brown, J.S. & VanLehn, K. (1980). Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426. • Burton, R.R. (1982) Diagnosing bugs in a simple procedural skill, in (eds.) D.Sleeman and J.S.Brown, Intelligent Tutoring Systems, Academic Press, pp.157-184. • Clancey, W.J. (1983) GUIDON, Journal of Computer Based Instruction, 10:(1+2) 8-15. • Clancey,W. (1986) Qualitative Student Models', in First Annual Review of Computer Science, ACM, pp. 381-450. • Shortliffe, E.H. (1976) Computer-Based Medical Consultations: MYCIN. New York: American Elsevier. • VanLehn,K. (1987) Learning one sub-procedure per lesson, Artificial Intelligence, 31, 1, pp.1-40. • Wenger, E. (1987)Artificial intelligence and tutoring systems: computational and cognitive approaches to the communication of knowledge. San Francisco: Morgan Kaufmann. Adaptive Learning Environments