How does model tracing work?

1. How does model tracing work? CPI 494 Feb 10 Kurt VanLehn

2. Example tracing • Tutor’s domain knowledge = • an explicit representation of the prescribed solution paths to the problem • Called “example” and “behavior graph” • To check if a student’s step is correct, • Tutor searches graph for matching step (caveats) • When asked what step to hint, • Tutor picks the next step along the (which?) path

3. Model tracing • Tutor domain knowledge = • A procedure for solving problems • Called a “cognitive model” or “Jess production rules” • To check if a student’s step is correct • Tutor searches (how?) for a match among the steps that it would generate next • When asked what step to hint, • Tutor picks one of the steps that it would generate next

4. Key differences • Example tracing tutor has all correct steps generated and stored as data • Model tracing tutor can generate such steps dynamically as needed • If example has branches (multiple solution paths, orders), then model must be a non-deterministic procedure

5. A simple example of a non-deterministic procedure • Do in order: • Sit down. • Do both in either order: • Don left shoe. • Don right shoe. • Do both in either order: • Tie left shoe • Tie right shoe. • Stand up.

6. Using production (if-then) rules to represent non-deterministic procedure • If not sitting, then sit down. • If sitting and left foot exposed, then don left shoe. • If sitting and right foot exposed, then don right shoe. • If left shoe on foot and untied, then tie left shoe. • If right shoe is on foot and untied, then tie right shoe. • If left shoe tied and right shoe tied, then stand up

7. Production rules can have variables that are “bound” when condition matches • If not sitting, then sit down. • If sitting and ?X foot exposed, then don ?X shoe. • If ?X shoe on foot and untied, then tie ?X shoe. • If left shoe tied and right shoe tied, then stand up

8. Can have fancy conditions • If not sitting, then sit down. • If sitting and ?X foot exposed, then don ?X shoe. • If ?X shoe on foot and untied, then tie ?X shoe. • If not (there exist an exposed foot ?foot), andnot(there exists an untied shoe ?shoe),then stand up

9. Why prefer production rules over other non-deterministic programming languages? • If not sitting, then sit down. • If sitting and ?X foot exposed, then don ?X shoe. • If ?X shoe on foot and untied, then tie ?X shoe. • If not (there exist an exposed foot ?foot), andnot(there exists an untied shoe ?shoe),then stand up • Each rule can be learned independently • Conditions express when the rule applies, so may apply even outside this procedure • Can write hints for a rule without referring to other rules • If same rule appears in multiple procedures, can count number of times applied & hence measure competence

10. What do conditions match against? • Working memory, which is a set of facts • E.g., left_shoe_on • A fact can represent • Something on the student’s screen • E.g., numerator of first given fraction is 4 • Goals (intentions: specific to this problem, this time) • E.g., I will convert the first given fraction to have 12 as the denominator • Temporary, mentally held info • E.g., the least common denominator is 12

11. What happens during execution • WM (working memory) starts with representation of student’s initial screen • Loop until Exit action executed: • Collect all rules whose conditions match in WM – called the conflict set • Pick one member of the conflict set • Execute it: With variables bound by the condition, substitute their values into the action side, do the actions thus modifying WM

12. Example of execution 1 WM • “Left shoe untied” • “Left foot exposed” • “Right shoe untied” • “Right foot exposed” Rules that match (Conflict set): • 1 • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

13. Example of execution 2 WM • “Left shoe untied” • “Left foot exposed” • “Right shoe untied” • “Right foot exposed” • “sitting” Rules that match: • 2 with ?X=left • 2 with ?X=right • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

14. Example of execution 3 WM • “Left shoe untied” • “Right shoe untied” • “Right foot exposed” • “sitting” • “left shoe on foot” Rules that match: • 2 with ?X=right • 3 with ?X=left • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

15. Example of execution 4 WM • “Left shoe untied” • “Right shoe untied” • “sitting” • “left shoe on foot” • “right shoe on foot” Rules that match: • 3 with ?X=left • 3 with ?X=right • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

16. Example of execution 5 WM • “Right shoe untied” • “sitting” • “left shoe on foot” • “right shoe on foot” Rules that match: • 3 with ?X=right • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

17. Example of execution 6 WM • “sitting” • “left shoe on foot” • “right shoe on foot” Rules that match: • 4 • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

18. Example of execution 7 WM • “left shoe on foot” • “right shoe on foot” Rules that match: • 1 (opps) • If not(“sitting”), then add “sitting”. • If “sitting” and “?X foot exposed”, then remove “?X foot exposed” & add “?X shoe on foot”. • If “?X shoe on foot” and “?X shoe untied”, then remove “?X shoe untied” • If not (“?X foot exposed”), and not(“?X shoe untied”), then remove “sitting”

19. It like ordinary programming… • Except that must check that all solution paths are desired • Except that can see WM facts

20. How to match student’s step? • Pre-generate the examples, then use example tracing • At any given time, see if any of the rules that can fire produce an action that matches the student’s step • Complexity: Can chain through invisible facts such as goals and temporary info. • Must keep student on a correct solution path

21. How to select a correct, pedagogically appropriate next step if student asks for a hint? • Sequence or prioritize rules so that first one in the conflict set is best • Store hints on that rule • Give first hint of the sequence • Can substute in the values of ?X variables