Author: Fang Wei Advisor: Prof. Blank Department of Computer Science Lehigh University

1. A Student Model for an Intelligent Tutoring System Helping Novices LearnObject Oriented Design Author: Fang Wei Advisor: Prof. Blank Department of Computer Science Lehigh University May 10, 2007

2. Publications Background Research questions Methodology Evaluation Conclusions and future work Presentation Outline

3. Publications • Wei, F. & Blank, G.D. (2007) Atomic Dynamic Bayesian Networks for a Responsive Student Model, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007 • Wei, F. & Blank, G.D. (2006) Student Modeling with Atomic Bayesian Networks, Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, pp. 491-502 • Wei, F., Moritz, S., Parvez, S., & Blank, G.D. (2005) A Student Model for Object-Oriented Design and Programming. The Journal of Computing Sciences in Colleges (CCSC), Vol. 20, pp. 260-273. "Best Paper" Award • Blank, G. D., Parvez, S., Wei, F., Moritz, S (2005) A web-based ITS for object-oriented design. Poster for 12th International Conference on Artificial Intelligence in Education, Workshop of Adaptive Systems for Web-Based Education: Tools and reusability, Amsterdam, The Netherlands, June • Moritz, S., Wei, F., Parvez, S., & Blank, G. D. (2005), From objects-first to design-first with multimedia and intelligent tutoring. The Tenth Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), Monte da Caparica, Portugal, June.

4. Publications Background Research questions Methodology Evaluation Conclusions and future work Presentation Outline

5. Intelligent Tutoring System (ITS) • A computer-based instructional system • has knowledge bases for instructional content and teaching strategies • uses a student’s level of mastery of topics to adapt instruction dynamically • A cost-effective means of one-on-one tutoring to provide novices with individualized attention • Computer Assisted Instruction (CAI) system does not model what a student is learning and cannot adapt to student • CAI provides same instruction, problems and feedback to every student

6. Intelligent Tutoring System • Typically contains three main components: • An expert evaluator that observes a student’s work and identifies errors in his/her solution • A student model that diagnoses gap in student’s knowledge • A pedagogical advisor that provides feedback to student

7. Student Model • Maintains a model of students’ current knowledge state by representing and updating • Provides information for intelligent pedagogical decisions and actions including: • curriculum sequencing • interactive problem solving support • pedagogical tutoring customized to each individual student’s learning state

8. Motivation • Novices in high school and college have much difficulty in object oriented design • ITSs can aid learners with complex problem-solving • DesignFirst-ITS developed to help novices learn object-oriented design (Blank et al. 2005, Moritz & Blank 2005) • Research has shown that an ITS that adapts to accurate student knowledge enhances learning (Corbett et al. 2000) • Corbett’s ITS focuses on adaptive problem sequencing rather than adaptive feedback • Our ITS focuses on adaptive feedback

9. Student Model in Wei & Blank (2006,2007) compared with other BN Student Models

10. Layers of Student Knowledge(Self 1994) • Domain knowledge layer • explain all vocabulary for discussing or solving problems • Reasoning knowledge layer • contain reasoning relationships between propositions in domain knowledge • Monitoring knowledge layer • specify how to solve a problem using reasoning knowledge and domain knowledge • Reflective knowledge layer • specify appropriate strategies students should have in a learning environment

11. Common Problems with Student Models • Do not consider relationship between individual concepts • Do not represent layered knowledge (Self 1994) • Do not simulate students’ knowledge history • Separate the inferred students’ knowledge from closed- and open-ended exercises • Do not consider students’ cognitive strategies including general and domain-specific • Bayesian student models require exponential updating time and hence cannot provide real-time tutoring adaptive to individual student’s learning state

12. Publication Background Researchquestions Methodology Evaluation Conclusions and future work Presentation Outline

13. Research Questions (1 of 2) • Can this student model provide information for pedagogical decisions? • How should this student model represent a student’s current knowledge state and the student’s knowledge structure? • How will the student model track students’ knowledge state over time? Under this research question there are two sub questions: • Would tracking a history of students’ knowledge state be useful for pedagogical decisions? • Can a history be maintained efficiently enough to be responsive in real-time?

14. Research Questions (2 of 2) • How to synthesize information from two different sources, open-ended problem solving (object-oriented class diagram design) and closed-ended exercises (multiple choice quizzes or drag-and-drop exercises)? • What cognitive strategies should the student model consider and how to consider them?

15. Publication Background Research questions Methodology Evaluation Conclusions and future work Presentation Outline

16. Three Layered Architecture • CM recognizes cognitive strategies that a student is using • HM simulates students’ hierarchical knowledge in a history • PDM simulates current students’ hierarchical knowledge

17. Curriculum Information Network actor double_int actor_object variable_attribute numeric datatype object object_class object_constructor class_method class_attribute variable_parameter variable_returntype method_parameter datatype variable attribute_constructor method_constructor pass in only constructor attribute_method int double int_string actor_method string double_string datatype_returntype object_method datatype_variable object_attribute class class_constructor method attribute returntype parameter method_returntype A B A is prerequisite of B attribute_parameter

18. Two kinds of concepts • Unique concept, such as attribute or parameter • Relationship concepts, such as attribute_parameter • Relationships emerge because of student’s confusions between concepts • E.g., student defines movieTitle as a parameter when he has already defined movieTitle as an attribute

19. Prerequisite relationships • Prerequisite is relationship between concepts: • The concepts a learner needs to understand before understanding a concept • E.g., one needs to understand int and double in order to understand numericDatatype • Relationship concepts are prerequisites of unique concepts and vice versa • E.g., class_constructor -> constructor • Understanding constructor doesn’t imply understanding of class, just how to define a constructor for a class

20. ku au Connecting Knowledge with Performance • Student action unit and knowledge unit make a pair(KU,AU) • Infer understanding of a concept (KU) from a student solution step (AU) • Action unit (AU): • A single action or step in a student’s solution • E.g., add an attribute to a class • Knowledge unit (KU) – concept a student need to learn • KU directly causes a student action unit • KU is a concept in Curriculum Information Network (CIN)

21. Atomic Bayesian Network (ABN) …… d-prereq(ku)N d-prereq(ku)1 d-prereq(ku)2 Noisy-and generalizes logical-and ku Students must understand all direct prerequisites of the concept kuin order to understand ku au

22. How to generate an ABN • Student model generates an ABN in response to a student solution step • First, define the structure of an ABN, i.e., the causal relationship between KU and AU, and the direct-prerequisites of KU • Second, determine conditional probability tables for this ABN

23. 0 d-p(ku)N 1 … d-p(ku)N … 0 1 d-p(ku)2 0 d-p(ku)1 1 d-p(ku)1 ku 0 ku 1 d-p(ku)2 0 au 1 au Atomic Dynamic Bayesian Network (ADBN) for HM layer

24. How to generate an ADBN • Student model generates an ADBN in response to a student solution step • First, look for the ABN in response to previous student solution step • Second, generate an ABN in response to current student solution step • Third, determine conditional probability tables for the ADBN

25. Concrete Example • Student defined movieTitle as a parameter for method displayMovieTitle after she has already defined movieTitle as an attribute to a class TicketMachine • EE determines that movieTitle should not be a parameter • SM determines that the center concept of an ABN is attribute_parameter, andfinds all direct prerequisites, attribute and parameter, from CIN

26. Concrete Example • attribute’s prior can be found from the database • parameter’s prior is 0.5, students’ knowledge state is assessed based on the difference between prior and posterior probabilities (VanLehn et al. 1998, Millán & Pérez-de-la-Cruz 2002) • SM determines: • student has good understanding of class, attribute,methods, and parameter but low understanding of attribute_parameter • the tutoring need is: explanation of attribute_parameter

27. Concrete Examplefeedback • “Since you have added movieTitle as an attribute to the class TicketMachine, you shouldn’t also make it a parameter to the method displayMovieTitle. To decide whether movieTitle should be an attribute or a parameter, remember: attributes are accessible anywhere within the scope of a class, while parameters are accessible only within the scope of a method”

28. Publication Background Research questions Methodology Evaluation Conclusions and future work Presentation Outline

29. Evaluation of ABNs with simulated students • Hypotheses: • Pre-setting slip and guess values will lead to a reliable student model • Varying slip and guess values will affect the accuracy of the student model

30. Pre-setting slip and guess values to same (relatively small e.g. <=0.1) values produces accuracy of at least 93%, confirming the first hypothesis • Changing of presetting slip and guess causes accuracy to change from 79.1% to 94.3%, confirming the second hypothesis • Correct diagnostic rates are higher when slipp, guessp and slipe, guesse take same value • No significant difference when slip and guess take a same small value (<=0.1)

31. Evaluation of ADBNs with simulated students • Hypotheses: • Pre-setting slip and guess values can lead to a reliable student model • Modeling learning history with ADBNs will enhance the accuracy of the student model

32. The significant difference between correct diagnostic rates using ABNs versus using ADBNs demonstrates that ADBNs enhance the accuracy of the student model, confirming the second hypothesis

33. Pre-setting slipp/e, guessp/e, to relatively small (e.g. <=0.1) values produces accuracy of at least 97%, confirming the first hypothesis. • The accuracy is not sensitive to change of slip and guess values so along as the values are relatively small (<=0.1)

34. Evaluation of student model for CIMEL multimedia with real students • Hypotheses: • Pre-setting slip, guess will lead to a reliable student model • Pre-setting slip and guess to various values will affect the accuracy of the student model • The student model will perform in real-time, i.e., it will support responses as students are working on exercises • Results: • The average response time of the student model after a students enters the solution step is 0.24 seconds

35. Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 80.1%. • Changing the presetting slip and guess causes the accuracy to change from 71.7% to 80.1%

37. Has a higher value of r than student model by Corbett et al. (2000)

38. Evaluation of student model for DesignFirst-ITS with real students • Hypotheses: • Pre-setting slip, guess will lead to a reliable student model • The student model will perform in real-time, i.e., it will support responses as students are working on problem-solving steps • Results: • The average response time of the student model after a student enters a solution step is 0.63 seconds

39. Presetting slip and guess with relatively small values (<=0.1) can produce accuracy of up to 81.8% • Varying the slip and guess value does not affect the accuracy of the student model, so long as slip and guess values are relatively small (<=0.1)

40. Has a higher value of r than student model by Corbett et al. (2000)

41. Evaluation of diagnoses integration(multimedia and DesignFirst-ITS) with real students • Hypothesis: • Integrating diagnoses from closed-ended questions will enhance the accuracy of diagnoses of student model for open-ended questions

42. Accuracy increases 7.7% when adding diagnoses from closed-ended exercises in multimedia, confirming hypothesis

43. Comparing with non-advanced-numerical student models • Non-advanced-numerical techniques include match, summation and subtraction • Advanced-numerical techniques include Bayesian networks • Hypotheses: • Straightforward algorithm will not lead to a reliable student model • ADBNs will perform better than “match” student model

44. From same set of evidences, ADBNs perform more than two times better than “match” student models

45. Publications Background Related Research Research questions Methodology Evaluation Conclusionsandfuturework Presentation Outline

46. Conclusions • Student models with ADBNs can diagnose student knowledge states accurately in real-time • Accuracy of ADBN-based student model is significantly higher than ABN student model • Integrating diagnoses from closed- and open-ended exercises is an effective way to increase accuracy of student models • Student models using ADBNs perform much better than the student models that use straightforward algorithm

47. Future work • Implement cognitive model to simulate monitoring knowledge and reflective knowledge • Consider students learning gain from reviewing feedback • how do we determine the conditional probability table for the ADBN so as to simulate the real student learning? • how do we update the new ADBN? • how do we convey empirical studies with simulated students and human subjects? • Diagnose students’ learning state in other domains, such as object-oriented programming

48. An ADBN considers feedback

49. Contributions (1 of 2) • A novel way to represent students’ knowledge structure, where both concepts and relationship between concepts are knowledge units that students need to learn • A novel three-layered architecture which can be standardized in modeling various stratums of students’ knowledge • ABN – a novel Atomic Bayesian network that provides a refined representation of prerequisite relationships, diagnoses student’s knowledge structure, and guarantees real-time responsiveness

50. Contributions (2 of 2) • ADBN – an innovative dynamic Bayesian network that represents refined representation of prerequisite relationships and diagnoses students’ knowledge structure in real-time considering learning history • A unique student model that integrates knowledge from open-ended problem solving (object-oriented class diagram design) and closed-ended exercises • A general approach for student models that help students learn complex problem solving in real time