210 likes | 328 Views
Reference Model for Evaluating Intelligent Tutoring Systems. Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université de Montréal. Introduction.
E N D
Reference Model for Evaluating Intelligent Tutoring Systems Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université de Montréal
Introduction • Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner • Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process • building an ITS needs also to evaluate the system • lack of evaluation methodology Laboratoire HERON - TICE 2000
Architecture of an ITS Student Laboratoire HERON - TICE 2000
Evaluation of ITS • Evaluation can be formative or summative • Summative: after the design process • Formative: during the design (define and refine goals and methods) • Evaluation techniques used in • ITS • Software engineering Laboratoire HERON - TICE 2000
Evaluation of some ITS • SHERLOCK : experimental evaluation • VCR Tutor : 4 versions compared • Formative qualitative evaluation • iterative design • formative methods • qualitative : characteristics of a situation • quantitative : finding causes and consequences • case studies Laboratoire HERON - TICE 2000
Lester’s evaluation • On animated pedagogical agents (Design a Plant) --- botanical and physiology • 100 middle school students • Important educational benefits : improved problem solving • Better than less expressive animated agents Laboratoire HERON - TICE 2000
Littman- Soloway • What is the educational impact on students ? • What is the relationship between the architecture of an ITS and its behavior ? • External evaluation of the student model (Proust) • Internal evaluation : architecture, how ITS respond to input values Laboratoire HERON - TICE 2000
Van Lehn • Teaching metacognitive skills to implement and evaluate an ITS (SE-Coach guide self-explanation) • Empirical evaluation • Fundamental questions • How the design tools are used efficiently ? • How the learner improve his knowledge ? Laboratoire HERON - TICE 2000
Software engineering techniques • Product : Boehm’model • software doing what the user want it to do • use resources correctly • easy to learn • well designed, code • easily tested and maintained Laboratoire HERON - TICE 2000
HCI techniques • Evaluating design • cognitive walthrough (how easy a system is to learn) • heuristic evaluation (visibility, consistency, flexibility, helps for the user) • model-based evaluation (GOMS) predict user performance Laboratoire HERON - TICE 2000
Evaluationg Implementation • Empirical and experimental methods • Observational methods (user completes a set of tasks) • Query techniques • interviews of the users about their experience • questionnaires : questions fixed in advance Laboratoire HERON - TICE 2000
Learner model • Layers : • cognitive • affective • inferential • Learning time : speed of knowledge acquisition • Tracability : of learner’s actions Laboratoire HERON - TICE 2000
Levels of knowledge Based on works of Gagné (1985) 7 levels of knowledge 4 principal levels Novice Beginner Intermediate Expert Learner model Laboratoire HERON - TICE 2000
Learning strategies • One-on-one • Co-learner • Learning Companion • Learning by Disturbing Laboratoire HERON - TICE 2000
Learning strategies • Diversity • Adaptability (switching to different strategies) • Modification • Memorization • Feedback • Reduce cognitive load Laboratoire HERON - TICE 2000
Curriculum • Generalisation • Consistency • Knowledge articulation • Reusability • Navigability • Maintenance Laboratoire HERON - TICE 2000
Interface • Intuitive : learner should understand easily the functions to apply • Interactivity : allowing the learner to be active in his learning environment • Matching : using words, phrase and concepts Laboratoire HERON - TICE 2000
General • Productivity rate : the most important factor. Time spent to produce one hour of ITS based course • Learning outcomes : performance obtained by the learner after the session • Ease of use of the tools Laboratoire HERON - TICE 2000
Conclusion • Evaluation process is complex but needs to be realistic • Develop first , evaluate later • Evaluation criteria as a guideline for estimation Laboratoire HERON - TICE 2000
Merci de votre attention Laboratoire HERON - TICE 2000
Stratégies d ’apprentissage Tuteur Co-apprenant Compagnon Apprentissage par explication Laboratoire HERON - TICE 2000