: F lexible O pen L earner M odeling

1. :Flexible Open Learner Modeling Sergey Sosnovsky,PAWS@SIS@PITT

2. References • Susan Bull, ,UK. • Mabbott, A. & Bull, S. (2004). Alternative Views on Knowledge: Presentation of Open Learner Models, ITS2004, 689-698. • Mabbott, A. & Bull, S. (2006). Student Preferences for Editing, Persuading and Negotiating the Open Learner Model, ITS2006, 481-490. • Kerly, A. & Bull, S. (2006). The Potential for Chatbots in Negotiated Learner Modelling, ITS2006, 443-452.

3. Outline • Open User Model • Flexi-OLM: • Viewing LM • Editing LM • Persuading LM • Negotiating LM • Multiple LM Presentations • Evaluation • Demo

4. Open Learner Modeling • What:Visualization of the learner model, providing a learner with a mechanism to explore it, sometimes, negotiate it. • Why:When a learner is engaged in the analysis of the learner model he is reflecting upon his domain knowledge and experience re-calling and re-considering ideas of which he is aware.

5. Flexi-OLM • Models student understanding of basic C programming • Uses color coding for telling students about the concept knowledge levels: • While – limited understanding • Pale yellow – somewhat limited • Yellow/green – moderate • Bright green – excellent • Red – misconception • Grey – insufficient data • Large topics include smaller concepts. Clicking on a topic in the model brings more detailed concept-wise information about this topic understanding. • Knowledge are assessed with the help of short questions • After playing with the system: • Questions correspond to only one concept • No knowledge Inference between concepts • Very simple modeling formula (seems like average with linear thresholds for knowledge levels)

6. Editing LM • Flexi-OLM allows direct editing of LM • Possible scenarios for this will be: • new learner wishes to inform the system about topics she already understands; • the learner grasps a concept outside the system and wants LM to reflect this; • the learner correctly guesses a series of answers => LM has a higher knowledge level than she believes she has.

7. Scrutinizing and Persuading LM • Less direct method: • A learner registers a disagreement with the LM and propose a change • Flexi-OLM explains its believes by presenting the evidence supporting these beliefs • If the learner still wishes to proceed, she has the opportunity to ‘persuade’ the LM by answering a series of test questions. • Possible Scenarios: • A learner believes her knowledge may be different than the system asserts, but lacks the confidence to edit it unchallenged, • A learner seeks the satisfaction of proving the system wrong

8. Negotiating LM • Flexi-OLM supports conversation-based negotiation of LM: • A learner is chatting with the system (as he thinks) • Flexi-OLM maintains separate believe models for LM and for a learner • It is ensured that the same dialogue moves are available to both parties => Each party: • has full control over their own beliefs, • can challenge the other’s belief, • can seekjustification for the other’s belief (on the LM side justification is based on the past learner’s answers), • may request justification before changing their own beliefs, • may ultimately decide to leave their belief unchanged. • If the difference between LM’s and Student’s beliefs is: • 1 level – The LM accepts the learner’s suggestion • 2 levels – A compromise is offered (of changing both beliefs by one level) • 3 levels – The systems seeks a justification (the learner will be asked to answer a question) • [possible hack] – gradual change of the LM belief by one level

9. Negotiating LM (cont.)The “Wizard-of-Oz” Paradigm • Human experimenter takes the role of the chatbot – “Wizard” • The “Wizard” follows a protocol designed to ensure: • that responses to students remained consistent between users, and • that the ‘chatbot’ was believable to users. • To enact the protocol, the Wizard was provided with some 350 pre-authored ‘chatbot’ negotiation initiations and responses to user inputs. • Typical conversation:

10. LM Presentations in Flexi-OLM • hierarchy, a logical grouping of related concepts; • lectures, where topics are organized the same as in the related lecture course; • concept map showing relationships between the topics; • prerequisites, showing possible sequences for studying topics; • index, an alphabetical list; • ranked, where topics are listed in order of proficiency; • textual summary.

11. Experiment 1 (2004) • Two questions: • Is it beneficial for students to have a choice over presentation of open LM, or it causes information overflow? • Is there any strong preference for a particular LM view among individuals and if so, can it be predicted on the basis of learning style? • 23 undergraduate students • Experiment flow: • Pre-test (control flow in C) to populated LM • Browsing session, where students can choose among 4 different presentations

12. Experiment 1: Results

13. Experiment 2 (2006) Editing LM (full student control) Persuading LM (full system control) Negotiating LM • Main question: • What are the students preferences concerning editing, negotiating and persuading LM? • 8 third-year undergraduate students • Experiment flow: • Initial testing to populate LM • 1-hour LM exploring session (edit & persuade) • 20 minutes of negotiating with LM

14. Experiment 2: Results Sum 20 36 23 11 34

15. Experiment 3 (2006) • The goal: • To explore the feasibility of using achat-based interface in an OLM system • 11 final year undergraduates • Experiment flow: • Self-assessment of student knowledge of each concept, providing the initial user’s beliefs to the system. • Interacting with the system – to populate LM and provide it with its beliefs about student knowledge • Then students were shown how to use the chatbot and asked to interact with it forat least 20 minutes. • Post-experiment questionnaire

16. Experiment 3(2006)

17. Demo http://olm.eee.bham.ac.uk/flexi-olm/login.php

18. Thank You for Your Questions