Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context

1. Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context • Motivation • Previous Study on Human Tutoring • Comparison between • Script based Learning (S) • Slightly more Exploratory Problem Solving Condition (PS) • Human Tutoring considered as Negotiated Problem Solving Goals (NPSG) Condition • Other Results • AutoTutor System (Computer Literacy): 0.5σ Improvement • WHY-AutoTutor System (Qualitative Physics) • Dialog better than Textbook Reading The Exploratory Learning Continuum Problem Solving Goals Provided Worked Examples Tutorial Learning Negotiated Problem Solving Goals Pure Exploratory Learning Greetings General Talk Interface Issues Setup Initialization Cycle Change Tmax Analysis Modeling Assumptions Reheat Concept Pmax Analysis Material Constraints Thermodynamics Concept Sensitivity Analysis Pmin Analysis Steam Quality Constraint Regeneration Concept Splitter-Mdot Analysis Experimental Method • Three hour session: 9 Steps • Consent Form followed by Introductory Exercise • Pre-Test (32 Questions, 50 points) • Reading of Rankine Cycle Overview (15 Minutes) • Focused Materials 1: Simple Rankine Cycle • Reading, Initialization, Analysis/Optimization • Focused Materials 2: Rankine Cycle with Reheat • Focused Materials 3: Rankine Cycle with Regeneration • Two free exploration exercises • Post Test • Questionnaire Three Conditions S: Script based Learning PSHELP: CyclePad + Pseudo Tutors + KCDs for Help Messages (fired when student asks for help) PSSUCCESS: CyclePad + Pseudo Tutors + KCDs for Help Messages & Success Messages (fired when student successfully performs a correct action) Rohit Kumar, Carolyn Rosé, Vincent Aleven, Ana Iglesias, Allen Robinson • Tutorial Dialogue • Benefits • High Degree of Cognitive Engagement from Students • Great Deal of Flexibility in presentation for Tutors • Costs • Time & Effort • Where would Tutorial Dialogue have the greatest Impact • Learning Environments where benefits would outweigh the costs We evaluate first implementation of the NPSG Approach to Instructional Support in Simulation based Learning Environment The CycleTalk System Content Development for KCDs Topic Analysis of Human Tutoring Corpus Cognitive Task Analysis Cognitive Tutors Authoring Tools Pseudo Tutors Koedinger, K. R., et. al. (CMU) Behavior Graphs created demonstrating correct and incorrect solutions to the problems Rankine Cycles • 12 KCDs integrated with pseudo tutors • 5 (of 12) were more conceptual • 7 (of 12) focused on specifics of Optimization Foundation for the design of majority of Steam based power plants in the United States Tutor: Let’s stop for a minute and reflect on what we have seen. Having seen how changing T-Max (i.e., the Maximum Temperature at the Heater Outlet) affects the efficiency of the cycle, do you think it is a good idea in general for the Maximum Temperature to be increased and kept high? Student:yes Tutor: Okay. Let’s think now about one important consequence of keeping T-Max high. What happens to the steam quality at the turbine exit if you increase the maximum temperature of the cycle? Student: Increases Tutor: Right. As increasing the temperature makes the efficiency and steam quality better, we should increase the maximum temperature of the cycle. But can we keep on increasing the maximum temperature at the heater outlet indefinitely, or are there limitations to keep in mind? Student: there are limitations such as material constraints Tutor: Great. we should try to keep T-Max (i.e., the maximum temperature at the Heater outlet of the cycle) as high as possible without risking safety concerns or exceeding the maximum temperature the material can withstand. Keeping T-Max high increases efficiency and keeps the steam quality high, which are both important. This is an important principle to keep in mind while optimizing the efficiency of your cycle. Simple Rankine Cycle Each Arc in the Behavior Graph represents a student action Correct Action are associated with Hints and Success Messages Incorrect Actions are associated with Error Messages Rankine Cycle with Reheat Knowledge Construction Dialogues (KCDs) Rosé, et. al. Rankine Cycle with Regeneration Provides students with opportunities to construct their own knowledge by conversing (in typed-form) with a Natural Language-based ITS KCDs integrated with the Pseudo Tutors by allowing dialogues to be launched instead of Hints and/or success messages when the student performs the corresponding action. Results Human Tutoring covered 4 to 11 (of 15) distinct topics for each student PSHELP: 1.8 KCDs on Average PSSUCCESS: 2.7 KCDs on Average Need to increase opportunities to view KCDs Comparing PSHELP & PSSUCCESS: 0.35σ (p < 0.05) advantage in PSSUCCESS Follow-up Experiment in USNA (Feb. 2006): 0.25σ (p < 0.05) advantage in PSSUCCESS over S Condition where more KCDs were viewed has significant learning advantage