1 / 38

Cognitive Analysis of Student Learning Using LearnLab

Cognitive Analysis of Student Learning Using LearnLab. University of Pittsburgh. Brett van de Sande, Kurt VanLehn, & Tim Nokes. Agenda. LearnLab methodology Demonstration of Andes, an intelligent homework tutor Log File Analysis. Goal: To understand physics learning. Policy level

lavonn
Download Presentation

Cognitive Analysis of Student Learning Using LearnLab

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cognitive Analysis of Student Learning Using LearnLab University of Pittsburgh Brett van de Sande, Kurt VanLehn, & Tim Nokes

  2. Agenda • LearnLab methodology • Demonstration of Andes, an intelligent homework tutor • Log File Analysis

  3. Goal: To understand physics learning • Policy level • e.g., Physics for high school freshman? • Instructional level • e.g., How much assistance to give? • e.g., How much practice per topic? • e.g., How to handle errors? • Neurocognitive level • e.g., Can neuroimaging distinguish deep from shallow studying of a text? Our focus

  4. Traditional methodsfor studying learning • Design experiment • Modify text, classroom activities, tests… • e.g., Project Scale-up • Lab experiment • Modify just one factor • Brief; money instead of grades, …

  5. PSLC methods • Educational data mining • Logs from instrumented courses • Some analysis is automated • In vivo experiments • Control of variables • Instrumented courses Next

  6. Instrumented courses(Called LearnLab courses) • Existing class + data collection • Homework done on a tutoring system or photocopied and analyzed • Photocopies of quizzes, exams • FCI given before and after the course • Demographics, GPAs, Majors… • Handouts, slides, clicker data,… • Instructor, student & IRB cooperation • Anonymity

  7. Existing Physics LearnLab Course(s) • US Naval Academy • Course take by all 2nd year students • LearnLab is in 4 of about 20 sections • Profs. Wintersgill, McClanahan • Your course here

  8. Basic data mining question • What features of students’ histories are statistically associated with learning gains? • e.g., What are the differences between histories of Student A and Student F?

  9. Knowledge decomposition hypothesis • Decompose knowledge to be learned into a set of knowledge components • e.g., Newton’s third law • e.g., Centripetal acceleration • Assume each knowledge component is learned independently • An approximation/idealization

  10. Data mining with knowledge components (KCs) For each KC, find statistical associations between histories and gains.

  11. History decomposition hypothesis • Decompose the student’s history into events such that each event addresses only one (or a few) knowledge components. • Reading a paragraph about Newton’s 3rd law • Drawing a reaction force vector • Seeing the instructor draw a reaction force • Drawing a centripetal acceleration vector • Assume that a KC’s learning gain depends only on that KC’s events

  12. Events for 1 student on 1 KC(e.g., Newton’s 3rd law)

  13. Some events are not currently available

  14. More feasible data mining • Predict learning gains of a KC given the sequence of events relevant to that KC • On an event that assesses mastery of a KC, predict the student’s performance during that event given the sequence of preceding events relevant to that KC

  15. Predicting correctness ofevents that assess mastery

  16. Learning curves • Plot assessment events on x-axis • Ordered chronologically • Plot measure of mastery on y-axis • Usually aggregated across subjects e.g., proportion of 100 subjects who performed correctly on this event

  17. An expected learning curve 1.0 Frequency of correct 0.5 0 1 2 3 4 5 6 7 8 Assessment events

  18. Summary of PSLC educational data mining • Given knowledge to be learned • Decompose into knowledge components • Given students’ histories from an instrumented course • Divide into assessment/instruction events • such that one KC (or a few) per event • For each KC, find a function on a sequence of events that predicts the KC’s • learning gain during the course • learning curve

  19. PSLC methods • Educational data mining • Logs from instrumented courses • Some analysis is automated • Andes produces logs with KCs • DataShop draws learning curves, etc. • Correlation ≠ Causation • In vivo experiments • Control of variables • Instrumented courses Next

  20. Two major typesof in vivo experiments • Short & fat • During one lesson or one unit • Long & skinny • During whole course • “invisibly”

  21. Example of a short, fat, in vivo experiment (Hausmann 07) • During a 2-hour period (usually used for lab work) • ~25 students in the room, each with a laptop and a headset mike • Repeat 3 times: • Study a video while explaining it into the mike • Solve a problem • 4 experimental conditions, varying the content of the video and the instructions for explaining it • Random assignment of students to conditions • Dependent measures include learning curves • Result: Instructions to self-explain worked best regardless of content of the video

  22. Example of a long, skinny in vivo experiment (Katz 07) • During 8 weeks of a 13-week course • Random assignment to 2 conditions: • Experimental group: After solving certain homework problems, the student discussed the solution with a natural language tutoring system • Control group: Extra homework problems • Result: Experiment > Control on some conceptual measures

  23. Robust Learning • Immediate learning • During an immediate post-test • Similar content to training (near transfer) • Robust learning • Far transfer • Retention • Acceleration of future learning • Does manipulation of instruction on topic A affect rate of learning of a later topic, B?

  24. Summary of PSLC methodology • Data mining • Instrumented (LearnLab) courses • Knowledge components • Instructional and assessment events • Learning curves • In vivo experiments • Short & fat vs. long & skinny • Robust learning

  25. Agenda • LearnLab methodology • Demonstration of Andes, an intelligent homework tutor • Log File Analysis Next

  26. Define variables Draw free body diagram (3 vectors and body) Define coordinates (3 choices for this problem) Upon request, Andes gives hints for what to do next

  27. Red/green gives immediate feedback for student actions Principle-based help for incorrect entry

  28. # Log of Andes session begun Tuesday, July 17, 2007 12:12:28 by [User] on [Computer] ... 05:03 DDE (read-problem-info "S2E" 0 0) ... 02:35 Axes Axes-671 64 335 143 296 02:35 Axes-dlg Axes-671 || … 02:38 C dir 40 02:42 BTN-CLICK 1 OK 02:42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z") 02:42 DDE-COMMAND assoc step (DRAW-AXES 40) 02:42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES 02:42 DDE-COMMAND set-score 39 02:42 DDE-RESULT |T| ... 10:02 E 0 F1_y+F2_y=0 10:02 EQ-SUBMIT 0 10:02 DDE (lookup-eqn-string "F1_y+F2_y=0" 0) 10:47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL1_1_40 Yc_Fn_BALL_WALL2_1_40) 0) 10:47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM 10:47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40 Yc_Fn_BALL_WALL2_1_40 Yc_Fn_BALL_WALL1_1_40) 0)) 10:47 DDE-COMMAND assoc op WRITE-NFL-COMPO 10:47 DDE-RESULT |NIL| ... 10:50 DDE-RESULT |!show-hint There is a force acting on the ball at T0 that you have not yet drawn.~e| ... 16:38 END-LOG problem name student action (draw axes) interpretation: compare to model green student action (equation) error analysis: intended action red session time

  29. Demonstration by Tim Nokes # Log of Andes session begun Wednesday, April 18, 2007 21:08:07 by [user] on [computer] ... 0:02 DDE (read-problem-info "FARA9" 0 0) ... 0:13 Help-Hint 0:13 DDE (Get-Proc-Help) 0:13 DDE-COMMAND assoc (NSH NEW-START-AXIS 0) 0:13 DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an axis. This helps to ground your work and will be useful later on in the process.~e| … 0:17 Begin-draw 50001 Axes-1 185 331 ... 0:30 New-Variable resistance ... 0:39 DDE (define-variable "R" |NIL| |resistance| |R| |NIL| |NIL| Var-2 "20 ohm") 0:39 DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R)) 0:39 DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR 0:39 DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm)) 0:39 DDE-COMMAND set-score 3 0:39 DDE-RESULT |T| .... 0:50 DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3) 0:50 DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) ZERO) 0:50 DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO 0:50 DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) OUT-OF) 0:50 DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR 0:50 DDE-COMMAND set-score 2 0:50 DDE-RESULT |NIL| ... 9:51 DDE-RESULT |T| 9:55 END-LOG

  30. Agenda • LearnLab methodology • Demonstration of Andes, an intelligent homework tutor • Log File Analysis Next

  31. Principle B Op2 Op3 Op5 Op8 Op10 Model Solution Set Solution 1 Solution 0 Principle A Op1 Op3 Op6 Op7 Principle C Op10 Op11 Op12 Principle A Op1 Op3 Op6 Op7 Principle D Assumption: Opi = KC

  32. # Log of Andes session begun Friday, July 27, 2007 14:29:38 by bobh on BOBH … 0:02 DDE (read-problem-info "S2E" 0 0) … 11:45 Vector-dlg Vector-673 || … 11:48 CLOSE type instantaneous 11:48 SEL type 1 instantaneous 11:51 BTN-CLICK 1 OK 11:51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T0| Vector-673) 11:51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL :TIME 1) ZERO) 11:51 DDE-COMMAND assoc op ACCEL-AT-REST 11:51 DDE-RESULT |T| … problem name student actions match model solution: assoc step = entry Assoc op = operator green

  33. 14:03 E 8 Fearth_y = m*g 14:11 EQ-SUBMIT 8 14:11 DDE (lookup-eqn-string "Fearth_y = m*g" 8) 14:11 DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (* m_BALL g_EARTH)) 14:11 DDE-COMMAND assoc error MISSING-NEGATION-ON-VECTOR-COMPONENT 14:11 DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (* m_BALL g_EARTH))),(EQN (= Yc_Fw_BALL_EARTH_1_0 (- Fw_BALL_EARTH_1))) 14:11 DDE-COMMAND assoc op WT-LAW,COMPO-PARALLEL-AXIS 14:11 DDE-COMMAND set-score 74 14:11 EQ-F 8 14:11 DDE-RESULT |NIL| student actions error interpretation guess intended red

  34. Review Video Match steps in video to log file

  35. Researchable questions Timing Sequencing (order of steps) Hint Usage Problem solving skills Self-correction of errors Errors as window to mental state

  36. DataShop https://learnlab.web.cmu.edu/ > Launch DataShop > New user? Sign up now! > (Create account)

More Related