LearnLab : Bridging the Gap Between Learning Science and Educational Practice

1. LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken KoedingerHuman-Computer Interaction & Psychology, CMUPI & CMU Director of LearnLab

2. Real World Impact of Cognitive Science Algebra Cognitive Tutor Based on ACT-R theory & cognitive models of student learning Used in 3000 schools600,000 students Spin-off: Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

3. Cognitive Tutors: Interactive Support for Learning by Doing Feedback within complex solutions Authentic problems Progress… Personalized instruction Challenging questions … individualization

4. Success ingredients • AI technology • Cognitive Task Analysis • Principles of instruction & experimental methods • Fast development & use-driven iteration

5. Cognitive Task Analysis: What is hard for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with? Equation x * 6 + 66 = 81.90

6. Expert Blind Spot! 100 90 % Correctly ranking equations as hardest 80 70 60 50 40 30 20 10 0 Elementary Middle High School Teachers School Teachers Teachers Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction. Data contradicts common beliefs of researchers and teachers Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.

7. Cognitive Tutor Algebra course yields significantly better learning Course includes text, tutor, teacher professional development ~11 of 14 full-year controlled studies demonstrate significantly better student learning Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

8. Success? YesDone? No! Why not? • Student achievement still not ideal • Field study results are imperfect • Many design decisions with no research base • Use deployed technology to collect data, make discoveries, & continually improve

9. PSLC Vision Why?Chasm between science & ed practice Purpose: Identify the conditions that cause robust student learning Educational technology as instrument Science-practice collaboration structure Core Funding:2004-2014

10. What we know about our own learning What we do not know Do you know what you know? You can’t design for what you don’t know!

11. Transforming Education R&D Ed tech + wide use = “Basic research at scale” Algebra Cognitive Tutor • Fundamentally transform • Applied research in education • Generation of practice-relevant learning theory + = Chemistry Virtual Lab English Grammar Tutor Educational Games

12. Ed Tech => Data => Better learning LearnLab Course Committees LearnLab Thrusts

13. How you can benefit from LearnLab • Research • General principles to improve learning • Methods • Cognitive task analysis, in vivo studies • Technology tools • People • Masters students & projects

14. What instructional strategies work best? • More assistance vs. more challenge • Basics vs. understanding • Education wars in reading, math, science… • Research on many dimensions • Massed vs. distributed (Pashler) • Study vs. test (Roediger) • Examples vs. problem solving (Sweller,Renkl) • Direct instruction vs. discovery learning (Klahr) • Re-explain vs. ask for explanation (Chi, Renkl) • Immediate vs. delayed (Anderson vs. Bjork) • Concrete vs. abstract (Pavio vs. Kaminski) • … Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review.

15. Knowledge-Learning-Instruction (KLI) Framework: What conditions cause robust learning LearnLab research thrusts address KLI elements Cognitive Factors Charles Perfetti, David Klahr Metacognition& Motivation Vincent Aleven, Tim Nokes-Malach Social Communication Lauren Resnick, Carolyn Rose Computational Modeling & Data Mining Geoff Gordon,Ken Koedinger Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

16. Results of ~200 in vivo experiments =>Optimal instruction depends on knowledge goals

17. Without decomposition, using just a single “Geometry” KC, no smooth learning curve. But with decomposition, 12 KCs for area concepts, a smoother learning curve. Cognitive Task Analysis using DataShop’s learning curve tools Upshot: Can automate analysis & produce better student models

18. How you can benefit from LearnLab • Research • General principles to improve learning • Methods • Cognitive task analysis, in vivo studies • Technologies • Tutor authoring • Language processing • Educational Data Mining • People: Masters students & projects

19. Questions?

20. Question for you What do you need in a learning science professional?

22. Extra slides

23. Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can 3(2x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

24. Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can Hint message: “Distribute aacross the parentheses.” Bug message: “You need tomultiply c by a also.” Known? = 85% chance Known? = 45% 3(2x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9 • Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction • Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

25. Cognitive Task Analysis Improves Instruction Studies: Traditional instruction vs. CTA-based Med school catheter insertion (Velmahos et al., 2004) Radar system troubleshooting (Schaafstal et al., 2000) Spreadsheet use (Merrill, 2002) Lee (2004) meta-analysis: 1.7 effect size!

26. Learning Curves

27. Inspect curves for individual knowledge components (KCs) Some do not =>Opportunity to improve model! Many curves show a reasonable decline

28. DataShop’s “leaderboard” ranks alternative models100s of datasets from ed tech in math, science, & language Best model finds 18 components of knowledge (KCs) that best predict transfer

29. Data from a variety of educational technologies & domains Statistics Online Course English Article Tutor Algebra Cognitive Tutor Numberline Game

30. Model discovery across domains Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of Educational Data Mining. [Conference best paper.] Variety of domains& technologies 11 of 11 improved models

31. Data reveals students’ achievement & motivations We have used it to • Predict future state test scores as well or better than the tests themselves • Assess dispositions like work ethic • Assess motivation & engagement • Assess & improve learning skills like help seeking …

32. Researchers Schools Learn Lab LearnLab courses at K12 & College Sites • 6+cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English • Data collection • Students do home/lab work on tutors, vlab, OLI, … • Log data, questionnaires, tests DataShop Chemistry virtual lab Physics intelligent tutor REAP vocabulary tutor

33. Bridging methodology: in vivo experiments

34. Knowledge Components • Definition: An acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks • Includes: • skills, concepts, schemas, metacognitive strategies, malleable habits of mind, thinking & learning skills • May also include: • malleable motivational beliefs & dispositions • Does not include: • fixed cognitive architecture, transient states of cognition or affect • Components of “intellectual plasticity” Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

35. General knowledge components, sense-making, motivation, social intelligence Possible domain-general KCs • Metacognitive strategy • Novice KC: If I’m studying an example, try to remember each step • Desired KC: If I’m studying an example, try to explain how each step follows from the previous • Motivational belief • Novice: I am no good at math • Desired: I can get better at math by studying & practicing • Social communicative strategy • Novice: If an authority makes a claim, it is true • Desired: If considering a claim, look for evidence for & against it

36. What is Robust Learning? • Achieved through: • Conceptual understanding & sense-making skills • Refinement of initial understanding • Development of procedural fluency with basic skills • Measured by: • Transfer to novel tasks • Retention over the long term, and/or • Acceleration of future learning

37. Intelligence does not improve generically KLI summary • Learning occurs in components (KCs) • KCs vary in kind/cmplxty • Require different kinds of learning mechanisms • Optimal instructional choices are dependent on KC complexity Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

38. Conclusions • Learning & education are complex systems • Lots of work for learning science! • Use ed tech for “basic research at scale”=> Bridge science-practice chasm