Data mining with DataShop

1. Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University Ryan S.J.d. BakerPSLC/HCII Carnegie Mellon University

2. Overview Motivation for educational data mining DataShop Learning curves to improve cognitive models Past project example Conclusion Next

3. What is educational data mining? • “The area of scientific inquiry centered around the development of methods for making discoveries within the unique kinds of data that come from educational settings, and using those methods to better understand students and the settings which they learn in.” (Baker, under review)

4. What is educational data mining? • More informally: using “large” data sets to answer educational and psychological questions • What “large” means is always changing • Developing methods or algorithms to aid in discovery

5. What is educational data mining? • One popular data source is “instrumented” computer tutors • Fine grained, longitudinal, often across contexts • Other data sources • Records of online courses (e.g. WebCAT) • District or university-level student records • Example: www.icpsr.umich.edu/IAED

6. Educational Data Mining is a hot topic! • 2008: First International Conference on Educational Data Mining • 2008: Launch of Journal of Educational Data Mining • 2009: Second International Conference on Educational Data Mining • Submissions due in March 2009 • www.educationaldatamining.org

7. Data Mining Questions & Methods • How can we reliably model student knowledge or achievement? • Bayesian Knowledge Tracing • Simple type of “Bayes Net”, getting less simple all the time • Item Response Theory (IRT) • Basis for standardized tests, SAT, GRE, TIMSS… • Version of “logistic regression” • Many variations & generalizations … • See slides of Brian Junker’s EDM08 invited talk

8. Data Mining Questions & Methods • What’s the nature of knowledge students are learning? • How can we discover cognitive models of student learning? • Learning Factors Analysis (LFA) • Extends IRT to account for learning • Search algorithm: Discover cognitive model(s) that capture how student learning transfers over tasks over time • Rule space, knowledge space, …

9. Data Mining Questions & Methods • How can we model students, beyond just what they know? • Models of • Choices: Metacognitive & Motivational • Help-seeking • Gaming the System • Off-Task Behavior • Self-explanation • Affect • Involves prediction methods such as classification, regression (not just linear regression)

10. Data Mining Questions & Methods • What features of a tutor lead to the most learning? • Learning Decomposition • Explores different rates of learning due to different forms of pedagogical support • Close relative of Learning Factors Analysis

11. Data Mining Questions & Methods • How to extract reliable inferences about causal mechanisms from correlations in data? • Causal modeling using Tetrad

12. Data Mining Questions & Methods • And one generally useful tool for figuring out what’s going on, in any of these cases: Exploratory data analysis • Summary & visualization tools in DataShop • Tools in Excel • Clustering algorithms • Visualization packages

13. Overview Motivation for educational data mining DataShop Learning curves to improve cognitive models Past project example Conclusion Next

14. Find DataShop at learnlab.org/datashop

15. Video Intro of DataShop … • View here:

16. DataShop – Dataset Tabs Private datasets you can’t view. Email us and the PI to get access. Datasets you can view or edit. You have to be a project member or PI for the dataset to appear here. Public datasets that you can view only.

17. Analysis Tools • Dataset Info • Performance Profiler • Learning Curve • Error Report • Export • Sample Selector

18. Dataset Info • Meta data for given dataset • PI’s get ‘edit’ privileges, others must request it Papers and Files storage Problem Breakdown table Dataset Metrics 18

19. Performance Profiler Multipurpose tool to help identify areas that are too hard or easy • View measures of • Error Rate • Assistance Score • Avg # Hints • Avg # Incorrect • Residual Error Rate • Aggregate by • Step • Problem • KC • Dataset Level

20. Learning Curve Visualizes changes in student performance over time View by KC or Student, Assistance Score or Error Rate Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC

21. Error Report • Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior • Attempts are categorized by student View by Problem or KC

22. Export You can also export the Problem Breakdown table and LFA values! • Two types of export available • By Transaction • By Step • Anonymous, tab-delimited file • Easy to import into Excel!

23. Sample Selector Easily create a sample/filter to view a smaller subset of data • Filter by • Condition • Dataset Level • Problem • School • Student • Tutor Transaction Shared (only owner can edit) and private samples

24. Help/Documentation • Extensive documentation with examples • Contextual by tool/report • http://learnlab.web.cmu.edu/datashop/help Glossary of common terms, tied in with PSLC Theory wiki

25. New Features • Manage Knowledge Component models • Create, Modify & Delete KC models within DataShop • Addition of Latency Curves to Learning Curve Reporting • Time to Correct • Assistance Time • Problem Rollup & Export • Enhanced Contextual Help

26. Overview Motivation for educational data mining DataShop Learning curves to improve cognitive models Past project example Conclusion Next

27. Cognitive Modeling Challenge • Premise: High quality instructional design requires a high quality cognitive model of student thinking • Problem: Creating such a Cognitive Model is hard to get right • Hard to program, but more importantly … • A high quality cognitive model requires a deep understanding of student thinking • Cognitive models created by intuition are often wrong (e.g., Koedinger & Nathan, 2004)

28. Significance of improving a cognitive model • A better cognitive model means better: • Assessment • Instructional feedback & hints (model tracing) • Activity selection & pacing (knowledge tracing) • Better cognitive models advance basic cognitive science

29. Using student data to build better cognitive models • Cognitive Task Analysis methods • Think alouds, Difficulty Factors Assessment • General lecture Tuesday • Peer collaboration dialog analysis • TagHelper track • Data mining of student interactions with on-line tutors • DataShop track

30. Knowledge components are the “germ theory” of transfer Germs are hidden elements that carry disease from one agent to another Knowledge components are hidden elements that carry learning experiences from one situation to another -- they account for transfer

31. DataShop Supports Theory Integration • Makes micro theory concrete • Knowledge decomposability hypothesis • Acquisition of academic competencies can be decomposed into units, called knowledge components, that yield predictions about student task performance & the transfer of learning. • Not obviously true • “learning, cognition, knowing, and context are irreducibly co-constituted and cannot be treated as isolated entities or processes” (Barab & Squire, 2004)

32. Learning curves: • Student data • Statistical modelfit (blue line) • Based on micro level analysis: • learning event opportunities • Averaged across knowledge components Learning curves show performance changes over time

33. Not a smooth learning curve -> this knowledge component model is wrong. Does not capture genuine student difficulties.

34. This more specific knowledge component (KC) model (2 KCs) is also wrong -- still no smooth drop in error rate.

35. Ah! Now we get smoother learning curve. A more specific decomposition (12 KCs) better tracks nature of student difficulties & transfer from one problem situation to another (Rise near end due to fewer observations biased toward poorer students)

36. Summary: KC model as “germ theory” Without decomposition, using just a single “Geometry” KC, no smooth learning curve. But with decomposition, 12 KCs for area concepts, a smooth learning curve. Upshot: A decomposed KC model fits learning & transfer data better than a “faculty theory” of mind

37. Overview • Motivation for educational data mining • DataShop • Learning curves to improve cognitive models • Past project example • Conclusion Next

38. Past Project Example • Rafferty (Stanford) & Yudelson (Pitt) • Analyzed a data set from Geometry • Applied Learning Factors Analysis (LFA) • Driving questions: • Are students learning at the same rate as assumed in prior LFA models? • Do we need different cognitive models (KC models) to account for low-achieving vs. high-achieving students?

39. A Statistical Model for Learning Curves Predicts whether student is correct depending on knowledge & practice Additive Factor Model (Draney, et al. 1995, Cen, Koedinger, Junker, 2006) Learning rate is different for different skills, but not for different students

40. Low-Start High-Learn (LSHL) group has a faster learning rate than other groups of students

41. Rafferty & Yudelson Results 2 • Is it “faster” learning or “different” learning? • Fit with a more compact model is better for low start high learn • Students with an apparent faster learning rate are learning a more “compact”, general and transferable domain model • Resulted in best Young Researcher Track paper at AIED07

42. Overview Motivation for educational data mining DataShop Learning curves to improve cognitive models Past project example Conclusion Next

43. Lots of interesting questions to be addressed with Ed Data Mining!! • Assessment questions • Can on-line embedded assessment replace standardized tests? • Can assessment be accurate if students are learning during test? • Learning theory questions • What are the “elements of transfer” in human learning? • Is learning rate driven by student variability or content variability? • Can conceptual change be tracked & better understood? • Instructional questions • What instructional moves yield the greatest increases in learning? • Can we replace ANOVA with learning curve comparison to better evaluate learning experiments? • Metacogniton & motivation questions • Can student affect & motivation be detected in on-line click stream data? • Can student metacognitive & self-regulated learning strategies be detected in on-line click stream data?

44. Data Mining-Data Shop Offerings Data Mining Track: Tues 9:15 Using DataShop for Exploratory Data Analysis Tues 1:30 Learning from learning curves Item Response Theory Learning Factors Analysis Wed 9:30 Discovery with Models General lecture: Tues 3:30 Educational Data Mining Bayesian models of knowledge tracing Causal models with Tetrad

45. Questions?

46. Extra slides …

47. Sample tutor interactions (from 1997 version) that generated Geometry Area data set used in example of learning curves …

48. TWO_CIRCLES_IN_SQUARE problem: Initial screen

49. TWO_CIRCLES_IN_SQUARE problem: An error a few steps later

50. TWO_CIRCLES_IN_SQUARE problem: Student follows hint & completes prob