Searching for Patterns: Sean Early PSLC Summer School 2007

1. Searching for Patterns: Sean Early PSLC Summer School 2007 Question: Which is a better predictor of performance in a cognitive tutor, error rate or assistance score? Method: Logistic Regression using LFA preliminary analysis to determine assistance score Outcome: Deeply aggregated data shows that both are good predictors (p<.001); Complete case analysis incomplete due to hurdles [data scrubbing, software, insufficient memory]

2. Searching for Patterns Sean Early University of Southern California PSLC Summer School 2007 Data Mining Track June 22, 2007

3. Background • Assistments data set from 2004-2005 school year (Heffernan) • N=179 • MCAS 39 step table • Data covers one year of 8th grade math online tutoring system results

4. Question: Which is a better predictor of performance in a cognitive tutor, error rate or assistance score?

5. Method • Logistic Regression using LFA preliminary analysis to determine assistance score LFA represents an extension of Rasch modeling such that the probability of a correct response is equal to the underlying ability of the student, the difficulty of the knowledge component, and the number of opportunities that the student has had to respond correctly to that knowledge component

6. Model The Learning Factors Algorithm can be represented as: ln(p/1-p)= αi + βj + Γjt-1

7. Results • Preliminary results using standard logistic regression showed no significant difference in the predictive power of errors and requests for assistance • These variables were significantly correlated with each other, and with post-test such that the fewer errors or hints, the better the final performance

8. Unanswered Questions • The simple logistic regression model leaves much to be desired. It fails to account for the alpha term (where the student’s initial skill level was) or for the rate of growth across time. Both of these questions can be addressed through the LFA model, as the alpha term represents an intercept term while the gamma value represents opportunities to learn before mastery.

9. Data Mining Hurdles I Didn’t See Coming: or, What I Wish I Knew Then • Data scrubbing (deleting repeated cases due to multiple knowledge components in one problem) needs to happen before creating aggregate data sets • Excel is more powerful than I thought (Pivot Tables may indeed be the best thing MS has given us) • 65,000 cases of information applied to a logistic regression algorithm makes my computer seize up with a case of the “not todays” • Codebooks are our friends

10. Final Thoughts • For those of us who believe in the mastery v. performance orientation framework, please take my lack of a more concrete product to share as evidence of my mastery orientation. I feel as though I learned a great deal this week. I also feel that with about 10 more hours of work and a more powerful machine at my side, I just may be able to answer my initial question in a way that captures some of the more interesting subtleties in the data set.