Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models

1. Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models Yun Huang, University of Pittsburgh YanboXu, Carnegie Mellon University Peter Brusilovsky, University of Pittsburgh

2. This talk… • What? More effective student modeling and performance prediction • How? A novel framework reducing content model without loss of quality • Why? Better and cheaper • Reduced to 10%~20% while maintaining or improving performance (up to 8% better AUC) • Beat expert based reduction

3. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

4. Motivation • In some domains and some types of learning content, each content problem (item) is related to largenumber of domain concepts (Knowledge Component, KCs) • It complicates modeling due to increasing noise and decreasing efficiency • We argue that we only need a subset of the most important KCs!

5. Content model • The focus of this study: Java • Each problem involves a complete program and relates to many concepts • Original content model • Each problem is indexed by a set of Java concepts from ontology • In our context of study, number of concepts per problem can range from 9 to 55!

6. An example of original content model • class definition • static method • public class • public method • void method • String array • int type variable declaration • int type variable initialization • for statement • assignment • increment • multiplication • less or equal • nested loop

7. Challenges • Select best concepts to model problems • Traditional feature selection focuses on selecting a subset of features for all datapoints(a domain). item level not domain level

8. Our intuitions of reduction methods Intuition 1 “for statement” appears 2 times in this problem -- it should be important for this problem! • Three types of methods from different information sources and intuitions: “assignment” appears in a lot of problems -- it should be trivial for this problem! Intuition 2: When “nested loops” appears, students always get it wrong -- it should be important for this problem! Intuition 3: Expert labeled “assignment”, “less than” as prerequisite concepts, while “nested loops”, “for statement” as outcome concepts --- outcome concepts should be the important ones for current problem!

9. Reduction Methods • Content-based methods • A problem = a document, a KC = a word • Use IDFand TFIDF keyword weighting approach to compute KC importance score. • Response-based Method • Train a logistic regression (PFA) to predict student response • Use the coefficient representing the initial easiness (EASINESS-COEF) of a KC. • Expert-based Method Use only the OUTCOME concepts as the KCs for an item.

10. Item-level ranking of KC importance • For each method, we define SCORE function assigning a score to a KC in an item • The higher the score, the more important a KC is in an item. • Then, we do item-level ranking: a KC's importance can be differentiated • by different score values, or/and • by its different ranking positions in different items

11. Reduction Sizes • What is the best number of KCs each method should reduce to? • Reducing non-adaptively to items (TopX): Select x KCs per item with the highest importance scores. • Reducing adaptively to items (TopX%): Select x% KCs per item with the highest importance scores

12. Evaluating Reduction on PFA and KT • We evaluate by the prediction performance of two popular student modeling and performance prediction models • Performance Factor Analysis (PFA): logistic regression model predicting student response • Knowledge Tracing (KT): Hidden Markov Models predicting student response and inferring student knowledge level *We select a variant that can handle multiple KCs.

13. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

14. Tutoring System Collected from JavaGuide, a tutor for learning Java programming. Java code Students give values for a variable or the output Each question is generated from a template, and students can try multiple attempts

15. Experimental Setup • Dataset • 19, 809 observations, about 69.3% correct • 132 students on 94 question templates (items) • A problem is indexed into 9 ~ 55 KCs, 124 KCs in total • Classification metric: Area Under Curve (AUC) • 1: perfect classifier, 0.5: random classifier • Cross-validation: Two runs of 5-fold CV where in each run 80% of the users are in train, and the remaining are in test. • We list the mean AUC on test sets across the 10 runs, and use Wilcoxon Signed Ranks Test (alpha = 0.05) to test AUC comparison significance.

16. Reduction v.s. original on PFA • Flat (or roughly in bell shapes) with fluctuations • Reduction to a moderate size can provide comparableor even better prediction than using original content models. • Reduction could hurtif the size goes too small (e.g. < 5), possibly because PFA was designed for fitting items with multiple KCs.

17. Reduction v.s. original on KT • Reduction provides gain ranging a much bigger span and scale! • KT achieves the best performance when the reduction size is small: it may bemore sensitive than PFA to the size! • Our reduction methods have selected promising KCs that are the important ones for KT making predictions!

18. Automatic v.s. expert-based (OUTCOME) reduction method • IDF and TFIDF can be comparable to or outperform OUTCOME method! • E-COEF provides much gain on KT than PFA, suggesting PFA coefficients can provide useful extra information for reducing the KT content models. (+/−: signicantly better/worse than OUTCOME, : the optimal mean AUC)

19. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

20. “Everything should be made as simple as possible, but not simpler.” -- Albert Einstein

21. Conclusion • “Content model should be made as simple as possible, but not simpler.” • Given the proper reduction size, reduction enables prediction performance better! • Different model reacts to reduction differently! • KT is more sensitive to reduction than PFA • Different models achieve the best balance between model complexity and model fitin different ranges • We are the first to explore reduction extensively! • More ideas for selecting important KCs? • Larger datasets? • Other domains?

22. Acknowledgement • Advanced Distributed Learning Initiative (http://www.adlnet.gov/). • LearnLab 2013 Summer School at CMU(Dr. Kenneth R. Koedinger, Dr. Jose P. Gonzalez-Brenes, Dr. Zachary A. Pardosfor advising and initiating the project)

23. Thank you for listening !