The Q-matrix method: A new artificial intelligence tool for data mining

1. The Q-matrix method: A new artificial intelligence tool for data mining Dr. Tiffany Barnes Kennedy 213, tbarnes2@uncc.edu PhD - North Carolina State University

2. Overview • Introduction • Adaptive Teaching and Data Mining • Student Model Extraction • Conclusions & Future Work The Q-matrix method

3. Research challenge • Turn the computer into a private tutor • Diagnose and correct misconceptions • Diagnosis tolerates careless errors & guesses • Build a scientific approach to improving computer based education • Build in fault tolerance, robustness • Optimize for student performance • Optimize teaching strategies for effectiveness The Q-matrix method

4. The Problem • Students take a tutorial and quiz online • Determine what students know • Redirect students to new/repeat material The Q-matrix method

5. Adaptive Tutorial Flow Determine concept state Question Engine Concept Model Select new material Ask questions Diagnostic Engine Teaching Strategy student Determine learning path Student responds

6. Behavior Known Contents Unknown Data mining for knowledge Assume contents affect behavior student The Q-matrix method

7. Knowledge & student model Concepts Student concepts Tutorial questions Student responses Goal: Mine to extract student concepts The Q-matrix method

8. References: Data Mining Server @ http://dms.irb.hr/tutorial Data mining & adaptive teaching • Problem understanding • Effective direction of student learning • Data understanding • Data from online tutorials • Data preparation • Select relevant variables • Modeling: Q-matrix, cluster, factor • Evaluation of results • Misconceptions diagnosed? The Q-matrix method

9. match Q-matrix 00011 10010 How the model works Student response 11100 Predicted responses: 01100 Err: 1 01101 Err: 2 11100 Err: 0 11111 Err: 2 Tutorial & Questions 11100 Err: 0 Teaching Strategy Student understands Concept 1 but not 2. The Q-matrix method

10. How the model works-2 • Concept state – a bit string that describes understanding • Concept state 01: understands concept 2 but not concept 1 • Q-matrix: concepts v. questions • Each state has an “ideal response vector” computed from Q-matrix The Q-matrix method

11. Binary Q-matrix example q1 q2 q3 q4 q5 Con1 0 0 0 1 1 Con2 1 0 0 1 0 Concept State IDR 00 01100 01 11100 10 01101 11 11111 The Q-matrix method

12. Research questions • Are Q-matrix models interpretable? • What factors affect Q-matrix extraction? • How well does the Q-matrix method compare with other data mining methods? The Q-matrix method

13. References: Brewer 1996. NCSU Masters Thesis. Results on simulated students • Brewer tested 2 Q-matrix extraction methods based on ideal students + noise in ideal response vectors • Q-matrix method needs few students for high noise tolerance, factor analysis needs many more The Q-matrix method

14. Student model extraction • Q-matrix, factor, and cluster models • Compared for error on student data sets • Q-matrix and cluster also compared by maps and by cluster convergence The Q-matrix method

15. Q-matrix model • Assumes concepts underlie questions • Students are in “concept states” C: • C1 = 1 understands concept 0 • C2 = 0 doesn’t get concept 2 • For each state, compute IDR • Assign students to state with closest IDR The Q-matrix method

16. Q-matrix creation Until convergence criterion met: • Increment number of concepts • Create random q-matrix • Fill concept states & compute error • Vary q-matrix • Fill concept states & compute error • Repeat steps 4-5 until error not improving • Repeat steps 2-6 to avoid local minima The Q-matrix method

17. Factor analysis model • Each tutorial question is a variable • Create covariance matrix for vars • Derive eigenvectors/values to explain most of the variance in the covar matrix • Assumes that linear combinations of the variables will be able to explain the vars • Eigenvectors ROTATED The Q-matrix method

18. Cluster analysis model • Answer vectors as points in plane • Iterate until convergence: • Choose random seed from data set • Assign vectors to nearest seed • Set new seeds to cluster medians • Chooses random seeds, assigns vecs to closest seed, set new seed to cluster median • Similar to q-matrix except seeds are Ideal Response Vectors The Q-matrix method

19. Q-matrix vs. Factor Analysis • CFA generated 4 factors/matrix • Compared to q-matrix with 4 concepts • Factor matrix converted to 0/1 • Threshold of 0.3 -> 1, less -> 0 • Factor matrix used as q-matrix • Error computed for both • Q-matrix performed significantly better (at least 19% less error/stud) on all 14 problems • Smallest diff in performance when large amount of variance in student answers The Q-matrix method

20. Q-matrix and factor errors per student

21. Ratio of q-matrix to factor error and relative # of distinct observations

22. Q-matrix vs. Cluster Analysis • Cluster Analysis does not map to q-matrix as factor anal. does • However, q-matrices do form clusters of students in the same concept state • Ran Cluster Analysis with same number of clusters as q-matrix • Similar clusters generated by both The Q-matrix method

23. Clustering comparisons • Determine equivalent concept state & cluster groupings (by largest overlap) • These are in BOLD • Count elements NOT in overlaps • Overall diff = total NOT overlapping / total elements The Q-matrix method

24. Proof 8 Q-matrix Cluster Comparison 6/15 clus different 105,205,305 Con2-35 231 Con3-777 274 14,15 16 Con 0-4 Con 1-444 402,441,446,622 546,646,744

25. Differences in cluster overlap Ratio of different to total cluster assignments

26. Q-matrix vs. Cluster Analysis 2 • Each cluster has a “seed” • Distances from seeds determine cluster membership • For each cluster, summed differences between seeds & answer vectors • Total error less than that of q-matrix clusters for all experiments The Q-matrix method

27. Q-matrix vs. Cluster Analysis 3 • Why is total error less for clusters? • Because we force the IDRs in q-matrix method to be based on concepts • This yields higher errors but more help in directing teaching strategies The Q-matrix method

28. Q-matrix v. Clusters Summary • If we used cluster results, how would we determine what to do for each student after the analysis? • Cluster and q-matrix analyses could be used together for large data sets. • Important: student outcomes The Q-matrix method

29. Conclusions • Full automation of economically expandable adaptive teaching system • Method for diagnosis of misconceptions • Q-matrix model interpretable by humans • Q-matrix outperforms factor analysis in student modeling • Q-matrix forms clusters similar to those in cluster analysis The Q-matrix method

30. Future Work • Any lesson can be augmented with diagnostic engine • Different teaching strategies can be compared • Apply Q-matrix method to benchmark data mining datasets • Perform detailed time analysis and determine improvements • Cross-validation tests to determine accuracy of model • Missing data adaptations The Q-matrix method

31. Thank you! • Email: tbarnes2@nuncc.edu • This work was partially supported by NSF grants #9813902 and #0204222. The Q-matrix method

32. How the model works-2 • Student takes quiz • Assigned to state with nearest IDR • Error determined from difference between IDR & response, Q-matrix • Q-matrices varied until error over all students is minimized The Q-matrix method

33. References: Birenbaum, et al. 1993, Tatsuoka 1983. Manual concept mapping • Expert analysis of algebra tasks into rules • Evolved into Q-matrix • Relationship between questions & concepts • Applications: • Student assessment • Group performance measure • Finding new rules (student innovations) The Q-matrix method

34. References: Hubal 1992. NCSU Masters Thesis. Prediction of student data • Hubal found that randomly generated rules were better predictors of student data than Tatsuoka’s Q-matrix • This suggests that student data should be used to generate dynamic Q-matrices • Mining for what the students know! The Q-matrix method

35. Knowledge Assessment • Comparison with expert models • Remediation • Tutorial effectiveness The Q-matrix method

36. Remediation • Analyze student states and apply a teaching strategy to direct next step • Process: Find the least-understood concept, and have student retake the first lesson related to that concept The Q-matrix method

37. Remediation results • Self-guided choices compared with q-matrix choices • Less than half of self-guided students chose differently • Exam performance: q-predicted equal or worse than self-chosen • Conclusion: remediation at least as good as student remediation The Q-matrix method