Modeling Student Knowledge Using Bayesian Networks to Predict Student Performance

1. Modeling Student Knowledge Using Bayesian Networks to Predict Student Performance By Zach Pardos, Neil Heffernan, Brigham Anderson and Cristina Heffernan Collaborators Sponsors Goal Predicting student responses within the ASSISTment tutoring system Student Test Score Prediction Process To evaluate the predictive performance of various fine-grained student skill models in the ASSISTment tutoring system using Bayesian networks. Bayesian Belief Network • Result: • The finer-grained the model, the better prediction accuracy. The finest-grained WPI-106 performed the best with an average of only 5.5% error in prediction of student answers within the system. • Skill probabilities are inferred from a student’s responses to questions on the system The Skill Models • The skill models were created for use in the online tutoring system called ASSISTment, founded at WPI. They consist of skill names and associations (or tagging) of those skill names with math questions on the system. Models with 1, 5, 39 and 106 skills were evaluated to represent varying degrees of concept generality. The skill models’ ability to predict performance of students on the system as well as on a standardized state test was evaluated. • The five skill models used: • WPI-106: 106 skill names were drafted and tagged to items in the tutoring system and to the questions on the state test by our subject matter expert, Cristina. • WPI-5 and WPI-39: 5 and 39 skill names drafted by the Massachusetts Department of Education. • WPI-1: Represents unidimensional assessment. Predicting student state test scores • Arrows represent associations of skills with question items. They also represent conditional dependence in the Bayesian Belief Network. • Probability of Guess is set to 10% (tutor questions are fill in the blank) • Probability of getting the item wrong even if the student knows it is set to 5% • Result: • The finest-grained model, the WPI-106, came in 2nd to the WPI-39 which may have performed better than the 106 because 50% of its skills are sampled on the MCAS Test vs. only 25% of the WPI-106’s. 2. Inferred skill probabilities from above are used to predict the probability the student will answer each test question correctly Bayesian Networks • A Bayesian Network is a probabilistic machine learning method. It is well suited for making predictions about unobserved variables by incorporating prior probabilities with new evidence. Background on ASSISTment Conclusions • ASSISTment is a web-based assessment system for 8th-10th grade math that tutors students on items they get wrong. There are 1,443 items in the system. • The system is freely available at www.assistment.org • Question responses from 600 students using the system during the 2004-2005 school year were used. • Each student completed around 260 items each. • The ASSISTment fine-grained skill models excel at assessment of student skills (see Ming Feng’s poster for a Mixed-Effects approach comparison) • Accurate prediction means teachers can know when their students have attained certain competencies. • Probabilities are summed to generate total test score. • Probability of Guess is set to 25% (MCAS questions are multiple choice) • Probability of getting the item wrong even if the student knows it is set to 5% This work has been accepted for publication at the 2007 User Modeling Conference in Corfu, Greece.