Conclusion

1. The ASSISTment Project Blends ASSISTance and assessMENT By Mingyu Feng In collaboration with Neil Heffernan, Joseph Beck & Kenneth Koedinger Collaborators Sponsors Student proficiency Assistance metrics How does ASSISTment work Background on ASSISTments • Students will first be administered a problem (main question). If they got it right, they will get a new one; otherwise, they are provided with a small “tutoring” session where they are forced to answer a few questions (called scaffolding questions) that break the problem down into steps. • On-demand hint messages and context sensitive buggy messages • The ASSISTment System is a web-based system that attempts to blend both computer-based tutoring and assessing. It offers instruction to students while providing a more detailed evaluation of their ability to teachers. • More than 3000 Worcester middle school students used ASSISTments as part of their math class. Goal I: Assess students accurately Goal II: Teach students effectively • Q: How do we predict student end-of-year exam score? • Collect data while students working in ASSISTment • Student proficiency score – how student performed on main questions • Assistance metrics – How student interacted with the system • # attempts • Help seeking behavior • Speed • performance on scaffolding questions • On-taskness • Build backwards linear regression model to predict real MCAS score A sample learning opportunity group (Skill: Area) The Rasch model: • Conclusion • Our prediction model did a good job at predict 8th grade math proficiency. It can be used to estimate 10th grade score fairly well, too. But we are disappointed that it can not do better than MCAS even we got more data. • Students learn from ASSISTments while being assessed. • Follow-up questions: • How much do students learn? • Will this raise state test score? Q: Are we doing a good job at predicting? • Q: Do students learn in ASSISTment? • Item response theory (IRT) approach • IRT model assumes no learning during tests • Students who learn will be underestimated • Modeling process • Train Rasch model and compute residuals • Do student analysis and item analysis to compare residuals at different opportunities • Results • Students learned mostly from 1st opportunity • Learning slows down afterwards predicts r = 0.731 MCAS8 MCAS10 Predicted score r = 0.874 r = 0.729 MCAS8’ Feng, M., Heffernan, N.T. (2007). Towards live informing and automatic analyzing of student learning: Reporting in the Assistment system. Journal of Interactive Learning Research (JILR) 18(2)  pp. 207-230. Chesapeake, VA: AACE. Feng, M., Heffernan, N. & Koedinger, K. (2006a). Predicting state test scores better with intelligent tutoring systems: developing metrics to measure assistance required. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eighth International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 31-40. Feng, M., Heffernan, N. T., & Koedinger, K.R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. Pardos, Z., Feng, M. & Heffernan, N. T. & Heffernan-Lindquist, C. (2007) Analyzing fine-grained skill models using bayesian and mixed effect methods.  In Luckin & Koedinger (Eds.) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press. Razzaq, Feng, Heffernan, Koedinger, Nuzzo-Jones, Junker, Macasek, Rasmussen, Turner & Walonoski (2007). A Web-based authoring tool for intelligent tutors: Assessment and instructional assistance. In Nadia Nedjah, et al. (Eds.) in Intelligent Educational Machines.  Intelligent Systems Engineering Book Series. Springer. • Student proficiency score correlates significantly with MCAS8 (r = .731) • The interaction data matters a lot. A better prediction is obtained using assistance metrics (r = .864, reliably higher than .731) • Our model did as well as MCAS on predicting exam scores two years later. Feng, Mingyu, Beck, J,. Heffernan, N. & Koedinger, K. (In preparation) Can an intelligent tutoring system predict math proficiency as well as a standardized test? To be submitted to the 1st International Annual Conference on Education Data Mining. Montréal 2008. Feng, Mingyu, Heffernan, N. (In preparation). Do students learn within ASSISTments? To be submitted to the 1st International Annual Conference on Education Data Mining. Montréal 2008.