Open community authoring of worked example problems

1. Open community authoring of worked example problems Turadg Aleahmad, Vincent Aleven, Robert KrautHuman Computer Interaction Institute, Carnegie Mellon University turadg@cmu.edu Authoring Interface Quality of Open Access to Authoring Abstract Conclusions and Future Work • Important precondition for community-based authoring seems to be met: volunteers contribute high-quality math examples • Separating the wheat from the chaff required little effort • Both professional educators and amateurs contributed a large portion of useful materials. • Contributions from math teachers were not superior to those from others • Math teachers did write the best problem statements but amateurs wrote the best solutions • The student profile feature of the interface successfully drew out personalized resources • On every attribute the profile increased the likelihood of targeting it Open collaborative authoring systems such as Wikipedia are growing in use and impact. How well does this model work for the development of educational resources? In particular, will sufficient numbers of volunteers participate and contribute materials of high quality? Further, will they create resources that are targeted to students’ specific learning needs and, in order to engage students more, their personal characteristics? We conducted an experiment to explore these questions through the use of a novel prototype tool for community authoring of worked examples for a specific math skill. Participants were professional teachers (math and non- math) and amateurs. Participants were randomly assigned to the standard tool, or to an enhanced version that prompts authors to create materials for a specific (fictitious) student. We find that while there are differences by teaching status (math teacher, other teacher or amateur), all three groups made contributions of worth (as judged by two independent teacher raters) and that targeting a specific student leads contributors to author materials with greater potential to engage students. The experiment suggests that community authoring of educational resources is a feasible model of development and can enable new levels of personalization. • Of the raw submissions made, half were incompliant but also trivial to filter by simple automated methods • Of the remaining, a novice and a veteran teacher were able to rate each of them on three attributes in less than a minute each • About 1/10th were ready to help students learn without needing any modification • Statements were the highest quality components and solutions were the most difficult parts to author well • Math teachers were best at authoring problem statements • Amateurs authored the best worked solutions Example profile Problem statement Illustrating solution Work to solution Motivation • Open collaborative development can work as well as centralized development by experts (Gilles 2005) • Traditional model-based tutors cost much time from skilled experts (Murray 1999) • Worked examples complement tutoring systems (Schwonke et al. 2007). • Personalization improves student engagement and test scores (Ku and Sullivan 2002 , Anand and Ross 1987) Literature Cited Aleahmad, T,. Aleven, V,. and Kraut, R. (in press) Open community authoring of worked example problems. In Proceedings of the International Conference of the Learning Sciences. Aleahmad, T,. Aleven, V,. and Kraut, R. (under review) Open community authoring of targeted worked example problems . Proceedings of Intelligent Tutoring Systems 2008. Anand, P. G., & Ross, S. M. (1987). Using Computer-Assisted Instruction to Personalize Arithmetic Materials for Elementary School Children. Journal of Educational Psychology, v79 n1 p72-78 Mar 1987. Giles, J. (2005). Internet encyclopaedias go head to head, Nature, 438(7070), 900-901. doi: 10.1038/438900a. Ku, H., & Sullivan, H. (2002). Student performance and attitudes using personalized mathematics instruction. Educational Technology Research and Development, 50(1), 21-34. Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, pp. 98-129. Schwonke, R., Wittwer, J., Aleven, V., Salden, R., Krieg, C., & Renkl, A. (2007). Can tutored problem solving benefit from faded worked-out examples. European Cognitive Science Conference, 23–27. Count of submissions in each quality classification Illustrating solution steps Explanation of work Effect of Student Profiles on Authoring • All features of the profile display were accounted for in the problems submitted. • Discussing a male very much more likely than a female • Female profile brings likelihood to on par with males Materials and Methods • Mixed methodology experiment drew upon volunteers over the web to create worked example problems • Contributions were rated by machine and by two experts • Three components of worked example evaluated by the criteria below: Statement, Work and Explanation (alpha=0.61 for statement and 0.78 for all) Sample Submission Probabilities of authoring matching an attribute (†p<.10 *p<.05 **p<.001 ) Acknowledgements Thanks to the ASSISTments project team. Thanks for Flickr user jenrock for photos used in personas. The research reported  here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B040063 to Carnegie Mellon University. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through "Effective Mathematics Education Research" program grant #R305K03140 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education. • High and low reading levels differed by almost a grade level • Submissions for profiles with high general math skill level were one third less likely to make use of simple 3-4-5 triangle problems • Submissions analyzed across teacher status of participant: math teacher, other teacher, or amateur • Experimental manipulation in which the subjects were presented with a student profile and told to provide instruction for that student Correspondence of verbal and math skill levels with the authoring interface