Agile Technologies for Personalizing Instruction

1. Agile Technologies for Personalizing Instruction Faisal Ahmad, Sebastian de la Chica, Qianyi Gu, Shaw Ketels, Ifi Okoye Tammy Sumner, Jim Martin, Alice Healy, Kirsten Butcher, Michael Wright Digital Learning Sciences University of Colorado at Boulder University Corporation for Atmospheric Research This work is supported in part by an ICS Generalization Grant, and NSF awards #0537194 and #0734875

2. Central Challenge Enable personalized learning, while still supporting recognized learning goals Do it at scale • How People Learn (NRC) • Extreme Diversity (KnowledgeWorks) • Disrupting Class (Christiansen) • N=1, R=G (Prahalad)

3. Curriculum Customization www.DLESE.org CLICK Personalization Service Strandmaps.NSDL.org

4. CLICK Personalization Service • Automatically identify potential learner misconceptions by analyzing student work • Customize the selection and presentation of learning resources based on identified misconceptions • High school plate tectonics

5. Guiding Principles • Personal and intentional • Build on learner understanding • Learner control • Learning goals organize and guide • Agile technologies • Domain independent: knowledge maps for human cognition and machine reasoning • Automatic: NLP and ML • Embeddable: web services, not applications • Open: leverage existing web content

7. DEMO

8. Major CLICK Components • What should students know? • Domain knowledge map • What do they already understand? • Compare student and domain maps • What learning activities would be useful? • Select resources to address misconceptions and gaps • How to embed in learning environments? • Provide web service to application and portal developers

9. Detecting Potential Knowledge Gaps

10. Human-Centered Methodology • Expert studies to inform algorithms (Ahmad et al 2007) • Domain knowledge map creation • Student essay to student knowledge map • Knowledge gap diagnosis • Personal instruction plan generation • Expert scoring of intermediate results • Mixed-method learning study

11. Algorithms • Concept extraction (de la Chica 2008) • MEAD: multi-document summarization toolkit (Radev et al 2004) • Custom sentence scoring features: standards, gazetteer, hypertext, content word density • Eliminate redundancy, rank and choose top 5% • Student essays – lexical chains (de la Chica 2008) • Knowledge gaps – NLP and graph structure comparisons (Ahmad 2008) • Personalized information retrieval – concept matrix (Gu 2008)

12. CLICK Personalization Web Service Misconception diagnoses and knowledge map generation exposed via request types (Ahmad 2008) • Submit or remove a concept map • Construct student map from essay • Construct domain map from URLs • Get student misconceptions • Get important concepts • Get related concepts

13. Mixed-Method Learning Study • 32 undergraduates • 16 – CLICK to revise essays on Earthquakes and Plate Tectonics • 16 – control Digital Library environment • Data collected • original essays, revised essays, detailed screen capture “movies”, reflective questions, factual knowledge tests

15. Essay Content Revisions • Shallow revisions • Copying out of resource, Paraphrasing, Integrated copying, Integrated paraphrasing, Concept deletion • Deep revisions • Integrated sentence paraphrasing to create new sentence, Integrated resource paraphrasing to create new sentence, Inferencing, Generation • Codes based on Wiley and Voss 1999, Constructing arguments from multiple sources

16. Types of Content Revisions • Omissions • Gaps in student content knowledge such as missing details and missing concepts • Incorrect Statements • Coding still underway CLICK > Control: F (1, 27) = 6.490. P = 0.17 (SIG EFFECT)

17. Process Data • Exploration Episodes • Exploring learning resources and personalized feedback • Essay Episodes • Revising or working with essay • Switches • Moving between essay and exploration • Integration of content resources and developing essay • Recognizing need for outside knowledge source Exploration. CLICK>Control: F (1, 27) = 6.076, p = .02 (SIG EFFECT) Essay. CLICK>Control: F (1, 27) = 6.815, p = .015 (SIG EFFECT) Switches. CLICK>Control: F (1, 27) = 6.447, p = .017 (SIG EFFECT)

18. Conclusions • Learning - Initial CLICK results promising • Encourages deep content revisions • Promotes integration between information seeking and knowledge transformation • Students more likely to recognize that they need new knowledge, a critical element of self-directed learning • Algorithm Generalization: Promising results for “near” domain • Misconception prioritization and link generation need further work

19. Further Reading • Ahmad, F., S. de la Chica, K. Butcher, T. Sumner, and J. Martin. (2007). Towards automatic conceptual personalization tools.In Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2007): Vancouver, Canada (June 18-23), pp. 452-461. • Butcher, K. and S. de la Chica. (in press). Supporting student learning with adaptive technology: Personalized conceptual assessment and remediation. In M. Banich and D. Caccamise (Eds.), Generalization of Knowledge: Multidisciplinary Perspectives. London, England: Taylor and Francis. • de la Chica, S., F. Ahmad, J. Martin, and T Sumner. (2008). Pedagogically useful extractive summaries for science education.22nd Meeting of the International Committee for Computational Linguistics (COLING 2008). • de la Chica, S., F. Ahmad, T. Sumner, J. Martin, and K. Butcher. (2008). Computational foundations for personalizing instruction with digital libraries. International Journal of Digital Libraries. To appear in the Special Issue on Digital Libraries and Education. • Gu, Q., de la Chica, S., Ahmad, F., Khan, H., Sumner, T., Martin, J., Butcher, K. (2008). Personalizing the Selection of Digital Library Resources to Support Intentional Learning. Research and Advanced Technology for Digital Libraries, 12th European Conference, ECDL 2008, Aarhus, Denmark, September 14-19. Lecture Notes in Computer Science, pp. 244-255.

20. Examples of “Good” Concepts

22. Detecting Potential Knowledge Gaps