1 / 50

Generalizability of Goal Recognition Models in Narrative-Centered Learning Environments

Generalizability of Goal Recognition Models in Narrative-Centered Learning Environments. Alok Baikadi Jonathan Rowe, Bradford Mott James Lester. North Carolina State University. Goal Recognition in Narrative-Centered Learning Environments.

ronat
Download Presentation

Generalizability of Goal Recognition Models in Narrative-Centered Learning Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Generalizability of Goal Recognition Models in Narrative-Centered Learning Environments Alok Baikadi Jonathan Rowe, Bradford Mott James Lester North Carolina State University

  2. Goal Recognition in Narrative-Centered Learning Environments • Task: Identify the specific objective that the player is attempting to achieve • Goal recognition models enable the following: • Preemptively augmenting narrative experiences • Assessing problem solving in narrative-centered learning environments • Iteratively refining learning environment

  3. Generalization of Goal Recognition • Goal Recognition is typically very domain dependent • Plan libraries • Many domain-independent techniques are only evaluated on one domain • Research Question: Can a domain-specific goal recognition model be applied in a principled way to a new domain and achieve similar results?

  4. Related Work • Goal recognition is a restricted form of plan recognition (Carberry 2001; Kautz & Allen, 1986; Singla & Mooney, 2011) • Investigated widely in numerous domains (Blaylock & Allen, 2003; Charniak & Goldman, 1993; Lesh, Rich & Sidner, 1999) • IO-HMM approach for recognizing high-level goals in simple action-adventure game (Gold, 2010) • PHATT-based approach for behavior recognition in real-time strategy game (Kabanza, Bellefeuille & Bisson, 2010) • N-gram and Bayesian network approaches for goal recognition to support dynamic narrative planning (Mott, Lee & Lester, 2006) • MLN-based approaches (Singla and Mooney, 2011 ; Ha et al., 2011 ; Sadilek and Kautz, 2012)

  5. Outline • Goal Recognition Approach • Goal Recognition Corpora • Evaluation & Discussion • Conclusions and Future Work

  6. Markov Logic Networks (MLNs) • Statistical relational learning • Combines first-order relational reasoning with statistical learning • Input: A set of first-order predicate calculus formulae, along with weights • Formulae can be expanded into a Markov Random Field for learning and inference • The joint probability distribution is defined as: • Toolkit: Markov TheBeast(Riedel, 2008)

  7. Representation

  8. Context in Goal Recognition • Actions are not always independent • Traditional goal recognition formulation allows for all previous observations • Can lead to sparsity issues • Solution: Look for key events in the history that provide insight to the player’s context • Use the structure of the narrative to provide the context

  9. Discovery Events • Task progress is represented by a sequence of discovery events • Partial Answers to Central Questions are clues towards the solution • Provides a context for goal recognition: What has the user discovered?

  10. Discovery Event Formulae • Milestone formulae recognize which discovery events have already occurred • Uses a cardinality constraint to capture existence

  11. Outline • Goal Recognition Approach • Goal Recognition Corpora • Evaluation & Discussion • Conclusions and Future Work

  12. Crystal Island: Outbreak • 8th grade microbiology • Valve Software’s Source engine • Science mystery • Goal: Identify source and treatment of outbreak

  13. Crystal Island: Introduction • Student plays the role of a new visitor to the island. 2. Student discovers that several team members have fallen sick.

  14. Crystal Island: Gathering Information • Student gathers clues from sick team members. 4. Student asks the camp’s pathogen experts about microbiology concepts.

  15. Crystal Island: Gathering Information • Student views microbiology-themed posters. 6. Student reads books about different types of pathogens.

  16. Crystal Island: Hypothesis Testing • Student conducts tests using laboratory equipment. 8. Student interacts with the lab technician toview microscopic images of pathogens.

  17. Crystal Island: Reporting Findings 9. Student presents findings and recommended treatment to camp nurse.

  18. Corpus Collection • Eighth grade class from public middle school • 153 participants • No prior experience with Crystal Island • Played game for 1 hour, or until they were finished • 7 goals available to students (Ha et al., 2011)

  19. Crystal Island: Uncharted Discovery Upper Elementary Science Subject • 5th grade science • Standards aligned Content • Landforms • Maps, models & navigation Story • Adventurous adolescent • Shipwrecked crew • Complete quests to explore island

  20. Crystal Island: Uncharted DiscoveryVideo Video

  21. Corpus Collection • Onsite at 8 schools • 831 fifth grade students • 62% Caucasian, 14% African American, 8% Asian, 16% Other • Teacher-driven implementation in classrooms • 6 one hour sessions over 4 weeks • 12 goals available during the first 2 weeks

  22. Goal Extraction Procedure • Goal-achieving actions were identified • Actions between previous goal and current goal were labeled with current goal • Goal-achieving actions were removed

  23. Outline • Goal Recognition Approach • Goal Recognition Corpora • Evaluation & Discussion • Conclusions and Future Work

  24. Empirical Evaluation • State of the Art Baseline: • Factored model (Ha et al., 2011) • Uses MLNs to relate the current time step to the previous time step • Each model was evaluated using 10-fold student-level cross-validation • Each model was evaluated according to three metrics: • Accuracy: Measured as F1 score • Convergencerate: Percent of sequences which eventually predicted the correct goal • Convergence point: In sequences that converged, the percent of actions that had to be observed before a consistent prediction was made

  25. Baseline Model

  26. Crystal Island: OutbreakDiscovery Events Model

  27. Crystal Island: Uncharted DiscoveryDiscovery Events

  28. Experimental Results Crystal Island: Outbreak Crystal Island: Uncharted Discovery

  29. Outline • Goal Recognition Approach • Goal Recognition Corpora • Evaluation & Discussion • Conclusions and Future Work

  30. Conclusions • Goal recognition models show considerable promise for enhancing the effectiveness of narrative-centered learning environments • Encoding narrative discovery events in Markov Logic is a natural approach for representing context for student actions in goal recognition • Experimental findings from two narrative-centered learning environments suggest that narrative discovery events enhance the accuracy and convergence of state-of-the-art MLN-based goal recognition models.

  31. Future Work • Investigate combinations of discovery events • Some of the milestones may have provided more information than others • Use automated feature selection • Integrate goal recognition into a runtime environment • Can establish intuition for how accurate a model is necessary • Elicit feedback from player • Assumes goals achieved are intended • May cause some bias

  32. Research Staff EleniLobene Rob Taylor Postdoc EunyoungHa Digital Art Staff Kirby Culbertson Sarah Hegler KaroonMcDowell Graduate Students Julius GothWookhee Min Joe Grafsgaard Chris Mitchell Eunyoung Ha Jennifer Sabourin SeungLee Andy Smith Sam Leeman-Munk Undergraduate Student Stephen Cossa Collaborators Affiliated Faculty Carol Brown (East Carolina University) Roger Conner (East Carolina University) Patrick FitzGerald (Art + Design) Elizabeth Hodge (East Carolina University) James Minogue (Elementary Education) John Nietfeld (Educational Psychology) Marc Russo (Art + Design) Hiller Spires (Curriculum & Instruction) Eric Wiebe (STEM Education) Affiliated Post-Docs and Graduate Students (Art, Education, Psychology) Megan Hardy (Human Factors) Kristin Hoffman (Educational Psychology) Angela Meluso (Curriculum & Instruction) Lucy Shores (Educational Psychology) SinkyZheng (Curriculum & Instruction)

  33. Acknowledgments Support provided by the National Science Foundation under grant DRL-0822200. Additional support was provided by the Bill and Melinda Gates Foundation, the William and Flora Hewlett Foundation, EDUCAUSE, and the Social Sciences and Humanities Research Council of Canada.

  34. Goal RecognitionExample

  35. Goal Recognition Example

  36. Goal Recognition Example

  37. Goal Recognition Example

  38. Goal Recognition Example

  39. Goal Recognition Example

  40. Goal Recognition Example ?

  41. Goal Recognition Example

  42. Goal Recognition Example

  43. Goal Recognition Example

  44. Goal Recognition Example ?

  45. Goal Recognition Example

  46. Goal Recognition Example

  47. Goal Recognition Example

  48. Goal Recognition Example What is the player’s current goal?

  49. Goal Recognition Example

More Related