A New Approach to the Design of Interactive Storytelling Engines

1. A New Approach to the Design of Interactive Storytelling Engines TeongJoo Ong and John Leggett Texas A&M University

2. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

3. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

4. Introduction • Stories are used to convey information, cultural values, and experiences • New technologies have constantly provided increasingly sophisticated means to tell stories • Recent trend is the convergence of technology, entertainment, and art in the computer • The ancient art of storytelling can be used to efficiently convey information in our increasingly computerized world

5. Background • Interactive storytelling is a major research area • Many overlapping approaches have been used: • Immersive storytelling • Emergent storytelling • Plot-based systems • Interactive authoring of stories • Character-based systems • These approaches differ in design paradigms and technical solutions

6. Background (cont.) • Approaches used in interactive storytelling fall into the field of Narrative Intelligence • Narrative Intelligence commonly refers to research on human narratives and storytelling, and related technologies • Related research projects from this field: • CHOROS (N. M. Sgouros) • Fuzzy Cognitive Maps (FCM) (E. Maffre) • HyperSoap (J. Dakss) • Terminal Time (M. Mataes)

7. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

8. Motivation • Each of the different approaches used in interactive storytelling has its own strengths and weaknesses • A hybrid approach that can combine the strengths while reducing or eliminating the weaknesses of several different approaches could potentially be very important • Certain characteristics of the approaches will also be discarded to keep complexity and understandability in check

9. Motivation (cont.) • Our hybrid approach proposes the use of an evolutionary algorithm and story templates to construct story components • Current system states and interaction between fuzzy rules impose constraints, modifications and evaluations on the story components generated by the evolutionary algorithm • The end result is a template-based storytelling engine offering flexibility and robustness in storytelling

10. Scenarios of Use • A murder mystery that can yield different conclusions depending on the reader’s interaction with computer agents using dynamically generated story components • Sub-module of a computer game engine that generates story scripts dynamically • Interactive learning environment that presents concepts in a narrative thread that includes recombination and indeterminacy

11. Scenarios of Use in Education • Multiple experiences (points of view) through changing plots, story lines, and character relationships while adhering to a common thread • Student involvement in unfolding the story through interactive exploration of alternatives and role-playing • Encourages personalization of the story • Children as authors of their own stories

12. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

13. System Architecture authors/interactors Evolutionary Programming Subsystem (EPS) Integrated Authoring Environment (IAE) processes/generates story components user interactions evaluates/modifies parameters rendering instructions Knowledge Base sends combined output monitors/modifies parameters of rules Fuzzy Decision Based Subsystem (FDBS) Graphical/Object Library (GOL)

14. Interactive Storytelling Engine • Combines the global search and emergent characteristics of evolutionary programming and approximate reasoning of fuzzy logic to generate new story components • The solution can be any valid combination of story components in accordance with the author’s rules, system states and templates • Templates are used to describe story components and the valid contextual sets of actions, scripts, objects and agents

15. Interactive Storytelling Engine (cont.) • Story Components are the representation of a particular time step in the story. They facilitate the encoding process of the Evolutionary Programming Subsystem (EPS) • Contextual Sets are used to categorize various plot units, events, scripts, actions and characters within the story components • The Evolutionary Algorithm explores the problem space using solution instances of a particular story component • The Fuzzy Decision Based Subsystem (FDBS) imposes constraints, modifications and evaluations on the solution instances with rules and variables in the knowledge base

16. Interactive Storytelling Engine (cont.) • The Knowledge Base stores user interactions, events, fuzzy rules, discrete rules and various variables to keep track of the story’s progress • In order to ensure proper development and conclusion of the story, underlying story threads must be created by the author to act as guidelines for the engine • Underlying story threads are coded as a collection of story components so that solution instances generated by the EPS can be evaluated by the FDBS

17. Integrated Authoring Environment (IAE) • A collection of tools and interfaces for authors to create interactive stories • The collection of editing tools and library consists of: • Editing Tools • Agent Characters • Action Scripts • Story Components • The EPS • The FDBS • Library • The Graphical/Object Library (GOL)

18. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

19. User interacts with the story Knowledge base records user interactions Story Generation: Step 1 Modifies/Records user interactions Interacts Integrated Authoring Environment (IAE) Knowledge Base User

20. The knowledge base stores parameters from previous execution of the EPS Story Generation: Step 2 Stores Parameters from previous execution Evolutionary Programming Subsystem (EPS) Knowledge Base

21. Story Generation: Step 3 Provides rules/parameters for merging and checking Knowledge Base Fuzzy Decision Based Subsystem (FDBS) • Knowledge base provides various information and rules for the FDBS • Output of FDBS determines the firing strength of the various fuzzy rules

22. Story Generation: Step 4 Provides parameters and information from rules to generate story components from templates Fuzzy Decision Based Subsystem (FDBS) Evolutionary Programming Subsystem (EPS) Knowledge Base Provides templates and contextual sets used to populate story components • Information from the knowledge base and FDBS are merged by the EPS to generate new story components

23. Story Generation: Step 5 Stores story component and other necessary data for historical record Evolutionary Programming Subsystem (EPS) Knowledge Base • State information about the system is stored in the knowledge base to allow the user to return to a particular scene of the story

24. Story Generation: Step 6 Provides story component used to generate rendering instructions Knowledge Base Graphical/Object Library (GOL) • The resultant story component is sent to GOL to be interpreted into a series of instructions

25. Story Generation: Step 7 Renders various graphical objects and agents Integrated Authoring Environment (IAE) Graphical/Object Library (GOL) • Various objects and agents are displayed in the IAE based on the rendering instructions generated in the previous step • These steps are repeated until the story terminates

26. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

27. 1.234 2.345 1.11 3.211 0.121 2.3222 0.982 Encoding Function Parameters in Floating Point Floating point format • Each of the floating point numbers represent a gene in a chromosome • They can encode the unknown variables in a function (z = v + w + x) or the partition sizes of a surface

28. 1.234 2.345 1.11 3.211 0.121 2.3222 0.982 Encoding the Chromosomes Floating point format • Chromosomes are created by concatenating the floating point numbers • For the EPS, a chromosome consists of a series of floating point numbers • Each floating point number represents a particular story sequence in the template

29. Encoding the Story Components Gene in a chromosome 2 3 2 … Valid sets of action and object instances ……………………………….…… • For example, a template sequence numbered 2, with two object instances will be encoded as a floating point number 2.xx depending on the set being chosen

30. Crossover Chosen offset for crossover Chromosome A Chromosome B Chromosome A* Chromosome B* • Crossover operator allows chromosomes to exchange genetic materials • Single-point and Multi-point crossovers are supported by the EPS

31. Mutation Mutation will happen to this gene Chromosome A After mutation Chromosome A* • Mutation operator randomly changes the gene in a chromosome • Introduces variations in the population

32. Fitness Evaluation Chromosome in a population Story thread given by the author which may indicate preferences or avoidance of certain actions or object instances • The story components are evaluated against the story thread given by the author

33. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

34. The Three Little Pigs Story Example <storyline> <timestep order="1" name="introduction" sequence="and"> <set type="introduction" pickatmost="1"> <element name="act_10" addfitness="1"/> </set> </timestep> <timestep order="2" name="wolf threat" sequence="and"> <set type="wolf threat" pickatmost="1"/> </timestep> <timestep order="3" name="wolf attack" sequence="and"> <set type="wolf attack" pickatmost="2" placement="remove"/> </timestep> <timestep order="4" name="wolf's plan" sequence="and" loop="true" condition="{isPigStillAvailable}==true OR {isWolfDead}==true"> <set type="wolf's plan" pickatmost="1" placement="noremove"> <element name="act_13" addfitness="2"/> </set> </timestep> </storyline>

35. The Three Little Pigs Example (cont.) Provides background description of the story to the reader. It also determines the template sets that will be used in the rest of the story by selecting a particular variant of the story Introduction The wolf actor shows up. Depending on how the pigs react, certain action instances could dictate that the pigs are paralyzed with fear, and thus eaten by the wolf which can lead to an early conclusion to the story Wolf’s Threat Wolf’s Attack While the wolf is still alive or a pig is available for the wolf, these template sets provide various alternatives for the wolf to attack the pigs and the pigs’ response to those attacks Wolf’s Plan The wolf may have been killed or cooked by one of the pigs, or all of the pigs were eaten by the wolf, yielding a conclusion Conclusion

36. Overview • Introduction and Background • Motivation and Scenarios of Use • Architecture of the Interactive Storytelling Engine • Story Generation • Encoding and Evaluating the Story Components • Story Example • Evaluation Criteria, Contributions and Conclusions

37. Evaluation Criteria • Quality of story is based on user’s evaluation of the events, deviation from the author’s story line, and congruence of the transitions in the story • Responsiveness, performance and execution speed of the engine • Stability and consistency in the knowledge base

38. Contributions • Simple and integrated environment for authors to create interactive and recombinant stories • Materials used in one story can be reused in a related story setting • A useful educational tool when rules are integrated into the engine to reflect certain physical or social phenomena

39. Conclusions • The hybrid approach used in storytelling is not perfect since it lacks the ability to automatically generate story templates out of a knowledge base • Gives authors the power to have full control of the cyborg authoring environment and provides tools to author various aspects of the interactive stories