1 / 20

SceneMaker : Intelligent Multimodal Visualisation of Natural Language Scripts

SceneMaker : Intelligent Multimodal Visualisation of Natural Language Scripts. 100 Day Review Eva Hanser Supervisors:. Prof. PAUL MC KEVITT Dr. TOM LUNNEY Dr. JOAN CONDELL. School of Computing & Intelligent Systems, Faculty of Computing & Engineering, University of Ulster.

jera
Download Presentation

SceneMaker : Intelligent Multimodal Visualisation of Natural Language Scripts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SceneMaker :Intelligent Multimodal Visualisation of Natural Language Scripts 100 Day Review Eva Hanser Supervisors: Prof. PAUL MC KEVITT Dr. TOM LUNNEY Dr. JOAN CONDELL School of Computing & Intelligent Systems, Faculty of Computing & Engineering, University of Ulster

  2. Presentation Outline • Project Aims & Objectives • Literature Review • Project Proposal • Conclusion

  3. Project Aims & Objectives • Extraction of Visual and Emotional Information from Play/Film Scripts • Structuring for Visualisation Purposes • Synchronised Presentation in all Modalities(Vision, Speech, Illumination, Camera Work) • Compelling, Life-like Animations

  4. Literature Review 1. Detecting Emotions and Personality 2. Modeling Affective Behaviour of Virtual Actors 3. Multimodal Visualisation of 3D Scenes

  5. Literature Review1. Detecting Emotions and Personality • Which Emotions exist? Which are needed for realistic animation? • How to categorise all Emotions and Personality?

  6. Literature Review1. Detecting Emotions and Personality • Psychological Theories of Emotion 6 Basic, Universal Emotions (Ekman and Rosenberg, 1997) : Happiness, Sadness, Anger, Disgust, Fear, Surprise Pleasure-Arousal-Dominance (PAD) Model:Intensity Value Scale OCC Model (Ortony et al. 1988): 22 Emotion Categories arranged by Goals, Likelihood, Familiarity • Personality ModelOCEAN (De Raad, 2000): Openness, Conscientiousness, Extroversion, Agreeableness, Neuroticism • Social Roles • Story Character Roles

  7. Literature Review 1. Detecting Emotions and Personality • Fuzzy Rule Based System (Su et al., 2007) • Annotation of Scene Scripts: XML-based Scripting Languages BEAT (Cassell et al., 2001), MPML (Breitfuss et al., 2007) Tags to Control Body Language & Utterances ALMA (Gebhard, 2005), SCREAM (Prendinger and Ishiyuka, 2002) Emotion Expression Mapped to Body Language

  8. Literature Review 2. Modeling Affective Virtual Actors Gandalf, Rea (Vilhjálmsson and Thórisson, 2007), Max (Kopp et al., 2008), Greta(Pelachaud, 2005)Face-to-Face ConversationLanguage Understanding and GenerationGaze Responding to GesturesSpeaker and Listener FeedbackFace and Hand Expressions Casino Virtuell Greta Image Source: Pelachaud, 2005 Image Source: Gebhard and Schröder, 2008

  9. Literature Review 3. Visualisation of 3D Scenes • Sound and Music Effects • Cinematic Principles (Camera, Lighting, Staging) • EML : Expression Mark-up Language (De Melo and Paiva, 2006)Multimodal Expressions for Virtual Human and Environment

  10. Literature Review 3. Visualisation of 3D Scenes WordsEye(Coyne and Sproat, 2001)Static 3D scenesRestricted Language Input • CONFUCIUS (Ma, 2006)Multimodal 3D Animation of NL stories Image Source: Coyne and Sproat, 2001 Image Source: Ma, 2006

  11. Literature Review 3. Visualisation of 3D Scenes Madame Bovary(Cavazza et al., 2007) Scene Projection into RoomInteraction with Human Actor • Virtual Theatre Interface(Virtual Theatre, 2004)Different Audience ViewsStage Lighting and Positioning Image Source: Cavazza et al., 2007 Image Source: Virtual Theatre, 2004

  12. Project Proposal 1. THE PROJECT IDEA 2. SCENEMAKER ARCHITECTURE 3. SOFTWARE ANALYSIS 4. RELATION TO OTHER WORK 5. PROJECT SCHEDULE

  13. Project ProposalThe Project Idea • Affective Actors: Emotionally Influenced Body Language • Environmental Expression: Emotionally Influenced Multimedia :Music, Sound, Illumination, Timing and Camera Work • Software Prototype: SceneMaker Generates 3D Scenes from Screenplays • Web Based User Interface for Manual Manipulation • Accessible on Mobile Devices • Testing with Sample Screenplays of Different Genres

  14. Project ProposalSceneMaker Architecture

  15. Project ProposalSoftware Analysis • Constructionist Design Methodology (CDM) Constructionist AI Methodology (CAIM) (Thórisson, 2007) • Psyclone (Thórisson et al., 2004) , OpenAIR (Thórisson, 2007) Unified Messaging Format between Various Input and Output Modules/Modalities • CONFUCIUS (Ma, 2006) Natural Language Processing Animated 3D Scenes with H-Amin 3D Models • CSLU Text-To-Speech Synthesis (Sutton et al., 1998)

  16. Project ProposalRelation to Other Work • Emotion Visualisation from NL Text • Emotional Influence on Various Modalities • Mobile User Interface

  17. Relation to Other Work

  18. Project ProposalProject Schedule

  19. Conclusion Multimodal System, SceneMaker, for Automatic Visualisation of Affective Expressions in Story Scripts Assisting Directors, Drama Students, Writers and Animators to Test Ideas User Interface for Manual Adjustments Existing Systems Partially Solve Aspects of NLP, ECAs, Emotion Modeling and Multimodal Storytelling Test Basis: Screenplays of Different Genres

  20. Thank you for your attention. Questions or Comments? www.infm.ulst.ac.uk/~eva

More Related