Evolutionary Development of a Multimodal Recommender System

1. Evolutionary Development of a Multimodal Recommender System Pontus Johansson ponjo@ida.liu.se Linköping University Graduate School of Language Technology

2. Outline • Evolutionary dialogue system development • Method • Architecture • Re-Use • MadFilm – a personalized multimodal movie recommendation system

3. Dialogue System Development • Dialogue systems are hard to build: • Knowledge-intensive • Advanced interfaces – no standards • We need development methods that allow for: • Simplicity • Flexibility • Changing requirements: • functionality • usability • Re-use • between generations • between systems • Fast release of runnable (testable) prototypes

4. An Iterative Method • Inspired by agile (lightweight) methods • as sufficient, understandable, and simple as possible • maintaining overview and focus of project • design and coding in sync • Tools and Framework development

5. DS theory DS requirements specification Conceptual design DS design DS framework Other modules Framework customisation DS module Method Overview

6. Development Space DS Framework Customisation Tools X X X Framework templates X X X Code patterns X X X DS Design Knowledge representation Modularisation Interfaces

7. Dialogue System Capabilities • Atomic request handling: identify and handle user requests that require only a direct and single system response, (Q/A) • Dialogue history modeling: take the dialogue history into account • Sub-dialogue control: allow for more advanced dialogue features, e.g.: • consider what dialogue strategy to use • clarifications

8. DS Capabilities Sub-dialogue control X X X X X X X X X X X X X X X X X X Dialogue history Atomic request handling X X X X X X X X X Extending the Space DS Framework Customisation Tools Framework templates Code patterns DS Design Knowledge representation Modularisation Interfaces

9. Demo Dialogue Capabilities: NokiaTV Q/A: U: List actors in Star Wars S: [Marc Hamill, Harrison Ford, …] U: Who directed Star Wars? S: George Lucas • Iteration 1: Q/A-system • Iteration 2: Dialogue history • Iteration 3: Sub-dialogue Dialogue history: U: Who is starring in the Bond movie? S: Sean Connery U: And who directed it? S: Guy Hamilton Sub-dialogue: U: Show all movies today S: There are 30 shows that matches. Do you wish to see them all? U: Nah, only the ones after nine pm S: [TABLE]

10. An Incremental Architecture • Supporting the iterative method with an incremental framework… • Micro architecture (design pattern): • Phase Graph Processor (PGP) • Hosting developed tools, libraries, and demo applications as Open Source • nlpFarm (SourceForge)

11. Phase Graph Processor Pattern private void init() { Phase interp = new InterpPhase(); Phase pi = new PragmaticPhase(); … Phase memory = new MemorizationPhase(); Phase[] ordering = { interp, pi, dialog, ks, gen, memory}; PurePhaseGraph graph = new PurePhaseGraph(ordering); this.pp = new PhaseProcessor(graph); } public class InterpPhase extends AbstractPhase { /** * Reads the user utterance from the palette, and decides on the task * (rec or info), * which is written to the palette. */ public void evalAdapter(java.lang.Object inputState, PhaseExit exit) { State state = (State) inputState; String utterance = palette.getInKVSet().getString("userUtterance"); Parser p = new NuanceParser(".Top"); //grammar for detecting task Object interp = p.parse(utterance); KVSet kv = (KVSet)interp; palette.setInterp(kv); exit.next(palette); } }

12. Tools/Frameworks JavaChart Chart-parser with Java API Guidia Framework for GUI of dialog system Quaks Framework for connection NLP/knowledge-source Modi Generic GUI for wireless dialogue systems Demos BirdQuest NLP information system about birds (in Swedish) TvGuide NLP information system about tv (in English) Libraries nlpLib Generic NLP library (e.g. the PGP pattern) JGrLib Grammar library JavaConLib Context-based disambiguation library JBricksLib Generic non-NLP-specific library

13. Four Systems • NokiaTV • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • TV shows • TvGuide • Information-providing (typed interaction) • Question/Answer (atomic requests) • TV shows and movies (merging two independent knowledge sources) • BirdQuest (v1-2) • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • Scandinavian birds • AdFilm/MadFilm • Information-providing and recommending (spoken interaction) • Dialogue history and sub-dialogue • Movies • Multimodality (speech + direct manipulation)

15. Four Systems • NokiaTV • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • TV shows • TvGuide • Information-providing (typed interaction) • Question/Answer (atomic requests) • TV shows and movies (merging two independent knowledge sources) • BirdQuest (v1-2) • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • Scandinavian birds • AdFilm/MadFilm • Information-providing and recommending (spoken interaction) • Dialogue history and sub-dialogue • Movies • Multimodality (speech + direct manipulation)

17. Four Systems • NokiaTV • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • TV shows • TvGuide • Information-providing (typed interaction) • Question/Answer (atomic requests) • TV shows and movies (merging two independent knowledge sources) • BirdQuest (v1-2) • Information-providing (spoken interaction) • Dialogue history and sub-dialogue • Scandinavian birds • AdFilm/MadFilm • Information-providing and recommending (spoken interaction) • Dialogue history and sub-dialogue • Movies • Multimodality (speech + direct manipulation)

19. la = linguistic analysis th = task handling g = generation dm1-3 = dialogue management dk = domain knowledge mgmt pi = pragmatic interpretation m = memorization um = user modeling rh = recommendation handling Phase Graph Re-Use

20. MadFilm User Study • Corpus collection • 24 movie recommendation dialogues between: • 1 recommender • 1 customer • 2684 utterances (7.5 hours) • 112 utterances/dialog • Equipment & Setting • Information source (Imdb) • Paper sheets for note-taking • Living room home environment

21. Analysis • Categorization of dialogue phenomena • interleaving information and preference requests • Characterization of object management in the domain • search and organization for/of movie objects • Dialogue distillation: construction of use-scenarios • recommendation dialogue structure

22. Interleaving Information and Preference Requests U: Who is starring in Gladiator? S: <list of actors> U: And who directed it? S: Ridley Scott … • Information request dialogue • user-driven • strategy re-used from e.g. NokiaTV, TvGuide • Preference/Recommendation dialogue • system-driven • Combination • mixed-initiative • seamless integration S: What kind of movie do you feel like? U: A thriller would be nice S: Do you have a favorite thriller? U: Fight Club S: Ok. Have you seen Snatch? … S: What kind of movie do you feel like? U: A comedy would be nice S: Ok. Do you have a favorite comedy? U: Lock stock and two smoking barrels S: Ok. Have you seen Snatch? U: What is it about? S: <plot information> U: Oh yes, that one is great

23. Dialog Distilling: Recommendation Dialogue Flow Welcome to MadFilm. <welcome text> MadFilm is ready for your request. Any Node Recommendation request Don’t know Information request [Help dialog] Do you have a specific genre in mind? [Information dialog] [Similar TITLE] [With PERSON] GENRE Don’t know Can you please say the title of a good movie in that genre? Could you tell me one of your favorite movies? Yes No Yes No Ok. Do you like any actor/actress in particular? Have you seen this movie? Yes No Yes No What did you think about it? ACTOR acts in the following movie. Any good among these? Do you want to put it on your to see list? Rating Yes No Yes No Do you want a new recommendation? Yes No

24. The MadFilm Domain • Principal domain objects: movies • properties: information (i) and preference (p) • title (i, p) • year (i) • genres (i, p) • actors/actresses (i, p) • directors (i, p) • plot summary (i)

25. Modality Integrator Imdb Manager Collaborative Filtering Server ”Gesture” Thread (GUI) Speech Thread (GUI) Ratings DB Internet Movie DB (www.imdb.com) The MadFilm System Components State Phase Graph (Control Structure) Phase Layer Nuance 8 Speech Recognition Server Speech Grammars

26. Demo MadFilm

27. Resources • MadFilm project • http://www.ida.liu.se/~ponjo/adfilm/ • http://www.ida.liu.se/~ponjo/research/publications.html • nlpFarm Open Source project • http://nlpfarm.sourceforge.net/ • BirdQuest online (Swedish) • http://maj4.ida.liu.se:8080/birdquest/ • NLPLAB, Linköping University • http://www.ida.liu.se/labs/nlplab/