Software Development Processes , Reuse and Knowledge Sources in Spoken Dialog System

1. Software Development Processes, Reuse and Knowledge Sourcesin Spoken Dialog System 11761 Dialog Seminar Rohit Kumar Friday, Dec 1, 2006

2. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • Iterative Method • (DISC) Dialogue Engineering Life Cycle • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Template based • Tools and Methods required • Some ideas Also included: Full list of readings

3. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • Iterative Method • (DISC) Dialogue Engineering Life Cycle • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Template based • Tools and Methods required • Some ideas Also included: Full list of papers surveyed

4. Spoken Dialog System development is costly and time consuming • Two optimistic hypothesis (Allen et. al., 2000) • Practical Dialogue Hypothesis “The conversational competence required for practical dialogues, while still complex is significantly simpler to achieve than general human conversational competence” • Domain Independence Hypothesis “Within the genre of practical dialogue the bulk of complexity in the language interpretation and dialogue management is independent of the task being performed” • Agreeable ?! (I say “Yes”) Some basic ideas / Ground rules

5. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • Iterative Method • (DISC) Dialogue Engineering Life Cycle • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Template based • Tools and Methods required • Some ideas Also included: Full list of papers surveyed

6. Dialogue System Frameworks • Can be considered a consequence or a partial proof of the domain independence hypothesis • Various Frameworks • TRIPS • Adriane / Ravenclaw / Olympus (CMU) • Communicator (SRI, CMU, U. Colorado, …) • Speech Builder (MIT) • CSLU Rapid Application Developer • Industrial Frameworks: Voice Platforms (IBM, Microsoft, Nuance…) • Tutorial Dialog: TuTalk • Many more… Rapid Prototyping of Dialogue Systems

7. Rapid Prototyping of Dialogue Systems

8. Rapid Prototyping of Dialogue Systems

9. Rapid Prototyping of Dialogue Systems

10. (Denecke, 2002) • Able to develop very simple dialog systems with 8 – 10 hours of effort using the Adriane framework • Domain independent Interaction Patterns implemented • Question Undo, Correction, State • Development comprises of 7 steps • Backend Application • Databases • Ontology • Dialogue Goals • Database Conversion Rules • Parsing Grammars • Generation Templates Rapid Prototyping of Dialogue Systems

11. Rapid Prototyping of Dialogue Systems Observations Effort calculated as per the knowledge source. Rapid prototyping can be considered a method in which knowledge is added to a framework to customize it. All design exhibit lack of coverage in Language resources. Direct consequence of not doing any data collection ahead of time.

12. To reduce development costs: Re-Use • Can be considered to be the driving force being the Domain Independence Hypothesis (Hanna et. al. 2005) • While Domain Independence Hypothesis is acceptable, the challenge is to develop such an domain independent system • Part of the challenge is difference in terms of complexity of applications which in turn requires different degrees and forms of domain independence • Object oriented methodology to support extensibility and re-use of (dialog) components • All discourse object contain capabilities • On request, domain spotter polls and gets scores from each discourse object indicating object’s ability for successfully handle the request • Supports maintainability and re-use • Idea of hierarchies of increasing degrees of specialism Rapid Prototyping of Dialogue Systems

13. Rapid Prototyping of Dialogue Systems • Dynamic Linking of discourse objects • Keeps frames sizes small • Facilitates delegation and delimitation of agent responsibility • Open ended structure which can grow and adapt depending on objectives set by user on run time

14. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • (DISC) Dialogue Engineering Life Cycle • Iterative Method • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Template based • Tools and Methods required • Some ideas Also included: Full list of papers surveyed

15. Dialogue Engineering Life Cycle • Based on DISC project on best practices for development and evaluation of SLDSs • Includes: Life Cycle model, Grid (set of issues and options), Set of Evaluation Criteria • SLDS Development Methodologies

16. Grid comprises of 6 aspects of SLDS • For each aspect, the grid lists various issues along with options (based on the available state of the art at that time) along with pros and cons • Choices to be made by the developers • Essentially and Iterative Life Cycle with 2 legs • Development Phases • Grid Issues are addressed in these phases • Evaluation • Evaluation criteria derived from choices made for the various Grid issues are applied here • This life cycle does not talk much about the phase after integration i.e. maintenance. • Also scope for re-use is not discussed • SLDS Development Methodologies

17. SLDS Development Methodologies • Evaluation • Analysis: Sufficiency and clarity of documents • Design: Design goals and constraints are sound, non-contradictory and feasible • Simulation: WOZ with questionnaires • Construction: Glass box and black box tests • Integration: Collected user system interaction is analysed

18. Iterative Method • Work iteratively from 2 angles • Conceptual Design • Framework customization • Solid Lines • Creative Progression • Dotted Line • How levels of the method correlate • E.g. Interpretation Module • Conceptual design starts with Parsing theory and Requirement Specifications • Framework customization starts with Parser and builds on corpus created from WOZ or other approaches • SLDS Development Methodologies

19. Iterative Method: Recommended Pre-requisites • Classification of possible dialogues and identification of main use-cases • Specification of requirements for the system behavior in each class of dialogues • Considering Dialog Management in Iterative Development • Design • Modularization • Knowledge Representation • Interfaces • Customization • Tools • Framework templates • Code Patterns • Identified Design choices • Dialogue History, User Request Handling, Sub-Dialogue Control • SLDS Development Methodologies

20. Model View Controller Approach: Very Industrial • Derived from well established GUI development practice • Possible to upload the user interface without modifying the application data or control logic • SLDS Development Methodologies

21. SLDS Development Methodologies

22. Rapid Dialogue Prototyping Methodology (RDPM) • Focuses on Dialogue components only • 5 steps • Producing the Task model • Draw relations between attributes and task to be performed • Deriving the initial dialogue model • Generic Dialogue Nodes (GDN) • Simple (Static): User inputs • List (Dynamic): Selects from menu • Internal: Start/resets, etc. • Local / Global dialogue flow management strategy • Using a WoZ experiment to instantiate the initial dialogue model • Using an Internal field test to refine the dialogue model • Using an external field test to evaluate the final dialogue model • SLDS Development Methodologies

23. (Atwell et. al., 2000) • Experience with user involvement in development • Particular case for CALL applications • Involve users at every stage • Two distinct classes of users • Prospective End-Users • Meta Level Experts • Interesting findings reported in the paper within the application domain which was possible due to user involvements at all stages • SLDS Development Methodologies

24. Methodologies I have used • Case 1: ConQuest dialogue system development • Case 2: CycleTalk dialogue system development • Sorry no time to make the slides for the above. But lets discuss • SLDS Development Methodologies

25. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • (DISC) Dialogue Engineering Life Cycle • Iterative Method • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Template based • Tools and Methods required • Some ideas Also included: Full list of papers surveyed

26. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • Iterative Method • (DISC) Dialogue Engineering Life Cycle • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Tools and Methods required • Some ideas Also included: Full list of papers surveyed

27. Dialogue system development can be seen as a process of putting knowledge into an existing framework in order to customize the framework • Tools and methods must • Provide visual languages for specification of structured dialogue data • Represent all relevant data and the dependences between them • Support formalism for defining constraints on the models and informing the user of violation of these models • Generate code that can be interpreted be generic dialogue system • Support the reuse for formerly developed domain models • Knowledge Sources View

28. Development Team & Requirements 16 15 10 11 14 User Logs 9 12 7 13 8 Dialog Task Specification Randomly created Corpus 6 5 2 3 4 Templates Grammar 1 Voice Domain Knowledge Lexicon Language Model Acoustic Model

29. Dialogue system development can be seen as a process of putting knowledge into an existing framework in order to customize the framework • Tools and methods must • Provide visual languages for specification of structured dialogue data • Represent all relevant data and the dependences between them • Support formalism for defining constraints on the models and informing the user of violation of these models • Generate code that can be interpreted be generic dialogue system • Support the reuse for formerly developed domain models • Knowledge Sources View

30. Some basic ideas / Ground rules • Practical Dialogue Hypothesis • Domain Independence Hypothesis • Rapid Prototyping of Dialogue Systems • Dialogue system frameworks • Re-use • SLDS development methodologies & Best practices • Iterative Method • (DISC) Dialogue Engineering Life Cycle • Model View Controller (MVC) approach • Rapid Dialogue Prototyping (RDP) Methodology • User involvement in development • Methodologies i have used : 2 cases • Some “potential” rules of thumb ? • Knowledge Sources view of Dialogue Systems • Overview • Tools and Methods required • Some ideas • Template based Also included: Full list of papers surveyed

31. Somewhere in between the spectrum of domain independence and domain dependence • Based on extensive in domain re-use • Changes in knowledge sources with every re-use • Issues • Incorporating knowledge changes • Propagation of knowledge changes while maintaining constraints • Hence view of the template system development would be • Which knowledge source is to be changed ? Some Ideas: Template based development

32. Readings • An architecture for Generic Dialogue Shell, J. Allen, D. Byron, M. Dzikovska, G. Ferguson, L. Galescu, A. Stent • Rapid Prototyping for Spoken Dialogue Systems, Matthias Denecke • Developing Extensible and Reusable Spoken Dialogue Components: An examination of the Queen’s Communicator, Philip Hanna, Ian O’Neil, Xingkun Liu, Michael McTear • Iterative implementation of Dialogue System Modules, Lars Degerstedt Arne Jonsson • The Dialogue Engineering Life Cycle, Laila Dybkjaer, Niels Ole Bernsen • DISC Website, www.disc2.dk • Rapid Dialogue Prototyping Methodology, Trung H. Bui, Martin Rajman, Miroslav Melichar • User Guided System Development in Interactive Spoken Language Education, E. Atwell et. al. • OpenSpeech Dialog: More powerful applications at lower cost, SpeechWorks white paper • A survey of knowledge sources in dialogue systems, Annika Flycht-Erikssn • Universal Dialogue Specification for Conversational Systems, Anke Kolzer