Standards-Based Programming Model

1. Rafah Hosn, Team Lead Interaction Middleware and Standards for Portal Server IBM T.J. Watson Research Center Yorktown Heights, NY Standards-Based Programming Model For Next Generation Multimodal Applications Title slide

2. Programming Model What is a programming model? Why do we need to standardize it? Next Generation Multimodal Application Features & Requirements Standard Based Multimodal Programming Model Remaining Challenges Conclusion OUTLINE

3. What Is A Programming Model?

4. Programming Model • A programming model defines: • How applications are authored. • Processing model of components on which applications are deployed. • Examples: • Web programming model. • Java programming model.

5. Why Do We Need A Standard?

6. Open Standards • Transparency • Open architecture. • Well defined control flow. • Governance • Evolution of technology governed by group of vendors. • Eliminates dominance of any one vendor. • Consistency • Components can be assembled from multiple providers. • Important for rapid development.

7. Features Of Next Generation Multimodal Applications

8. Features Of Next Generation Multimodal Applications • Support diversity of clients and platforms • Interoperability. • Distributed processing. • Support rich conversational user interfaces • Pluggable dialog management strategies. • Appropriate interfaces for natural language processing, statistical language models, etc.

9. Features Of Next Generation Multimodal Applications (cont’d) • Reduce latency time • Partial user interface updates. • Efficient distribution. • Rapid Experimentation • Interoperability. • Higher level abstractions in authoring.

10. Requirements For The Multimodal Programming Model • Extensible. • Interoperable. • Distributable. • Easy to use.

11. Standards-Based Multimodal Programming Model

12. Elements Of A Standards-Based Multimodal Programming Model • Foundation Blocks • Event driven processing • MVC Architecture • Declarative Authoring

13. Elements Of A Standards-Based Multimodal Programming Model • Event driven processing • Glues components of a multimodal framework together. • Behaviour of components can be affected by events raised externally, this implies: • Same eventing interfaces. • Same eventing model. • No dependency on underlying implementation. • Enables interoperability • Enables efficient distribution

14. Elements Of A Standards-Based Multimodal Programming Model • Foundation Blocks • Event driven processing • MVC Architecture • Declarative Authoring

15. Elements Of A Standards-Based Multimodal Programming Model (cont’d) • Model-View-Controller (MVC) architecture • Centralized application state held in model: • Efficient for passing between components. • Used for partial update of client view. • Separate controller • Pluggable customized dialog strategies. • Decreases latency time • Increases extensibility

16. Elements Of A Standards-Based Multimodal Programming Model • Foundation Blocks • Event driven processing • MVC Architecture • Declarative Authoring

17. Elements Of A Standards-Based Multimodal Programming Model (cont’d) • Declarative authoring • Provides a higher level of abstraction • Easier to learn. • Easier to maintain. • Reduces development time. • Enables rapid experimentation

18. Web Programming Model • Why? • Proliferation of Web access. • Maturity of Web standards. • Satisfy multimodal requirements. • Standards For Web programming model • W3C XML data model (XForms), XML Schema. • W3C DOM 2 Event propagation algorithm, XML Events. • W3C standards for multiple modalities • Keyboard, mouse access (XHTML), Speech (VoiceXML) • W3C Multimodal Interaction (MMI) working group.

19. Multimodal Mobile Real Estate Agent Configuration HP iPaq WiFi/2.5G XHTML+Voice application XHTML browser on device. VoiceXML platform on server. Includes Natural Language (NL) processor Interaction Manager on server. Sample Interaction “Show me one or two bedroom condos in Manhattan, upper west side“ XHTML list of matching results returned to client Top 3 matches are played to user Find An Apartment Zip Code or City Type of Apt -- Bedrooms Standards-Based Multimodal Web Programming Model

20. Model Model DOM Tree DOM Tree Standards-Based Multimodal Web Programming Model Voice Modality GUI Modality View View (XHTML) Dialog Zip Code Or City Type Of Apt Bedrooms Dialog DOM Event DOM Event Controller Controller Presentation Model XML Application Application

21. Distributed Programming Model Device has GPRS, can automatically fill Zip Code Handler to fill Zip Code runs on server User clicks on Zip Code field “Click” event is raised on XHTML browser XML Event is dispatched on server GPRS location & XML Event sent to server Script to resolve GPRS invoked server side Results returned as XML instance Partial update to client view using XML instance User sees Zip Code result Distributed Standards-Based Multimodal Web Programming Model Find An Apartment ZipCode Click Here Type of Apt -- Bedrooms

22. XML Event “Click” Model DOM Tree Distributed Standards-Based Multimodal Web Programming Model Presentation Presentation View CIick Ev View (XHTML) Model DOM Tree CIick Ev Zip Code <script id=“resolveZip”> Controller XML Model Tier 1 Tier 2

23. Standard-Based Multimodal Web Programming Model • Author XML • Declarative authoring (XHTML, VoiceXML) • XML data model, XML Schema for application state • DOM events to wire interaction • Consume XML • Web based clients that consume and render XML • Support DOM 2 event propagation • Distribute Using XML • XML Events • XML data model

24. Remaining Challenges

25. Remaining Challenges • Interaction management • Introduce declarative hooks for dialog management. • Adapt synchronization in a distributed environment. • Context management • Dynamic context in mobile space • W3C MMI Systems & Environment group • Incorporate context • Application data and history for dialog management • User preferences & static device profile for output generation

26. Remaining Challenges (cont’d) • Distributed programming model • Hard to automate distribution. • Affects usability. • Additional XML standards for • Gaze • Gesture • Pen input • Touch point • 3D Gestures

27. Conclusion

28. Conclusion • Multimodal interaction is a necessity for next generation applications. • Multimodal is an emerging field with remaining technology challenges. • Standards-based multimodal programming model provides an open, extensible platform for testing and deploying innovating ideas in this area.

29. END !!