1 / 14

Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany

Multimodal Dialog Description for Mobile Devices. Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany. Motivating Example. Control and monitoring of electronic devices. Winamp Mediaplayer PC User Interface. Motivation MIRS DISL Implemen- tation

baba
Download Presentation

Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Dialog Description for Mobile Devices Steffen Bleul, Wolfgang Mueller, Robbie SchäferPaderborn University/C-LABPaderborn, Germany

  2. Motivating Example Control and monitoring of electronic devices Winamp Mediaplayer PC User Interface Motivation MIRS DISL Implemen- tation Summary &Conclusion Problem: UI does not scale for other platforms / devices / modalities

  3. Renderer Renderer Renderer Renderer Renderer MIRS - Multimodal Interaction and Rendering System • Dialog and Interface Specification Language - DISL • Structural modelDefinition of generic widgets (no specific modality)(inherited from UIML with ideas from Paterno, Plomp...) • Control/Dialog modelVariables, Rules, Actions, Events(inherited from Dialog Specification Notation – DSN) S-DISL Interpreter Motivation MIRS DISL Implemen- tation Summary &Conclusion S-DISL UI Spec (DISL) HW &UserProfile abstract to concrete mapping Apps XSLT http tcp Server (Mobile) Client

  4. Dialog and Interaction Specification Language - DISL • Modification of UIML Interface Section • structure - widget organization • Generic widget (modality independent widget) • New generic widget elements • style - properties • defines properties of generic widgets • behavior - user interaction • defines behavior for generic widgets • extension for advanced state-oriented dialog specification • states • rules • events • communication Motivation MIRS DISL Implemen- tation Summary &Conclusion

  5. DISL – Structure Elements Motivation MIRS DISL Implemen- tation Summary &Conclusion 1. Container Elements - variablefield - textfield - genericfield - widgetlist 2. Interaction Elements a. Input b. Misc - variablebox - textbox - genericbox - choice - command - confirmation - genericcommand

  6. DISL – Structure/Style/Behavior <interface ...> <structure> <widget id="Play" generic-widget="command"/> ... </structure> <style> <part generic-widget id="Play" > <property id="title">Title</property> <property id="description"> This is a Remote Control for Audio Playing Devices </property> <property id=“help“>Press Button to Start Playing</property> <property id=“visible“ >true</property> <property id=“selected“ >false</property> <property id=“activated“>false</property> </part> </style> <behavior> ... </behavior> Motivation MIRS DISL Implemen- tation Summary &Conclusion </interface>

  7. DISL - Behavior • <interface id=“MediaPlayer" state=“init"> • … • <behavior> • <variable ...> ... </variable> • ... • <rule ...> • <condition>...</condition> • <event...> ...</event> • </rule> • ... • <transition ...> • <if-true .../> • <action> • <statement ...> </statement> • ... • </action> • </transition> • ... • </behavior> • </interface> Motivation MIRS DISL Implemen- tation Summary &Conclusion

  8. Dialog Specification Notation (DSN)Basic Concepts • Events: • TogglePower • SetVolume • ToggleMute • PlayTitle • StopTitle • TogglePause • NextTitle • ... • States: • Power {#isOn,#isOff} • Volume {#0,#1,…,#100} • Mute {#MuteOn,#MuteOff} • Playing {#PlayingOn,#PlayingOff} • Paused {#PauseOn,#PauseOff} • ... Motivation MIRS DISL Implemen- tation Summary &Conclusion

  9. Dialog Specification Notation (DSN)Basic Concepts (cont‘d) Motivation MIRS DISL Implemen- tation Summary &Conclusion Rules: Power 1. #isOn TogglePower  #isOff Power 2. #isOff TogglePower  #isOn Mute 1. #isOn #MuteOn ToggleMute  #MuteOff Mute 2. #isOn #MuteOff ToggleMute  #MuteOn Playing 1. #isOn PlayTitle  #PlayingOn #PauseOff Playing 2. #isOn NextTitle  #PlayingOn #PauseOff Playing 3. #isOn PreviousTitle  #PlayingOn #PauseOff Playing 4. #isOn #PlayingOn StopTitle  #PlayingOff #PauseOff Paused 1. #isOn #PausedOn  #PausedOff #PlayingOn Paused 2. #isOn #PausedOff #PlayingOn  #PausedOn #PlayingOff

  10. DSN to DISL States: Power {#isOn,#isOff} Events: TogglePower Motivation MIRS DISL Implemen- tation Summary &Conclusion <variable id="Power" type="boolean">false</variable> <rule id="isOn"> <condition> <equal> <variable id=“Power">true</variable> </equal> </condition> </rule> <rule id="isOff"> <condition> <equal> <variable id=“Power">false</variable> </equal> </condition> </rule> <rule id="TogglePower"> <condition> <equal> <property generic-widget="TogglePower" id="selected">true</property> </equal> </condition> </rule>

  11. DSN to DISL Rules: Power 1. #isOn TogglePower  #isOff Motivation MIRS DISL Implemen- tation Summary &Conclusion <transition> <if-true rule-id="TogglePower" expression="and"> <if-true rule-id="isOn"/> </if-true> <action> <call source="http://localhost:8080/mediaplayer/servlet" id="tryoff" synchronized="yes" timeout="5s"> <parameter id="request"> <value-of>turnoff</value-of> </parameter> </call> <statement> <variable-content id=“isOn“><const>false</const></variable> </statement> </action> </transition> 2 conditions to be true • 2 actions • to execute • http request with 1 parameter • 1 variable assignment

  12. DISL – Additional Features Periodic Events <transition> <eventid=“checkstatus“ repeat=“yes“ time=“5s“> <action> <call source=„...“ synchronized="yes" timeout="5s"> </action> </event> </transition> Motivation MIRS DISL Implemen- tation Summary &Conclusion More Complex Expressions <condition> <op operator=“and“> <op-expression expression=“greaterthan“> ... </op-expression> <op-expression expression=“lessthan“> ... </op-expression> </op> </condition> Change of Interface at Runtime <transition> <if-true ...> ... </if-true> <action> <interface id="VolumeControl"/> </action> </transition> Content of virtually any element can be modified

  13. AWT-Renderer (J2ME/ MIDP1.0) Current Implementation kXML Parser Motivation MIRS DISL Implemen- tation Summary &Conclusion S-DISL UI Spec (DISL) http S-DISL InterpreterJ2ME/MIDP1.0 XSLT/Saxon Processor Now with Bluetooth support!! PC Tomcat http-server Siemens M55

  14. Summary & Future Work • Summary • XML-based specification language for general UIs • advanced state-oriented dialog specificationby variables, rules, actions, events(basic concepts inherited from Dialog Specification Notation – DSN) • definition of generic widgetsNOT for a specific modality, e.g., text, graphics, speech(basic concepts inherited from UIML, Paterno, Plomp...) • Implementation for Java enabled phones • Future Work • use shorthand vocabulary • Better structuring through object orientation (step from DSN to ODSN) • Cooperation with high level modelling techniques (bridging the gap between e.g. task models and rendered UI) • Proof of concept with more modalities (Speech and Gesture) Motivation MIRS DISL Implemen- tation Summary &Conclusion

More Related