1 / 43

Towards Virtualization of User Interfaces based on UsiXML

Web3D Symposium 2005 March 29 - April 1 2005, Bangor, UK. Towards Virtualization of User Interfaces based on UsiXML. J. P. Molina 1,2 , J. Vanderdonckt 1 , F. Montero 1,2 , P. González 1 1 Université catholique de Louvain 2 University of Castilla-La Mancha. www.usixml.org. Contents.

elvis-yates
Download Presentation

Towards Virtualization of User Interfaces based on UsiXML

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web3D Symposium 2005 March 29 - April 1 2005, Bangor, UK Towards Virtualization of User Interfaces based on UsiXML J. P. Molina1,2, J. Vanderdonckt1, F. Montero1,2, P. González1 1 Université catholique de Louvain 2 University of Castilla-La Mancha www.usixml.org

  2. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  3. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  4. Overview • A model-based approach is presented for structuring a development process of UIs based on UsiXML, a XML-compliant language. • UsiXML provides a Concrete UI description that remains independent from any platform (HTML, Java, VRML). • To support the rendering of this description in a virtual world, two toolkits have been developed: for VRML97 and X3D worlds. Web3D Symposium 2005, Bangor, UK

  5. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  6. Motivation • It is difficult to build a UI that relies solely on 3D direct manipulation. • Some tasks require 1D/2D interaction. • Support for legacy applications is one requirement for new 3D desktops. • 3D environments must handle 2D very well to break into desktop market. • 3D F/X of UI elements should be rendered in 3D, matching much better the mental model of the user. • There is a need to transform 2D GUIs into 3D. Web3D Symposium 2005, Bangor, UK

  7. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  8. Related work • Automated generation of a 3D UI from its specifications demonstrated in several domains. • Maquil’s tool [Maquil, 2004]. • Produces 3D dialog boxes describing the UI in terms of the StudierStube environment. • The tool is equipped with an automatic layout facility of the widgets. Web3D Symposium 2005, Bangor, UK

  9. Related work • VRML Widgets Working Group. • Aimed to define a standard set of widgets. • Two different architectures: • Layered approach [Isaacs & Becker]. • Time control and pointer-surface interaction. • Interpretation and control of interaction (“feel” of a widget). • Geometry (“look” of a widget). • Component-based approach [Seidman]. • Five components: sensors, internal glue, geometry, external glue and Virtual Reality User Interfaces (VRUIs). Web3D Symposium 2005, Bangor, UK

  10. Related work • VRML authoring tools. • ParallelGraphics Internet Scene Assembler. • Library of prototypes: button, toggle button, slider, checkbox, text label, etc. • Basic set for building HUD (Head-Up Display) interfaces for PC desktop virtual worlds. Web3D Symposium 2005, Bangor, UK

  11. Related work • X3D-related work. • Project CONTIGRA. • Defines a XML-based language for the specification of widgets, metaphors and interaction techniques. • X3D language is used to describe the final appearance of geometry of each 3D widget. • The set of components has been designed taking geometry apart from behaviour. • Many components have not been developed or are not publicly available yet. Web3D Symposium 2005, Bangor, UK

  12. Related work • Virtual Reality toolkits. • The INQUISITIVE toolkit [Boyd, 1999]. • Implemented using the MAVERIK toolkit, includes window, dial, slider, etc. • VIEWL [Larimer, 2003]. • Based on Qt and DIVERSE, windows are shown as polygons that are tangent to a sphere that surrounds the user. • VRIXML [Cuppens, 2004]. • A XML-based languge for the creation of 2D/3D hybrid interfaces. Web3D Symposium 2005, Bangor, UK

  13. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  14. Cameleon Reference Framework • Defines UI development steps for multi-context interactive applications [Calvary et al., 2003]. • Final UI (FUI). • UI running on a particular computing platform (HTML, Java, VRML). • Concrete UI (CUI). • A mock-up that is independent of any computer platform. • Abstract UI (AUI). • Defines abstract containers and individual components, independent of any modality of interaction. • Task & Concepts (T&C). • Describe tasks and domain-oriented concepts. Web3D Symposium 2005, Bangor, UK

  15. Cameleon reference framework • Multi-path UI development: Abstraction, Reification and Translation. Web3D Symposium 2005, Bangor, UK

  16. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  17. The CUI level and UsiXML • From a same specification of CUI, a GUI can be obtained in parallel to a virtualization of this GUI. • A description of a CUI can be obtained by successive forward engineering from the T&C level, the AUI level or directly. • UsiXML, a XML-compliant User Interface Description Language [url: http://www.usixml.org], provides a CUI description that remains independent from any toolkit, whether graphical or virtual. • This quality is often referred as platform independence. Web3D Symposium 2005, Bangor, UK

  18. The CUI level and UsiXML • A CUI model consists of a hierarchical decomposition of Concrete Interaction Objects (CIOs). • Each CIO can be sub-typed into sub-CIOs depending on the interaction modality chosen: graphicalCIO, auditoryCIO, etc. • Every final element of the CUI inherits properties depending on the category they belong to. • The properties chosen for UsiXML belong to the intersection of property sets of major toolkits and window managers: Windows GDI, Java AWT and Swing, HTML, etc. Web3D Symposium 2005, Bangor, UK

  19. Each graphicalCIO has specific properties such as isVisible, isEnabled, fgColor, and bgColor. Each graphicalCIO is sub-typed into one of the two possible categories: graphicalContainer, and graphicalIndividualComponent. The CUI level and UsiXML Web3D Symposium 2005, Bangor, UK

  20. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  21. Mapping from CUI to VRML/X3D • When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: • Direct mapping. • It is not possible to set a one-to-one mapping, as there is no VRML/X3D primitives that can be matched with a CIO. These Web3D languages specify basic elements that must be used together to create interactive elements. Web3D Symposium 2005, Bangor, UK

  22. Mapping from CUI to VRML/X3D • When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: • Direct mapping. • New mapping. • There is a need to fill a gap by introducing a new widget by appropriate implementation. • Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be added in order to describe the interface elements in the 3D world, such as position and size. Web3D Symposium 2005, Bangor, UK

  23. Mapping from CUI to VRML/X3D • When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: • Direct mapping. • New mapping. • No possible mapping. • There are CIOs whose VRML/X3D counterpart have not been done yet or are impossible or difficult to implement. An example of this case is the CIO box. Web3D Symposium 2005, Bangor, UK

  24. Mapping from CUI to VRML/X3D • Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world. • Display = “Screen” PROTO. • Rectangular area where screen units (pixels) are converted into world units (meters). • Serves as a container of UI elements (children field). • Mouse = User’s pointing device. • VRML/X3D sensors. • Keyboard? • VRML: requires non-standard extensions (PG’s KbdSensor). • X3D: StringSensor. • Virtual keyboard. Web3D Symposium 2005, Bangor, UK

  25. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  26. VRML97 & X3D toolkits • The architecture for both toolkits is based on: • The prototyping mechanism available in both languages. • Some ideas borrowed from the VRML Widget WG. • For each CIO, there is a final widget implemented as a PROTO that consists of: • An interface based on a set of fields. • Shapes for the geometry and appearance of the widget. • Sensors that allow user-widget interaction. • Internal logic that manages the state of the widget. • Routes that links shapes, sensors and logic. • Differences between both toolkits are meant to take advantage of new features of X3D specification. Web3D Symposium 2005, Bangor, UK

  27. VRML97 & X3D toolkits • GUIToggleButton: VRML97 PROTO interface. PROTO GUIToggleButton [ # USIXML: cio # ... # USIXML: graphicalCio # ... # USIXML: graphicalIndividualComponent # ... # USIXML: toggleButton field SFBool defaultState FALSE # VRML97 GUI Toolkit fields field SFInt32 top 0 field SFInt32 left 0 field SFInt32 width 75 field SFInt32 height 25 exposedField MFNode label [ ] eventOut SFTime touchTime eventIn SFBool set_state eventOut SFBool state_changed eventIn SFInt32 set_top eventIn SFInt32 set_left eventIn SFInt32 set_width eventIn SFInt32 set_height eventOut SFInt32 top_changed eventOut SFInt32 left_changed eventOut SFInt32 width_changed eventOut SFInt32 height_changed • From attributes specified for toggleButton in the CUI model. • Some UsiXML attributes have not a meaning in the toolkit. • Other attributes are added, plus set_ and _changed fields. • Type of attributes must be adapted, e.g.: • boolean = SFBool • string? If represents a colour, SFColor seems more appropriate. Web3D Symposium 2005, Bangor, UK

  28. VRML97 & X3D toolkits • GUIToggleButton: X3D PROTO interface. • inputOutput fields were not possible with VRML97 as this language does not allow us to match them with Script fields. <ProtoDeclare name="GUIToggleButton"> <ProtoInterface> <!-- USIXML: cio --> <!-- . . . --> <!-- USIXML: graphicalCio --> <!-- . . . --> <!-- USIXML: graphicalIndividualComponent --> <!-- . . . --> <!-- USIXML: toggleButton --> <field accessType="initializeOnly" name="defaultState" type="SFBool" value="false"/> <!-- X3D GUI Toolkit fields --> <field accessType="inputOutput" name="top" type="SFInt32" value="0"/> <field accessType="inputOutput" name="left" type="SFInt32" value="0"/> <field accessType="inputOutput" name="width" type="SFInt32" value="75"/> <field accessType="inputOutput" name="height" type="SFInt32" value="25"/> <field accessType="inputOutput" name="state" type="SFBool"/> <field accessType="inputOutput" name="label" type="MFNode"/> <field accessType="outputOnly" name="touchTime" type="SFTime"/> Web3D Symposium 2005, Bangor, UK

  29. VRML97 & X3D toolkits • GUIToggleButton: VRML97 vs. X3D PROTO. PROTO GUIToggleButton [ # USIXML: cio # ... # USIXML: graphicalCio # ... # USIXML: graphicalIndividualComponent # ... # USIXML: toggleButton field SFBool defaultState FALSE # VRML97 GUI Toolkit fields field SFInt32 top 0 field SFInt32 left 0 field SFInt32 width 75 field SFInt32 height 25 exposedField MFNode label [ ] eventOut SFTime touchTime eventIn SFBool set_state eventOut SFBool state_changed eventIn SFInt32 set_top eventIn SFInt32 set_left eventIn SFInt32 set_width eventIn SFInt32 set_height eventOut SFInt32 top_changed eventOut SFInt32 left_changed eventOut SFInt32 width_changed eventOut SFInt32 height_changed <ProtoDeclare name="GUIToggleButton"> <ProtoInterface> <!-- USIXML: cio --> <!-- . . . --> <!-- USIXML: graphicalCio --> <!-- . . . --> <!-- USIXML: graphicalIndividualComponent --> <!-- . . . --> <!-- USIXML: toggleButton --> <field accessType="initializeOnly" name="defaultState" type="SFBool" value="false"/> <!-- X3D GUI Toolkit fields --> <field accessType="inputOutput" name="top" type="SFInt32" value="0"/> <field accessType="inputOutput" name="left" type="SFInt32" value="0"/> <field accessType="inputOutput" name="width" type="SFInt32" value="75"/> <field accessType="inputOutput" name="height" type="SFInt32" value="25"/> <field accessType="inputOutput" name="state" type="SFBool"/> <field accessType="inputOutput" name="label" type="MFNode"/> <field accessType="outputOnly" name="touchTime" type="SFTime"/> Web3D Symposium 2005, Bangor, UK

  30. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  31. A case study • Internet Radio Player. • This case study is used as an example running from the topmost level of the reference framework until the bottom level, here in VRML97 and X3D. • The following tools are used at different levels, all of them based on UsiXML language: • T&C and AUI level: IdealXML, by F. Montero • CUI level: GrafiXML, by B. Michotte • FUI level: VRML97 & X3D toolkits www.usixml.org Web3D Symposium 2005, Bangor, UK

  32. A case study • Task & Concept Model. • User’s actions, independently of any implementation. Web3D Symposium 2005, Bangor, UK

  33. A case study • Abstract User Interface Model (AUI). • From the task model, a set of transformations can be applied to generate an AUI, later manually refined. Web3D Symposium 2005, Bangor, UK

  34. A case study • Concrete Model (CUI). • New transformations produce a CUI that can be refined with GrafiXML. Again, everything is saved in UsiXML. Web3D Symposium 2005, Bangor, UK

  35. A case study • Final User Interface (FUI). • According to given mappings, VRML97 and X3D versions of the FUI can be generated from UsiXML specifications. The rendered FUIs are genuine 3D objects, with real 3D effects. Web3D Symposium 2005, Bangor, UK

  36. A case study • Augmenting the GUI with virtual platform. • A virtual version of the old environment could be used to run the old applications [Roberts, 2000]. Web3D Symposium 2005, Bangor, UK

  37. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  38. Continuum of User Interfaces • Milgram and Kishino [1994] continuum distributes user interfaces on one dimension ranging from the real environment to the virtual environment by going through two forms of Mixed Reality (MR): Augmented Reality (AR) and Augmented Virtuality (AV). • We have extended this continuum by adding a more continuous range of UIs in the virtual part, and a second dimension to represent the degree of immersion. Web3D Symposium 2005, Bangor, UK

  39. Continuum of User Interfaces Web3D Symposium 2005, Bangor, UK

  40. Contents • Overview • Motivation • Related work • Cameleon reference framework • The Concrete UI level and UsiXML • Mapping from CUI to VRML/X3D • VRML97 & X3D toolkits • A case study • Continuum of User Interfaces • Conclusion Web3D Symposium 2005, Bangor, UK

  41. Conclusion and further work • UsiXML is a XML-compliant language that allows the designer to describe a UI, whether 2D or 3D, graphical or virtual. • It supports a multi-path framework that can be followed to produce, from a same specification, a GUI and a virtualization of such UI. • The implemented VRML97 & X3D toolkits make it possible to render the FUI in a 3D world. • The “look” is similar to 2D widgets, but the “feel” is a genuine 3D behaviour. • The continuum of user interfaces has been extended to accommodate Virtualized User Interfaces. Web3D Symposium 2005, Bangor, UK

  42. Conclusion and further work • With these toolkits, it also possible to simulate in the virtual world the look and feel of the same UI in virtual versions of different platforms, such as PDAs, PCs or even Interactive Worlds. • Current work deals with supporting the migration of a UI from one platform to another, by visually representing this transition. Web3D Symposium 2005, Bangor, UK

  43. Thank youMercíGracias More info: www.usixml.org Contact: jpmolina@info-ab.uclm.es

More Related