Ubiquitous Computing

1. UbiquitousComputing Lecture 13

2. Agenda • Model-based User Interface Design (MB-UID) • User Interface Description Languages (UIDLs) • MARIA (Modelbased lAnguage foR Interactive Applications) • Ubiquitous computing • Migratory user interfaces • Plastic user interfaces

3. Model-based user interface design MB-UID

4. MB-UID • Model-based user interface design -centered around a description of application objects and operations at a level of abstraction higher than that of code • A good model: • can be used to support multiple interfaces • help separate interface and application • describe input sequencing in a simple way, • check consistency and completeness of the interface • evaluate the interface’s speed-of-use • generate context-specific help and assist in designing the interface • designers rarely use computer-supported application modeling today and prefer less formal approaches such as story boards of user interface prototypes

5. MB-UID • the quality of traditional UIs depends in a strong factor on the experience of the designers and their skills in the platform and development tools • the market is moving to web interfaces, ubiquitous systems, and wireless. More user interfaces for every-day applications are needed in several devices (multiple user interfaces), also at the same time. • Developing UIs of this kind of systems has extra constrains and constitutes a challenge per se. • Different kinds of homogeneity among different platforms should be preserved in the User Interface. Otherwise, the user’s learning curve will force him to learn a new UI for each device he uses to access the system. • the user interface can be specified once and then refined for each target devices until reaching the implementation • Such approach can be supported by MB-UID methods and tools • using an approach capable of generating the UI code, a high percentage of time and resources can be saved.

6. Advantages of MB-UID • The abstraction level is higher that working with development environments. The specification is less dependent from the underlying technology. • Better productivity. A percent of the final UI can be directly generated and used in the final system without any further changes. • Better quality. Generated code has always the same elements in the same places. Repeating tasks exactly in the same way every time is easier for a machine than for humans, obtaining, in this way homogeneity. Conformity with standards is also easy to obtain by generating the code according to such standards. • Less errors. Generated code contains zero generation errors (if the generator is robust and it has been thoroughly tested). Semantic errors can still appear but as a consequence of misunderstanding requirements or modeling misprints.

7. Advantages of MB-UID • Provides a precise Engineering process. Development can be repeated, measured and tracked as a production line. Future projects can be estimated and scheduled with more precision based on previous data. • Multiple device support: using generators for multiples devices, platforms and architectures. • Less Time to Market. UIs can be built in less time allowing to put the product early in the market

8. Problems of M-BUID • Maintainability. Specifications and artifacts produced by the tools should have easy maintenance: allowing to keep the changes in the specification or, alternatively, allowing a sort of reverse engineering from the code to the model. • Scalability. Practitioners and designers need to apply these methods and tools to real-life problems. If the methods and tools have been poorly tested outside academic environments, is difficult to fine-tune the method and tools for scalability to guarantee the success in industrial scenarios.

9. Problems of MB-UID • Round Trip Problems. Code generated from a model has more constrains and less choices that the allowed in the final target device. Frequently, some sort of tweaking or beautification must be applied to modify the final UI to fit the usage scenario. These out-of-the-model changes should be reapplied (ideally automatically) whenever the model changes and the UI needs to be regenerated accordingly. • Integration with artists’ designs. Especially important for web sites and multimedia contents, artists have a relevant role in the developing of such systems. MB-UID artifacts must be integrated with this other source of elements to configure the final UI. To support this kind of development, special tools or conventions must be followed in both sides of the development.

10. Problems of MB-UID • Lack of standards like UML for software design or architecture. There is no standard for UI development (something similar to UML) • Lack of robust code generators. More code generators are needed to produce UIs for different kind of applications. Few tools are available on the market and are limited to specific context of usage.

11. Problems of MB-UID • Lack of integration with business logic. UIs per se are not enough to build running systems. Good integration mechanisms with application’s functionality (business logic) are needed • Lack of commercial tools supporting the methods. Few tools are available on the market. Practitioners interested in the field need to know them in first place, use and evaluate them to check if a tool fits their needs.

12. User interface description languages UIDLs

13. UIDL (User Interface Description Languages) • A User Interface Description Language (UIDL) is a formal language used in Human-Computer Interaction (HCI) in order to describe a particular user interface independently of any implementation • meta-models in order to rigorously define the semantics of such a UIDL - adhere to the principle of separation of concerns • meta-models: context of use (user, platform, environment), task, domain, abstract user interface, concrete user interface, usability (including accessibility), workflow, organization, evolution, program, transformation, and mapping • a UI may involve different interaction modalities (e.g., graphical, vocal, tactile, haptic, multimodal), interaction techniques (e.g., drag and drop) or interaction styles (e.g., direct manipulation, form filling, virtual reality). • A common fundamental assumption of most UIDLs is that UIs are modeled as algebraic or model-theoretic structures that include a collection of sets of interaction objects together with behaviors over those sets

14. UIDL • consists of a specification language that describes various aspects of a user interface under development • reduce the amount of time and effort needed to make user interfaces by providing useful abstractions and supporting automation of the design process • allow the same interface description to be rendered on multiple platforms. • beneficial for automatically generating interfaces that are personalized to individual users • involves defining a • syntax (i.e. how these characteristics can be expressed in terms of the language) • semantics (i.e., what do these characteristics mean in the real world) • It can be considered as a common way to specify a UI independently of any target language (e.g., programming or markup) that would serve to implement this UI.

15. UIDLs • Development goals support: • support for multi-platform • deviceindependence, • modality independence ( it can be used not only for communication through one channel or medium, but through several) • content delivery

16. UIDLs • Free • eXtensible Interaction Scenario Language(XISL) • Device Independent Authoring Language(DIAL) • Extensible MultiModal Annotation Markup Language(EMMA) • Simple Unified Natural Markup Language(SunML) • TeresaXML • MariaXML • Software Engineering for Embedded Systems using aComponent-Oriented Approach(SeescoaXML) • User Interface Markup Language(UIML) • USer Interface eXtensible Markup Language(UsiXML) • eXtensible Interface Markup Language(XIML) • Vendor: • MXML (Adobe) is used to describe UI layoutand behaviors, and Action Script for the FlexFramework. • Open Laszlo (Laszlo) is a XML-based languagefor rich Internet applications. • SISL(Lucent Technologies) is a XML-basedlanguage service logic that is shared across manydifferent user interfaces, including speech-basednatural language interfaces. • XAML (Microsoft) is a markup language for declarative application programming for theWindows Presentation Foundation. • XUL (Mozilla) is used to build feature-richcross platform applications that can run connected or disconnected from the Internet

17. Table 2 Properties Comparison of UIDLs UIL Models Methodology Tools Supported languages SupportedplatformsLevel Tags Concepts DISL Presentation,dialogandcontrolSpecificationof a generic, platform-independentmultimodal UIRenderingengineVoiceXML, JavaMIDP, Java Swing,Visual C++Mobile andlimiteddevicesModellevel NotspecifiedHead element,interface classes(structure, style, behavior), state,generic widgetsGIML Presentation,dialog,anddomainSpecificationof a genericinterfacedescription.GITK (GeneralizedInterfaceToolkit)C++, Java, Perl NotspecifiedMeta-model15 tags Interface, dialog,widget, objectsISML Presentation, task,dialog,domainSpecificationof a generic UIdescriptionUnder constructionJava, Microsoftfoundation class, Javaswing classesDesktopPC, 3DscreenModellevel NotspecifiedMappings andconstrains, actionevents, meta-objects,display parts,controller parts,interaction definitionRIML There isnoinformationSpecificationof a generic UIdescriptionThere is noinformationXHTML, XFORMS,XEvents, WMLSmart phone, pda,Mobile,Desktop PcModellevelThere isnoinformationDialog, Adaptation,layout, elementSeescoaXMLTask,Presentation,dialogSpecificationof a generic UIdescriptionCCOM(BetaVersion1.0 2002)PacoSuiteMSC Editor Java AWT, Swing,HTML, java.microedition,applet, VoxML, WMLJuggler Mobile,desktop PC,Palm IIIModellevel NotspecifiedComponent, port,connector, contract, participant, blueprint,instance, scenario, pltform, user, deviceSunML Presentation,dialog,domainSpecificationof a generic UIdescriptionSunMLCompiler Java Swing,voiceXML, HTML,UIML,Desktop Pc, pdaModellevel14 tags Element, list, link,dialog, interface,generic events,synchronizationTeresaXMLPresentation, task,dialogSpecificationof a generic UIdescriptionCTTE Toolfor task ModelsTeresaMarkup: Digital TV,VoiceXML,XHTML/SVG, X+VProgramming: C#DigitalTV,Mobile,DesktopPC,Modellevel19 tags Mappings, models, , platform, task, input,outputUIML Presentation,dialog,domainSpecificationof a generic UIdescriptionUIML.net,VoiceXMLrenderer,WMLrenderer,VB2UMILHTML, Java, C++,VoiceXML, QT,CORBA,and WMLdesktop PC,a handhelddevice, tv,mobileModellevel50 tags interconnection of the user interface to business logic,servicesWSXL Presentation,dialog, domainSpecificationof a generic UIdescription Not specified HTML PC, Mobile phone,Modellevel12 tags CUI=XForms,WSDL,Mapping=XLangWorkflow=WSFL,Logic=XML eventXICL Presentation,dialog,Specificationof a generic UIdescriptionXICLSTUDIOHTML, ECMAScript,CSS e DOM.desktop PC Modellevel NotspecifiedComponent,structure, script,events, properties,interfaceXIML Presentation, task,dialog,domainSpecificationof a generic UIdescriptionXIMLSchemaHTML, java swing,WLMMobile,desktop PC,PDAModellevel32 tags Mappings, models,sub models,elements, attributesand relations betweenthe elements 43 43 Properties comparison of UIDLS http://www.academia.edu/647242/A_theoretical_survey_of_user_interface_description_languages_Preliminary_results

18. UIDLs • UIML (www.uiml.org) • use­ML(http:// www.uni-kl.de/pak/useML/) • MariaXML • Usi­XML(www.usixml.org) • XIML (www.ximl.org).

19. MARIA Modelbased lAnguage foR Interactive Applications

20. MARIA • MARIA XML: • one language for the abstract description • a number of platform-dependent languages that refine the abstract one depending on the interaction resources considered: • graphical form-based • graphical mobile form-based • vocal • digital TV • graphical direct manipulation • multimodal (graphical and vocal) for desktop and mobile • advanced mobile (with support for multi-touch and accelerometers)

21. MARIA • Data Model • an abstract/concrete description of the underlying data model of the user interface, needed for representing the data (types, values, etc.) handled by the user interface • The interactorscomposing an abstract (concrete) user interface can be bound to elements of a type defined in the abstract (respectively, concrete) data model • The concrete data model is a refinement of the abstract one • The introduction of a data model also allows for more control over the admissible operations that can be performed on the various interactors. • the data model is described using the XSD type definition language. • useful for: correlation between the values of interface elements, conditional presentation connections, conditional layout of interface parts, and specifying the format of the input values. • The dependencies between the state of an interactor and the data model imply that at runtime, if a data element bound to an interactor changes its value, this change has to be notified to the interactor for updating its state and vice versa • The logic of this callback mechanism can be defined using the target technology constructs during the generation derived from the interface description

22. MARIA • Event Model. • an event model has been introduced at different abstract/concrete levels of abstractions • The introduction of an event model allows for specifying how the user interface responds to events triggered by the user • Property Change Events. These are events that change the status of some UI properties. The handlers for this type of event are only change properties, which indicate in a declarative manner how and under which conditions property values change. • Activation Events. These are events raised by activators, which are interactors with the purpose of activating some application functionality (e.g., access to a database or to a Web service). This type of event can have either change properties or script handlers (which have an associated generic script). • The abstract definition of these events contains the information for the generation of the dynamic behavior of the final UI.

23. MARIA • Supporting Ajax Scripts, which Allow the Continuous Updating of Fields. • the possibility of supporting continuous fields updating at the abstract level. We have introduced an attribute to the interactors: continuosly updated= “true”[“false”]. The concrete level provides more detail on this feature, depending on the technology used for the final UI (Ajax for Web interfaces, callback for stand-alone application, etc.). • It is worth noting that while at the abstract level a certain interactorhas to support continuous dynamic updating of its values from a certain abstract data source, at the concrete level, the specific platform-dependent technology used to support such continuous updating of the interactor must be specified.

24. MARIA • Dynamic Set of User Interface Elements • the possibility to express the need to dynamically change only a part of the UI - specified in such a way as to affect both how the UI elements are arranged in a single presentation, and how it is possible to navigate between the different presentations. • The content of a presentation can dynamically change (this is also useful for supporting Ajax techniques) • it is also possible to specify dynamic behavior that changes depending on specific conditions - use of conditional connections between presentations

25. MARIA • an interface is composed of : • one data model • one or more presentations • Each presentation is composed of: • Name • a number of possible connections • elementary interactors • interactor compositions • The presentation includes a data model and a dialog model, which contains information about the events that can be triggered by the presentation in a given time. • The presentation is also associated with a dialog model which provides information about the events that can be triggered at a given time. • The dynamic behavior of the events, and the associated handlers, is specified using the CTT temporal operators (for example, concurrency, or mutually exclusive choices, or sequentiality, etc.)

26. MARIA – Abstract Description • describes a UI only through the semantics of the interaction, without referring to a particular device capability, interaction modality or implementation technology • an AUI - composed by various Presentations that groups model elements presented to the user at once. • The model elements are of two types: • Interactor- every type of user interaction object • InteractorComposition- groups together elements that have a logical relationship • two types of interactor compositions: • grouping • relation.

27. MARIA – Abstract Description • Interactor: • an interaction object: • Selection: single choice and multiple choice • Edit: text edit, object edit, numerical edit, position edit • Control: navigator: navigate between different presentations; activator: trigger the activation of a functionality • only output object: • Object • Description • Feedback • Alarm • Text • further refinement of each of these objects can be done only by specifying some platform-dependent characteristics, therefore it is specified at the concrete level • all the interaction objects have associated events in order to manage the possibility for the user interface to model how to react after the occurrence of some events in their UI

28. MARIA – Concrete Description • provide a platform-dependent implementation language-independent description of the user interface • it assumes that there are certain available interaction resources that characterize the set of devices belonging to the considered platform • It provides an intermediate description between the abstract description and that supported by the available implementation languages for that platform

29. MARIA – composing interfaces • Web services based applications - designers and developers often have to compose existing functionalities and corresponding user interface specifications (annotations) • Four main aspects have been identified: • the abstraction level of the user interface description (abstract, concrete, implementation) • the granularity of the user interface (single objects, groups of objects may be composed, presentations may be joined to obtain the UI) • The types of aspects that are affected by the UI composition (the dynamic behavior, the perceivable UI objects or the data manipulated by the user) • The time when the composition occurs (design time/runtime).

30. Case study • Home application - users can control some interactive devices and appliances • Functionality: • suggestions for searching a device through a text editing interactor (for example, the user enters a part of the device name and some suggestions for the completion appear) • displaying information on a set of appliances in a part of the presentation while the user can dynamically add or remove elements from the appliance set

31. Case study • Specifying a web service <aui:external_functions> . . . <aui:function name="getSuggestions" type="web_service"> <aui:output type="UserSession/suggestions" /> <aui:input type="xs:string" name="inputString"/> </aui:function> . . . <aui:external_functions>

32. Case study • When the input string changes, the service must be called <aui:text_edit id="device_search"> <aui:events> <aui:value_change> <aui:handler> <aui:script> <![CDATA[ data:UserSession/suggestions =external:getSuggestions(ui:device_search.value); ]]> […] <aui:handler> <aui:change_propertyinteractor_id="device_suggestions" property_name="hidden" property_value="false" /> </aui:handler> […] </aui:text_edit> The input text The event Web service call Make results visible

33. Case study <aui:single_choice id="device_suggestions" data_reference="UserSession/suggestions"> <aui:events> <aui:selection_change> <aui:handler> <aui:change_property interactor_id="device_select_activator" property_name="enabled" data_value="true" /> <aui:change_property interactor_id="device_monitor_activator“property_name="enabled" data_value="true" /> <aui:change_property interactor_id="device_search" property_name="value" data_value="ui:device_suggestions.selected" /> <aui:change_property interactor_id="device_suggestions" property_name="hidden" property_value="true" /> […] </aui:single_choice> A single choice interactor is populated with the suggested devices

34. Case study • The concrete interfaces will be adapted to platform capabilities (screen size, processor speed) and interaction technique (vocal, multi-touch, mouse) • Differences: • Number of presentations • Different locations of the groups on the screen

35. Case study Concrete interfaces for desktop and iPhone

36. MARIAE – MARIA Environment • provides a novel solution able to exploit task models (represented in the ConcurTaskTrees notation) and user interface models (in the MARIA language) for the design and development of interactive applications based on Web services for various types of platforms (desktop, smartphones, vocal, multimodal, ...)

37. Gummy • a multiplatform graphical user interface builder that can generate an initial design for a new platform by adapting and combining features of existing user interfaces created for the same application • keeps all user interfaces consistent without requiring designers to considerably change their work practice • can generate an initial design for a new platform by adapting and combining features of existing user interfaces created for the same application • a computing platform -the combination of a hardware device, an operating system and user interface toolkit

38. Problems in multi-platform UI design approaches • high cost incurred in adding a new target device and keeping all user interfaces consistent using manual approaches • there is no clear separation between the user interface and the underlying application logic • A common solution to these issues is to specify the user interface in an abstract way by means of high-level models such as task models and dialogue models • The platform-specific user interfaces are then generated automatically from this abstract description • The user interface has to be specified only once, which makes it easier to make changes or add a new target platform

39. Problems in multi-platform UI design approaches • Issues: • the resulting user interfaces usually still lack the aesthetic quality of a manually designed interface. • The design process is not intuitive since designers have to master a new language to specify the high-level models and cannot accurately predict what the resulting user interface will look like

40. GUMMY • Gummy can generate an initial design for a new platform from existing user interfaces created for the same application but for other platforms • makes it it easy to target new platforms and keep all user interfaces consistent without requiring designers to considerably change their work practice • Gummy allows for a true WYSIWYG4 multi-platform user interface design process.

42. GUMMY approach • Gummy combines the benefits of both the manual approach and model-based techniques • Designers create and perform prototyping of a multi-platform graphical user interface (GUI) in the same way as they do when dealing with traditional GUI builders such as Microsoft Visual Studio2 and Netbeans3. • Gummy builds a platform-independent representation of the user interface and updates it as the designer makes changes • This allows for an abstract specification of the user interface while keeping the design process intuitive and familiar • An abstract user interface specification avoids a tight interweaving of application and presentation logic

43. GUMMY • The underlying language – UIML • UIML contains: • a platform-independent user interface description - used to describe the structure,style, content and behaviour of a user interface using platform-independent terms • a mapping vocabulary - contains mappings of the terms onto concrete widgets • The traditional rendering step - translate the platform-independent user interface • description into a concrete user interface using these mappings. • Gummy - the opposite approach: the concrete representations were used in the tool and were internally mapped onto the associated abstractions. • The designer works with the concrete graphical representations, avoiding the XML language, while the tool maintains a synchronised platform-independent UIML document forthe concrete design.

44. GUMMY – mapping vocabulary

45. GUMMY – the process of creating UIs

46. GUMMY – the process of creating UIs • the user interface designer specifies the target platform for which they want to design a user interface (possible platforms are a mobile phone with the Compact .NET framework, a digital TV with DVB-MHP, etc.) • According to the specified platform, Gummy automatically loads a GUI builder workspace that is fully equipped for designing user interfaces for this platform. • From a set of existing user interfaces that are all created for the same application, Gummy automatically generates an initial design for the selected target platform – using a transformation engine component. When there are no existing user interfaces available, this step generates an empty user interface for the selected platform.

47. GUMMY – the process of creating UIs • The designer can refine the initial user interface until it fits their vision. The resulting design is then added to the set of existing user interface designs where it may serve as an input for the transformation engine. After this step, a new iteration is started. • When the designer is finished, Gummy exports all designs as one or more UIML user interface descriptions– can be rendered on the target platforms.

48. Damask • prototyping tool which targets web UIs that run on PCs and mobile phones, and prompt-and-response style voice Uis • designers sketch out their design for one device while using design patterns to specify higher-level concepts within their design • Damask’s patterns include pre-built UI fragments that are already optimized for each device • designers also use layers to specify which UI parts are common across devices and which are specific to one device • Damask uses the sketches and patterns to generate designs for the other devices, which the designers can refine

49. UBIQUITOUS cOMPuting UBICOMP

50. What is • "Ubiquitous computing names the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives." --Mark Weiser