Recent Work in Model-Based User Interfaces

1. Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software

2. Last time… • Model-based User Interfaces • Automatic generation of the user interface so the programmer won’t do a bad job. • Dialog boxes are relatively easy to generate • The full application interface is hard to generate • Abstract descriptions of the interface can be longer and harder to generate than implementing the interface itself. • Interface builders turned out to be easier…

3. But work continued… • Focus Changed • Task models were leveraged more • Design assistant aspect emphasized • A Couple Projects of Interest: • Trident • Mecano & Mobi-D • FUSE • AIDE

4. TRIDENT • Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995. • An interface design assistant • Interesting features: • Knowledge-based approach (i.e. expert system) • Choosing Widgets • Doing Layout • Use of Task Models • Decides where separate windows are needed

5. Used a decision tree Chose abstract interaction objects (AIO) Similar to Brad’s Interactor Model Lots of parameters Continuous? Capacity Etc. Choosing Widgets

6. Choosing Layout Uses Right/Bottom Strategy • Next component is placed to the right or below the current component • Decision made by heuristics or designer

7. Right/Bottom Strategy

8. Basically used for constructing wizard-like interfaces What information should be on the first screen, etc. Windows from Task Models

9. What are task models, anyway? • Description of the process a user takes to reach a goal in a specific domain • Typically have hierarchical structure • Introduced by GOMS • Number of different task modeling languages • GOMS • UAN • ConcurTaskTrees

10. Developed by Fabio Paterno et al. for the design of user interfaces Goals Graphical for easy interpretation Concurrent model for representing UI tasks Different task types Represent all tasks, including those performed by the system ConcurTaskTrees

11. Three phases Hierarchically decompose the tasks Identify the temporal relationships among tasks at same level Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate. Temporal Relationships T1 [] T2 - Choice T1 ||| T2 - Interleaving T1 |[]| T2 - Synchronization T1 >> T2 - Enabling T1 []>> T2 - Enabling with Information Passing T1 [> T2 - Deactivation T1* - Iteration T1(n) - Finite Iteration [T1] - Optional T – Recursion Task Building Process

12. Note: First example is ambiguous Example

13. Another Example

14. Building/Editing Task Models • Tools are available • ConcurTaskTrees Environment http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”

15. Recent Systems • XIML – eXtensible Interface Markup Language • Developed by the makers of Mecano/Mobi-D and Trident • Kitchen-sink language for modeling any part of the interface design process • XWeb • Now known as ICE – Interactive Computing Everywhere • ICrafter • A system for integrating user interfaces from multiple devices • Personal Universal Controller • My research…

16. XIML eXtensible Interface Markup Language • Designed by RedWhale Software • Intended to support the full lifecycle of interfacebuilding

17. XIML Requirements • Central Repository of Data • For one user interface or many • Comprehensive Lifecycle Support • Abstract and Concrete Elements • Relational Support • Underlying Technology • XIML must be independent of particular tools

18. Models in XIML • An XIML document can contain any type of model • Task • Domain • User • Presentation • Dialog

19. Example Use for XIML Multi-platform interface development

20. Status of XIML • Used by RedWhale Software to drive their interface consultant business • They have developed many tools • move interaction data to/from XIML • Leverage data in XIML to better understand various interfaces • Automate parts of the interface design process

21. Model-Based Interfaces for Control • XWeb • ICrafter • PUC

22. XWeb • Work by Dan Olsen and group at BYU • Premise: • “Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.” • The web comes close to solving this problem, but is interactively insufficient.

23. XWeb Protocols • Based upon the architecture of the web • XTP Interaction Protocol • Server-side data has a tree structure • Structured Data in XML • URLs for location of objects • xweb://my.site/games/chess/3/@winner • xweb://automate.home/lights/livingroom/ • xweb://automate.home/lights/familyroom/-1

24. XWeb & XTP • CHANGE message (similar to GET in HTTP) • Sequence of editing operations to apply to a sub-tree • Set an attribute’s value • Delete an attribute • Change some child object to a new value • Insert a new child object • Move a subtree to a new location • Copy a subtree to a new location

25. Platform Independent Interfaces • Two models are specified • DataView – The attributes of the service • XView – A mapping of the attributes into high-level “interactors” • Interactors are somewhat like abstract interaction objects • Atomic • Numeric • Time • Date • Enumeration • Text • Links • Aggregation • Group • List

26. XWeb Example DataView

27. XWeb Example XView

28. XWeb Example Interface

29. Other XWeb Details • Has simple approach for adjusting to different screen sizes • Shrink portions of the interface • Add additional columns of widgets • Also capable of generating speech interfaces • Based on a tree traversal approach like Universal Speech Interfaces

30. ICrafter • Part of the Interactive Workspaces research project at Stanford • Main objective: • “to allow users of interactive workspaces to flexibly interact with services” • Contribution • An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service. • In practice, much of the interface generation is done by hand though automatic generation is supported.

31. ICrafter Architecture

32. How is aggregation accomplished? • High-level service interfaces (programmatic) • Data Producer • Data Consumer • The Interface Manager has pattern generators • Recognize patterns in the services used • Generate interfaces for these patterns • This means that unique functionality will not be available in the aggregate interface

33. Automatic Generation in ICrafter

34. Manual Generation in ICrafter

35. Personal Universal Controller • My work with Brad • Problem: • Appliance interfaces are too complex and too idiosyncratic. • Solution: • Separate the interface from the appliance and use a device with a richer interface to control the appliance: • PDA, mobile phone, etc.

36. Feedback Specifications Control Idea • Control existing appliances • Generate multi-modal interfaces

37. - Comm. Protocol - Interface Generators - Specification Lang. - Appliance Adaptors Architecture XML-based

38. Language Design Approach • Create reference interfaces • AIWA Shelf Stereo • AT&T Telephone/Answering Machine • Test interfaces with subjects • Users twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfaces • Analyze interfaces for functional information

39. Language Elements State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • “seek” button on a Radio Label Information • One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information

40. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves

41. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables • Equals, Greater Than, Less Than • Linked with logical operators (AND, OR) • For example, <and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

42. Interface Generators Generators for Two Modalities • Graphical • Implemented for PocketPC in Java 1.1 • Uses dependency information to generate panel structure of interface • Speech • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001] • Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

43. Graphical Interface Generator Focuses on panel structure of user interface • Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info!

44. Inferring Structure Find sets of variables that are “mutually exclusive” • Every variable in a set will never be active at the same time as a variable in another set Create structure with sets, using overlapping panels

45. Choosing Panel Types a) b) c) partial screen full screen tabbed

46. Making the Interface Concrete Finish conceptual layout • Choose controls (decision tree) • Choose row layouts (one column, two column, etc.) Allocate space • Examine panel contents and choose sizes Instantiate and place controls

47. Generating Speech Interfaces Automatically build USI tree from dependencies • Allows verbal navigation of functional groups Automatically generate grammar for parser • Phrases for query and control “What is playmode?” “Set playmode to play” “play” Automatically generate language model and pronunciation for recognizer