Describing Appliance User Interfaces Abstractly with XML

1. Describing Appliance User Interfaces Abstractly with XML Jeffrey Nichols Workshop on Developing User Interfaces with XML Advanced Visual Interfaces May 25, 2004Gallipoli, Italy

2. Appliances are complex and their user interfaces are often hard to use! Problem

3. Solution Specifications Control Feedback Separate the user interface from the appliance Interface Devices Appliances We call this approach the “Personal Universal Controller” (PUC)

4. Language Goals Inform with hand-designed interfaces • What functional information is needed to create interfaces? Additional Requirements • Support complete functionality of appliance • No specific presentation information • Short and concise • Easy to author “only one way to specify anything” • Sufficient detail for generating high quality interfaces without assistance Full documentation and XML Schema available at: http://www.cs.cmu.edu/~pebbles/puc/

5. Language Elements Elements • State variables & commands • Labels • Group tree • Dependency information • Smart Templates Example media player specification • Play, stop, pause, next track, previous track • Play list

6. Smart Templates A means to specify high-level information Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect • Parameterize the tags so one Smart Template can be used for multiple appliances and situations

7. Smart Templates A means to specify high-level information Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect • Parameterize the tags so one Smart Template can be used for multiple appliances and situations

8. Actual Appliances We have written specifications for many actual appliances • Sony Digital Camcorder • Windows Media Player • Axis UPnP Pan & Tilt Camera • Lutron Lighting • X10 Lighting • Audiophase Shelf Stereo • AudioReQuest MP3 player • GM Vehicle Information System • GM Vehicle Climate Control • Elevator • Telephone/Answering Machine • Several Alarm Clocks • Task Manager (To Do List) • GM Navigation System (in progress) • Mitsubishi DVCR (in progress) Used to build apps for Intel’s “Personal Server” Used by group at TU Vienna for controlling AR apps

9. Interface Generators Four Interface Generators for Two Modalities Graphical • Desktop, PocketPC, and Microsoft Smartphone Speech • Ph.D. work of Thomas Harris • Built on top of the PUC framework • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

10. Language Analysis Pros • Useful for generating both graphical and speech interfaces • Concise • Easily learned by multiple undergraduates • Capable of describing many existing appliances • Has enough detail to generate usable interfaces without designer intervention Cons • No task information Though some “flow” can be specified using dependencies • Describes appliance functions, not appliance user interface • Designed for appliance descriptions, not desktop applications

11. Funding National Science Foundation Microsoft General Motors Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent Lutron Lantronix PUC Project Members Brad A. Myers Kevin Litwack Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Pegeen Shen Jeffrey Stylos Suporn Pongnumkul Peter Lucas Acknowledgements

12. Thanks for listening! For more information… http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/

13. Describing Appliance User Interfaces Abstractly with XML Jeffrey Nichols Workshop on Developing User Interfaces with XML Advanced Visual Interfaces May 25, 2004Gallipoli, Italy

14. More Screenshots

15. Future Work Further develop Smart Templates idea Add task information • Focus on “distributed task information” among multiple appliances • Generate one combined user interface for multiple connected appliances (e.g. home theaters) Ensure interface consistency for similar appliances Evaluate generated interfaces and specification language • User studies, etc.

16. Research Approach • Hand-design remote control interfaces • Determine functional information needed from appliances to design user interfaces • Design language for describing appliance functions • Build interface generators for multiple platforms • Perform user studies to evaluate the interface generators

17. Benefits of Automatic Generation • Devices not pre-programmed with appliance details • Multiple modalities (GUI + Speech UI) • All interfaces consistent for a user • With conventions of the handheld • Even from multiple manufacturers • Create user interfaces that control multiple connected appliances e.g. a home theater

18. Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand • AIWA Shelf Stereo • AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

19. Palm Interfaces Initially designed paper-prototype interfaces for Palm telephone stereo

20. Hand-Designed Interfaces Interfaces for Microsoft’s PocketPC (simulated remote control) telephone stereo

21. Comparison Study Compared performance of first-time users (not experts) Procedure • Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics • Time to complete all tasks • Number of times a user manual was needed • Number of missteps

22. Comparison Study Results Using our interfaces, users were twice as fastand made half as many errors All differences are significant (p < 0.05)

23. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

24. Language Elements, cont. State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • e.g. “seek” button on a Radio

25. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

26. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

27. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

28. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

29. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) • Allows feedback to user when a function is not available

30. Smart Templates Example

31. Graphical Interface Generator Rule-based approach • Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface • Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info!

32. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

33. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

34. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

35. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

36. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

37. Generation Process With layout fixing rules Without layout fixing rules • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

38. High-Level Conventions Problem • Human designers rely partly on conventions when making an interface • Users expect their appliances to use conventions they know about

39. Smart Templates Need a way to specify high-level information to interface generators Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect

40. Smart Templates, cont. Features • Parameterized • Specified using primitive elements of specification language

41. Continuing Work Define and implement Smart Templates • date, mute, power, time-absolute, volume, etc. • Develop more as more appliances are specified Combinations of templates • Less implementation cost than a new template • e.g. date and time-absolute Use of templates with data already on controller device • e.g. calendar and address information • Might allow user to enter address from contact list into navigation system

42. Smart Templates, cont. One template can be used across multiple appliances and can be generated on multiple platforms