Automatically Generating High-Quality User Interfaces for Appliances

2. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Thesis Proposal Presentation Thesis Committee: Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.

3. Appliances are complex and their user interfaces are often hard to use! Problem, cont.

4. Problem, cont.

5. Problem, cont. May 2003 CNN – Jan 2004 April 29, 1991

6. Thesis Statement “ A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the appliances.”

7. Approach: Use Mobile Devices Common, cheap, and often have ability to communicate 10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers

8. Approach, cont. Specifications Control Feedback Use mobile devices to control all appliances in the environment Mobile Devices Appliances Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

9. Automatic Generation of UIs Benefits • Multiple modalities (GUI + Speech UI) • All interfaces consistent for a user • With conventions of the handheld • Even from multiple manufacturers Addresses hotel alarm clock problem • Create user interfaces that control multiple connected appliances e.g. a home theater

10. Important Work By Others INCITS/V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] Controlling Appliances • Xweb [Olsen Jr., UIST 2000] • Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] • Speakeasy [Newman, UIST 2002] Model-based User Interface Work • UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993] • MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995] • eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002] • User Interface Modeling Language (UIML) [Abrams, W3C 1999] None of these systems have been validated to generate high-quality interfaces without designer intervention.

11. Research Approach • Hand-design remote control interfaces • Determine functional information needed from appliances to design user interfaces • Design language for describing appliance functions • Build interface generators for multiple platforms • Perform user studies to evaluate the interface generators

12. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

13. Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand • AIWA Shelf Stereo • AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

14. Hand-Designed Interfaces Interfaces for Microsoft’s PocketPC (simulated remote control) telephone stereo

15. Comparison Study Compared performance of first-time users (not experts) Procedure • Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics • Time to complete all tasks • Number of times a user manual was needed • Number of missteps

16. Comparison Study Results Using our interfaces, users were twice as fastand made half as many errors All differences are significant (p < 0.05)

17. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

18. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

19. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

20. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

21. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

22. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

23. Language Design Informed by hand-designed interfaces • What functional information was needed to create interfaces? Additional Requirements • Support complete functionality of appliance • No specific layout information • Only one way to specify anything Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

24. Language Elements Elements • State variables & commands • Labels • Group tree • Dependency information Example media player specification • Play, stop, pause, next track, previous track • Play list

25. Language Elements, cont. State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • e.g. “seek” button on a Radio

26. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

27. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

28. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

29. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

30. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) • Allows feedback to user when a function is not available

31. Interface Generators Generators for Two Modalities Graphical • Desktop, PocketPC, and Microsoft Smartphone Speech • Ph.D. work of Thomas Harris • Built on top of the PUC framework • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

32. Graphical Interface Generator Rule-based approach • Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface • Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info!

33. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

34. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

35. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

36. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

37. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

38. Generation Process With layout fixing rules Without layout fixing rules • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

39. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

40. High-Level Conventions Problem • Human designers rely partly on conventions when making an interface • Users expect their appliances to use conventions they know about

41. Smart Templates Need a way to specify high-level information to interface generators Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect

42. Smart Templates, cont. Features • Parameterized • Specified using primitive elements of specification language

43. Preliminary Implementation A few templates: image, image-list, media-controls, time-duration Interfaces with Smart Templates

44. Continuing Work Define and implement Smart Templates • date, mute, power, time-absolute, volume, etc. • Develop more as more appliances are specified Combinations of templates • Less implementation cost than a new template • e.g. date and time-absolute Use of templates with data already on controller device • e.g. calendar and address information • Might allow user to enter address from contact list into navigation system

45. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

46. Interface Consistency PUC devices have a unique opportunity to provide consistency for the user • Personal device • Used for interacting with most appliances Two ways that PUC UIs can be made consistent • With other applications on the same device • With past interfaces for similar appliances

47. Consistency with Past Interfaces Two sub-problems to address: • Similarity Which functions of a new appliance are similar to the functions of interfaces generated in the past? • Consistency For similar functions, what rules from previous interfaces can be applied to ensure consistency?

48. Similarity Problem Difficult to conclusively know whether two functions from different appliances are the same • Very little semantic information in the specification language Can estimate similarity based on properties of state variables • Smart Template • Name • Group Name • Labels • Type Improve estimate by looking at relationships between multiple similar variables

49. Similarity & Consistency Problems new previous sparse similarity new previous new previous branch similarity significant similarity

50. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule