270 likes | 430 Views
Generating Remote Control Interfaces for Complex Appliances. Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October 30, 2002. The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France. The Problem. Appliances are too complex.
E N D
Generating Remote Control Interfaces for Complex Appliances Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October 30, 2002 The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France
The Problem Appliances are too complex
The Problem, cont. Each complex appliances has its own idiosyncratic interface! • Stereo systems • Telephones • VCRs • Alarm Clocks • … Increasingly Computerized Low Usability
Feedback Specifications Control Our Solution Separate the interface from the appliance! Handheld becomes personal universal controller (PUC) Key Features • Interface-independent appliance specification • Automatic generation of GUI and speech interfaces
Automatic Generation of UIs Benefits • All interfaces consistent for the user • With conventions of handheld • Even from multiple manufacturers Addresses hotel alarm clock problem! • Multiple modalities (GUI + Speech UI) • Can take into account user preferences • Will work on special purpose devices (for disabled)
Important Work By Others • INCITS V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] • User Interface Modeling Language (UIML) http:/www.uiml.org/ • Xweb (now ICE) [Olsen Jr., UIST 2000] • Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] • Speakeasy [Newman, UIST 2002]
Architecture - Comm. Protocol - Interface Generators - Specification Lang. - Appliance Adaptors XML-based
Language Design Approach • Create reference interfaces • AIWA Shelf Stereo • AT&T Telephone/Answering Machine • Test interfaces with subjects • Users twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfaces • Analyze interfaces for functional information
Language Elements State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • “seek” button on a Radio Label Information • One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information
Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves
Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables • Equals, Greater Than, Less Than • Linked with logical operators (AND, OR) • For example, <and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>
Interface Generators Generators for Two Modalities • Graphical • Implemented for PocketPC in Java 1.1 • Uses dependency information to generate panel structure of interface • Speech • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001] • Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)
Graphical Interface Generator Focuses on panel structure of user interface • Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info!
Inferring Structure Find sets of variables that are “mutually exclusive” • Every variable in a set will never be active at the same time as a variable in another set Create structure with sets, using overlapping panels
Choosing Panel Types a) b) c) partial screen full screen tabbed
Making the Interface Concrete Finish conceptual layout • Choose controls (decision tree) • Choose row layouts (one column, two column, etc.) Allocate space • Examine panel contents and choose sizes Instantiate and place controls
Generating Speech Interfaces Automatically build USI tree from dependencies • Allows verbal navigation of functional groups Automatically generate grammar for parser • Phrases for query and control “What is playmode?” “Set playmode to play” “play” Automatically generate language model and pronunciation for recognizer
Controlling Appliances We have built adaptors for many actual appliances • Sony Digital Camcorder • X10 Lighting • Audiophase Shelf Stereo • AudioReQuest MP3 player • Software Media Players (WinAmp, WMP) Written specifications for others • Elevator • Telephone/Answering Machine
Future Work • Improve quality of generated user interfaces • Back-tracking, etc. • Specification and generation of list interfaces • Necessary for many appliances • Adaptive interface generation • New interfaces modeled on past interfaces • General purpose appliance adaptor for HAVi and UPnP protocols
PUC Project Members Brad A. Myers Michael Higgins Joseph Hughes Thomas K. Harris Roni Rosenfeld Mathilde Pignol Stefanie Shriver Peter Lucas Kevin Litwack Funding National Science Foundation Pittsburgh Digital Greenhouse Microsoft Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent Acknowledgements
Thanks! User Interface Software & Technology (UIST) October 30, 2002 http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/
Determining Panel Structure • Find states that others depend upon • Search for the group that contains this state • Determine mutually exclusive sets within group children • Assign sets to panels • Based upon types, tree properties FMStation Mode == Any FMPresets Power AMStation Mode AMPresets Band Radio Mode == Radio PlayState CD CDTrack Mode == CD
Choosing Panel Types Rules for Choosing Panels • If… • state has boolean type • all controls are one set • Create… • Two full-screen overlapping panels.
Choosing Panel Types, cont. Rules for Choosing Panels • If… • state has enumerated type • there is a set for every value of the state • Create… • a tabbed panel
Choosing Panel Types, cont. Rules for Choosing Panels • If… • there are mutually exclusive sets of controls • Create… • overlapping panels • a control for the state not on the overlapping panel