UNIFORM: Automatically Generating Consistent Remote Control User Interfaces

UNIFORM: Automatically Generating Consistent Remote Control User Interfaces Jeffrey Nichols, Brad A. Myers, Brandon Rothrock Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Conference on Human Factors in Computing Systems (CHI 2006) April 25, 2006

Motivation • The number of computerized devices is increasing • In the kitchen • Ovens, blenders, microwaves, etc. • In the living room • Home entertainment systems: VCRs, televisions, stereos, etc. • Around the house • Thermostats, lighting, alarm clocks, outdoor watering systems, etc. • In the car • Stereos, climate controls, navigation systems, integrated cellular phones, etc. • At the office • Copiers, fax machines, printers, etc.

Motivation, cont. Appliance interfaces could be more usable if their interfaces were consistent Almost all appliances have some unique features Question: How might additional functionality be presented to the user while still providing a consistent interface?

Generates personally consistent user interfaces UNIFORM Specifications Control State Feedback Appliances Personal Mobile Devices Personal Universal Controller (PUC) system

UNIFORM Copier A Copier B Consistency disabled – Copier A used first

UNIFORM Copier A Consistent Copier B Copier B Consistency disabled – Copier A used first Consistency enabled – Copier A used first

Overview • Motivation • Previous Work on Consistency • Specification Authoring Studies • Requirements for Consistency • Architecture • Knowledge Base and Finding Similarity • Generating Consistent Interfaces • Discussion and Future Work

Related Work Theory of Consistency • 1988 workshop at CHI was unable to define consistency [Nielsen 88, Grudin 89] • Loosely defined as “doing similar things in similar ways” [Reisner 90] • Different dimensions of consistency [Kellogg 87] • Work on consistency as knowledge transfer (e.g. [Keiras & Polson 85, Polson 88])

Related Work, cont. Systems addressing Consistency • Some tools can automatically test for consistency GLEAN uses a GOMS-based approach [Kieras 95] Sherlock generates metrics for interpretation by designer [Mahajan 97] • Previous automatic generation systems have ensured consistency within an application or family of applications ITS [Wiecha 90] PUC [Nichols 02] SUPPLE[Gajos 05]

Specification Authoring Study High-End: Mitsubishi DVCR Mid-Range: Samsung DVD-VCR Low End: Panasonic VCR Study of Authoring Appliance Specifications • How can specifications vary for different appliances with similar functions? • How can specifications vary for different authors? Two User Studies • Study #1: An expert user (me!) wrote specs for three different VCRs • Study #2: Three subjects wrote specifications for the same VCR All subjects instructed to: • Include all functions • Be faithful to the existing appliance design

Results of Both Studies Functions • Some were identical • Others were similar but had different labels • Others had very different specifications Structure • Structure for the first study was very similar across appliances • Top-level structure was the same for the second study, but many differences were found at lower levels Results shows that specifications can differ substantially Analysis of results showed important distinction between functional and structural consistency

Consistency in UNIFORM We define a consistent interface to be one that incorporates elements and organization that are familiar to the user from previous interfaces We have refined this definition into six requirements, based on previous research and our authoring study

Requirements for Consistency • Interfaces should manipulate functions in the same way • Interfaces should locate similar functions in the same place • Interfaces should use similar labels for similar functions • Interfaces should maintain a similar visual appearance

Requirements for Consistency • Interfaces should manipulate functions in the same way • Interfaces should locate similar functions in the same place • Interfaces should use similar labels for similar functions • Interfaces should maintain a similar visual appearance • Usability of unique functions is more important than consistency

Requirements for Consistency • Interfaces should manipulate functions in the same way • Interfaces should locate similar functions in the same place • Interfaces should use similar labels for similar functions • Interfaces should maintain a similar visual appearance • Usability of unique functions is more important than consistency • Users must be able to choose to which appliance consistency is ensured

UNIFORM Architecture Knowledge Base Personally Consistent User Interface UNIFORM Interface Generation PUC ApplianceSpecification

UNIFORM Architecture <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase UNIFORM Interface Generation Input:Appliance Specification • Previous Specifications • Functional Mappings Between Specifications • Previously Generated UIs Knowledge Base Output:Consistent User Interface

UNIFORM Architecture <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase 3 Abstract Consistency Phase 5 Concrete Consistency Phase UNIFORM Interface Generation Input:Appliance Specification Abstract UI Building Phase 2 • Previous Specifications • Functional Mappings Between Specifications • Previously Generated UIs Concrete UIBuilding Phase 4 Knowledge Base Output:Consistent User Interface

UNIFORM Architecture <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase UNIFORM Interface Generation Input:Appliance Specification • Previous Specifications • Functional Mappings Between Specifications • Previously Generated UIs Knowledge Base Output:Consistent User Interface

Knowledge Base • Previous Specifications • Functional Mappings Between Specifications • Previously Generated UIs Knowledge Base Functional Mappings • We support six types of mappings, based on the results of the specification authoring studies general, state, node, group, list, and template • Specifies that a function in one specification is similar to a function in another Mappings are grouped within the knowledge base in mapping graphs

Mapping Graphs Answering Machine State: Mode { Play, Stop, Pause } Command: Play New Messages Mitsubishi VCR State: Mode { Play, Stop, Pause, Rewind, F-Fwd, Record } 0 3 • Previous Specifications • Functional Mappings Between Specifications • Previously Generated UIs Cheap VCR Command: Play Command: Stop Command: Pause Command: Rewind Command: F-Fwd Command: Record 1 0 Knowledge Base DVD Player State: Mode { Play, Stop, Pause } Command: Next Track Command: Previous Track Group semantically similar functions across appliances One graph for each function • Power • Volume • Media Controls • Etc. Node counts • Store generation history • Used to find basis for consistency Edges • Track whether consistency can be ensured Choice in consistency supported by manipulating the node counts

Where do mappings come from? <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase Input:Appliance Specification Three ways mappings may be found • Automatically extracted during mapping phase • Challenging because of limited semantic information in specifications • We have implemented two mapping algorithms, one based on work in automatic schema mapping from the database community • Current best algorithm finds about 60% of mappings • We are also exploring other techniques • Specified by the user • Manually entered as text • Interaction techniques? • Downloaded from the Internet • Manufacturers? • Internet communities? (e.g. www.remotecentral.com) Output:Consistent User Interface

Abstract UI Building Phase <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase Input:Appliance Specification Rules from previous PUC system Transforms specification into abstract user interface Abstract user interface includes: • Tree organization • Control choice Output:Consistent User Interface

Abstract Consistency Phase <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase Input:Appliance Specification Most of the work to ensure consistency happens in this phase • Operates on the abstract user interface • We have found that most consistency can be ensured at the abstract level • Also enables many of our rules to cross platforms with little or no additional work This phase is made up of two parts: • Functional sub-phase Similar functions are assigned similar labels Similar functions are converted, if necessary, to a consistent representation • Abstract Structural sub-phase Similar functions are placed in similar groups Output:Consistent User Interface

Abstract Structural Sub-Phase This sub-phase has two parts: • Moving Similar functions are moved to previous groups • Re-ordering Within each group, functions are re-ordered to match previous order

Moving Channel Status Setup Clock Base Base Mitsubishi DVCR Base Samsung DVD-VCR

Moving Setup Channel Status Setup Clock Base Base Mitsubishi DVCR Setup Base Samsung DVD-VCR

Moving Clock Status Channel Status Setup Clock Base Base Mitsubishi DVCR Clock Setup Base Samsung DVD-VCR

Moving Channel Status Setup Clock Base Clock Clock Status Setup Status Base Base Samsung DVD-VCR Mitsubishi DVCR

Moving Channel Setup Channel Status Setup Clock Base Base Mitsubishi DVCR Channel Channel Clock Setup Status Base Samsung DVD-VCR Unique functions are moved with their group but never by themselves

Re-Ordering

Re-Ordering When Timed Recordings for the Samsung DVD-VCR Length Type Source Channel Speed Conflict Date Time Timed Recordings for the Mitsubishi DVCR Conflict VCR+ Channel Day Start Stop Speed

Re-Ordering When Type Timed Recordings for the Samsung DVD-VCR When Type Channel Channel Speed Speed Conflict Conflict Timed Recordings for the Mitsubishi DVCR Conflict VCR+ Channel When Speed

Re-Ordering When Length Type Source Channel Conflict Timed Recordings for the Samsung DVD-VCR Speed Date Time Timed Recordings for the Mitsubishi DVCR Conflict VCR+ Channel Day Start Stop Speed

Concrete UI Building Phase <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase Input:Appliance Specification Rules from previous PUC system Transforms abstract user interface into concrete user interface Concrete user interface includes: • Organization necessary for screen size • Panel Structure • Instantiations of UI objects • Positions and sizes of controls Output:Consistent User Interface

Concrete Consistency Phase <?xml?> <spec> <stuff/> </spec> 1 Mapping Phase Abstract UI Building Phase 2 3 Abstract Consistency Phase Concrete UIBuilding Phase 4 5 Concrete Consistency Phase Input:Appliance Specification Differences in the available labels or number of controls may have created inconsistencies • For example, a rule may have not added organization in this interface that was in the previous interface • The side-panel orientation in a previous interface might have been chosen to allow a long label to fit Rules in this phase address these problems provided that the usability of unique functions is not affected • For example, side-panel orientation might not change if needed for a large label or control Output:Consistent User Interface

Discussion and Future Work Evaluation of Generated Interfaces (in progress) • Users will perform ~10 tasks with two all-in-one copier interfaces (HP or Canon) • Three conditions Actual Appliance vs. PUC vs. Uniform

Discussion and Future Work How else might we deal with unique functions? • Better semantic information • More detailed modeling? Better knowledge of consistency • Elaborate and improve on requirements for consistency • Dimensions of consistency and relative importance • More operational knowledge More consistency rules • Adding new organization • Support for “consistency ghosts” and other forms of “transformational consistency” [Richter 2006]

Co-Authors Brad A. Myers Brandon Rothrock Thesis Committee Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) PUC Project Members Duen Horng Chau Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Thomas Psik Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Jeffrey Stylos Stefanie Tomko Peter Lucas Acknowledgements

Thanks for listening! For more information… http://www.cs.cmu.edu/~jeffreyn/uniform/ http://www.pebbles.hcii.cmu.edu/puc/

UNIFORM: Automatically Generating Consistent Remote Control User Interfaces Jeffrey Nichols, Brad A. Myers, Brandon Rothrock Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Conference on Human Factors in Computing Systems (CHI 2006) April 25, 2006

Questions • Have you thought about using an approach where, given that you have 4-5 appliances, you generate all of the possible consistent interfaces and then find the right one (either by the user or through some kind of optimization or metric) – Allen Cypher • What happens when I start with a simple interface and then get more complex over time? Does the consistent structure cause everything to become hidden? – Margaret Burnett • Can you handle the case where on one appliance the function is a button and on another the function is multiple steps? – Henry Lieberman

Generated Phone Examples Copier A Copier B Consistency disabled

Generated Phone Examples Consistency enabled - Copier A used first Copier A Copier B

Generated Phone Examples Copier B Copier A Consistency enabled - Copier B used first

Functional Sub-Phase Inspects each function in the new specification • Determines whether there is a previous function with which to ensure consistency (using mapping graph) • Rules make changes to the abstract interface to ensure consistency • Eight rules covering a number of potential functional differences • Always ensure that previous labels are used • May change the controls Two sorting functions from different copier interfaces

UNIFORM: Automatically Generating Consistent Remote Control User Interfaces