Past, Present and Future of User Interface Software Tools

1. Past, Present and Future ofUser Interface Software Tools Brad A. Myers, Scott E. Hudson, and Randy Pausch Developed for HCIC’99 and TOCHI Updated 2009

2. Introduction • User Interface Software Tools • Help developers design and implement user interfaces • Focus on Tools, but influenced by future UIs • Today’s tools are highly successful • Window Managers, Toolkits, Interface Builders ubiquitous • Most software built using them • Arebased on HCI researchBrad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

3. Talk Outline • Historical Perspective • What worked • What didn’t catch on • Why • Lessons Learned • Future Prospects and Visions • UI Trends that will require new tools • Important issues

4. Historical Perspective • Themes • Address the useful & important aspects of UIs • Tools that succeeded helped (just) where needed • Threshold / Ceiling • Threshold = How hard to get started • Ceiling = how much can be achieved • Path of Least Resistance • Tools influence user interfaces created • Predictability • If not predictable, then not accepted by programmers • Moving Targets • Changing user interface styles makes tools obsolete

5. What Worked • Window Managers and Toolkits • Event Languages • Graphical, Interactive Tools • Component Architectures • Scripting Languages • Hypertext • Object Oriented Programming • Constraints

6. Window Managers • Multiple (tiled) windows in research systems of 1960’s: NLS, etc. • Overlapping introduced in Alan Kay’s thesis (1969) • Smalltalk, 1974 at Xerox PARC • Successful because multiple windows help users manage scarce resources: • Screen space and input devices • Attention of users • Affordances for reminding and finding other work

7. Toolkits • A collection of widgets • Menus, scroll bars, text entry fields, buttons, etc. • Toolkits help with programming • Help maintain consistency among UIs • Key insight of Macintosh toolbox • Path of least resistance translates into getting programmers to do the right thing • Successful partially because address common, low-level features for all UIs • Address the useful & important aspects of UIs

8. Event Languages • Create programs by writing event handlers • Many UIMSs used this style • Univ. of Alberta (1985), Sassafras (1986), etc. • Now used by HyperCard, Visual Basic, Lingo, etc. • Toolkits with call-backs or action methods are related • Advantages: • Natural for GUIs since generate discrete events • Flow of control in user’s hands rather than programmer’s • Discourages moded UIs • May not work well in future

9. Graphical Interactive Tools • Create parts of user interface by laying out widgets with a mouse • Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT • Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors” • Advantages: • Graphical parts done in an appropriate, graphical way • Address the useful & important aspects of UIs • Accessible to non-programmers • Low threshold

10. Component Architectures • Create applications out of components which are separately developed and compiled • In UI software, each component controls an area of the screen • Example: drawing component handles picture inside a document • Invented by Andrew research project at CMU (1988) • 1999: OLE, OpenDoc, ActiveX, Java Beans • Now: SOA • Address the useful & important aspects of UIs • Just the “glue” to hold together components

11. Scripting Languages • First GUIs used interpreted languages • Smalltalk, InterLisp • Rapid development, supports prototyping • Low threshold • Then C and C++ became popular • Now, bringing back advantages in scripting languages • tcl/tk, Python, perl • Visual Basic, Javascript, Ruby, … • But language must contain general-purpose control structures

12. Hypertext • Ted Nelson named it in 1965 and developed Hypertext system at Brown University • Important systems: NLS (1967), Hyperties (1986) • World-Wide Web • Phenomenal success due to: • Ease of use of Mosaic browser • Support for embedded graphics • Support for easy authoring • Low threshold both for authoring and viewing

13. Object Oriented Programming • Success of OO owes much to UI software field • Popularized by Smalltalk • GUI elements (widgets) seem like objects • Have state, accept events (messages) • Rise parallels GUIs • C++ with Windows 3.1 • Java for behaviors in WWW • 2009: Flash, etc.

14. Constraints • Declare a relationship and system maintains it • Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996) • 1999: hadn’t caught on • We thought would be mostly used for graphics • Now: Flash data bindings • Connect data to graphics • Address the useful & important aspects of UIs • Predictability • Constraint networks can be hard to debug • Especially in multi-way constraints • High threshold • Programmer must specify (or deduce) solving order • Constraints require thinking differently

15. What Hasn’t Caught On • User Interface Management Systems • Formal Language-Based Tools • Model-Based and Automatic Techniques

16. User Interface Management Systems • Original goal: like databases, provide high-level language that abstracts details of input and output devices • This separation has not worked in practice • Good user interfaces must take into account the pragmatics and detailed behavior of all objects • Standardization of GUI input and output devices has made goal somewhat moot • Doesn’t address the useful & important aspects of UIs

17. Formal Language Based Tools • Early UIMSs used grammars and state-transition diagrams • Focus on dialog management • Moving Targets • Direct manipulation made dialog management less important • Path of Least Resistance • State diagrams afford worse user interfaces • High threshold • Formal languages are often hard to learn

18. Model-Based and Automatic Techniques • Automatic techniques for generating UIs from a model or declarative specification of contents • Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993) • Try to separate specification of UI from content • May provide automatic reformating, retargeting, customization to users, etc. • Result is often unpredictable • Often can be worse UI than hand-drawn • Sometimes model is larger than the code it would replace

19. Discussion of Themes • Address the useful & important aspects of UIs • Narrower tools have been more successful than ones that try to do “everything” • Do one thing well • Threshold / Ceiling • Research systems often aim for high ceiling • Successful systems seem to instead aim for a low threshold • Impossible to have both?

20. Discussion of Themes, cont. • Path of Least Resistance • Tools should guide implementers into better user interfaces • Goal for the future: do this more? • Predictability • Programmers do not seem willing to release control • Especially when system may do sub-optimal things • Moving Targets • Long stability of Macintosh Desktop paradigm has enabled maturing of tools • 1999: We predicted a change soon • 2009?

21. Future Prospects and Visions • Important Trends • Ubiquitous Computing • Move to recognition-based interfaces • 3-D interfaces • End-user customization and scripting • Violate assumptions of today’s tools • Assumptions limit what designers can do • Often unrecognized • Implications for future tools

22. Ubiquitous Computing • Computation embedded in many kinds of devices • Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms • Next wave: easy communication with radio • E.g., BlueTooth: www.bluetooth.com • Significant Implications for tools • Tools for coordinating multiple, distributed,communicating devices • “Multi-computer” user interfaces • Moving target problem

23. Varying Input and Output • Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels • Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels • Cell phone to Stanford’s wall (3796 x 1436 pixels)

24. Need New Interaction Techniques • Interaction techniques for desktop will not work • No room on small devices • Can’t reach menubar on wall-size devices • 2009: iPhone doesn’t use desktop metaphors • Want to run same application on different devices

25. Need for Prototyping Devices • User interface will be in hardware • Rapid design and prototyping needed for hardware • Pragmatics and usability cannot be evaluated from a simulation on a screen

26. Multiple, Distributed, Communicating • Computers more for communication, not for computation • Already true for WWW, email, digital pagers, cell-phones • Computers as intermediaries between people • CSCW • But can’t assume have similar systems • Single person with multiple devices • Room-area networks like BlueTooth or HomeRF • People communicating with themselves • Tools will need to help with data sharing and synchronization

27. Limitations of Today’s Tools for UbiComp • Tools assume a Pointing Device • Hidden reliance on specific characteristics of common devices • Size of display • Many tools cannot handle a different number of mouse buttons • Change to a stylus on a touchpad requires different techniques • Assumptions about the setting • Assume user is sitting and looking at UI • Assume has user’s full attention

28. Move to Recognition-Based Interfaces • Speech, gestures, camera-based vision • Multimodal interaction • User will pick which modality to use • Use multiple modalities at same time • Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. • Need tools to hide these complexities

29. Fundamental Differences ofRecognition-based UIs • Input is uncertain • Recognition can make errors • Requires monitoring, feedback, correction • Interpreting input requires deep knowledge of data • Context of the application • “Move the red truck to here”

30. Implications of Recognition-based UIs • GUI event model no longer works • Do not produce discrete events • Separation of UI from application no longer works • Need a architecture based on accessible application data structures • “Reflection”, “Open Data Model”

31. 3-D Interfaces • Difficult to design the right abstractions for tools • Demise of VRML for Web • Need to settle on the 3-D widgets and interaction techniques that will be standard • Requirement for near-real-time interactivity • Need to hide the mathematics • 2009: but what useful for?

32. End-user Customization and Scripting • Spreadsheet enables end users to specify their own computation • Visual Basic, other “scripting” languages • Needed in all applications • Threshold for programming is too high • Need “gentle slope systems”

33. Gentle Slope Systems Programming in C Visual Basic Flash HyperCard C Programming MFC C Programming DifficultyofUse xCmds Click andCreate ActionScript HyperTalk Goal Basic Sophistication of what can be created

34. More Assumptions of Today’s Tools • Skill and Dexterity of users • Older users • Makes single, fixed library of widgets untenable • Non-overlapping and opaque components • Preclude translucency, magic lens interactions • Fixed libraries of components (widgets) • Creating new widgets is very difficult • New devices will require new interaction techniques • Interactive tools provide freedom of design • Aim for “Mechanism not Policy”

35. Operating Systems Considerations • What is in the OS? • Window Manager? Toolkit? Communication? Scripting facilities? • Need ever increasing services for applications • Need more access to low-level information • E.g., hardware buttons, whether on network • Ideally, API to support competition and research into these components

36. Some Design Guidelines for Future Tools • Many things require further research • Organize around providing rich context • Of application and device state • To inquire about data & methods; “reflection” • Enables EUP, Recognition-Based UIs, data sharing for UbiComp • Rather than event-based

37. More Design Guidelines • Replaceable User Interfaces • Ability to have multiple UIs • Enabled by procedural interface to everything in UI • Enables UbiComp devices, EUP • Aim for low threshold, rather than high ceiling • But cover the right parts of the interface • Predictable for programmers rather than “smart” or automatic • Need for support for evaluation

38. Conclusions • Research in tools necessarily trails innovation in UI design • Due to consolidation on desktop metaphor, significant progress in tools • UI design poised for radical changes • New opportunities and challenges for tools