Past, Present and Future of User Interface Software Tools

1. Past, Present and Future ofUser Interface Software Tools Brad A. Myers, Scott E. Hudson, and Randy Pausch Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University http://www.cs.cmu.edu/~bam bam@cs.cmu.edu

2. Introduction • User Interface Software Tools • Help developers design and implement user interfaces • Focus on Tools, but influenced by future UIs • Today’s tools are highly successful • Window Managers, Toolkits, Interface Builders ubiquitous • Most software built using them • Arebased on HCI researchBrad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html • Future tools must be different

3. Talk Outline • Historical Perspective • What worked • What didn’t catch on • Why • Lessons Learned • Future Prospects and Visions • UI Trends that will require new tools • Important issues

4. Historical Perspective • Themes • Address the useful & important aspects of UIs • Tools that succeeded helped (just) where needed • Threshold / Ceiling • Threshold = How hard to get started • Ceiling = how much can be achieved • Path of Least Resistance • Tools influence user interfaces created • Predictability • If not predictable, then not accepted by programmers • Moving Targets • Changing user interface styles makes tools obsolete

5. What Worked • Window Managers and Toolkits • Event Languages • Graphical, Interactive Tools • Component Architectures • Scripting Languages • Hypertext • Object Oriented Programming

6. Window Managers • Multiple (tiled) windows in research systems of 1960’s: NLS, etc. • Overlapping introduced in Alan Kay’s thesis (1969) • Smalltalk, 1974 at Xerox PARC • Successful because multiple windows help users manage scarce resources: • Screen space and input devices • Attention of users • Affordances for reminding and finding other work

7. Toolkits • A collection of widgets • Menus, scroll bars, text entry fields, buttons, etc. • Toolkits help with programming • Help maintain consistency among UIs • Key insight of Macintosh toolbox • Path of least resistance translates into getting programmers to do the right thing • Successful partially because address common, low-level features for all UIs • Address the useful & important aspects of UIs

8. Event Languages • Create programs by writing event handlers • Many UIMSs used this style • Univ. of Alberta (1985), Sassafras (1986), etc. • Now used by HyperCard, Visual Basic, Lingo, etc. • Toolkits with call-backs or action methods are related • Advantages: • Natural for GUIs since generate discrete events • Flow of control in user’s hands rather than programmer’s • Discourages moded UIs • Won’t work well in future

9. Graphical Interactive Tools • Create parts of user interface by laying out widgets with a mouse • Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT • Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors” • Advantages: • Graphical parts done in an appropriate, graphical way • Address the useful & important aspects of UIs • Accessible to non-programmers • Low threshold

10. Component Architectures • Create applications out of components which are separately developed and compiled • In UI software, each component controls an area of the screen • Example: drawing component handles picture inside a document • Invented by Andrew research project at CMU (1988) • Now: OLE, OpenDoc, ActiveX, Java Beans • Address the useful & important aspects of UIs • Just the “glue” to hold together components

11. Scripting Languages • First GUIs used interpreted languages • Smalltalk, InterLisp • Rapid development, supports prototyping • Low threshold • Then C and C++ became popular • Now, bringing back advantages in scripting languages • tcl/tk, Python, perl • Visual Basic, Javascript • But language must contain general-purpose control structures

12. Hypertext • Ted Nelson named it in 1965 and developed Hypertext system at Brown University • Important systems: NLS (1967), Hyperties (1986) • World-Wide Web • Phenomenal success due to: • Ease of use of Mosaic browser • Support for embedded graphics • Support for easy authoring • Low threshold both for authoring and viewing

13. Object Oriented Programming • Success of OO owes much to UI software field • Popularized by Smalltalk • GUI elements (widgets) seem like objects • Have state, accept events (messages) • Rise parallels GUIs • C++ with Windows 3.1 • Java for behaviors in WWW

14. What Hasn’t Caught On • User Interface Management Systems • Formal Language-Based Tools • Constraints • Model-Based and Automatic Techniques

15. User Interface Management Systems • Original goal: like databases, provide high-level language that abstracts details of input and output devices • This separation has not worked in practice • Good user interfaces must take into account the pragmatics and detailed behavior of all objects • Standardization of GUI input and output devices has made goal somewhat moot • Doesn’t address the useful & important aspects of UIs

16. Formal Language Based Tools • Early UIMSs used grammars and state-transition diagrams • Focus on dialog management • Moving Targets • Direct manipulation made dialog management less important • Path of Least Resistance • State diagrams afford worse user interfaces • High threshold • Formal languages are often hard to learn

17. Constraints • Declare a relationship and system maintains it • Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996) • Predictability • Constraint networks can be hard to debug • Especially in multi-way constraints • Programmer must specify (or deduce) solving order • High threshold • Constraints require thinking differently • May be appropriate for graphical layout • Address the useful & important aspects of UIs

18. Model-Based and Automatic Techniques • Automatic techniques for generating UIs from a model or declarative specification of contents • Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993) • Try to separate specification of UI from content • May provide automatic reformating, retargeting, customization to users, etc. • Result is often unpredictable • Often can be worse UI than hand-drawn • Sometimes model is larger than the code it would replace

19. Discussion of Themes • Address the useful & important aspects of UIs • Narrower tools have been more successful than ones that try to do “everything” • Do one thing well • Threshold / Ceiling • Research systems often aim for high ceiling • Successful systems seem to instead aim for a low threshold • Impossible to have both?

20. Discussion of Themes, cont. • Path of Least Resistance • Tools should guide implementers into better user interfaces • Goal for the future: do this more? • Predictability • Programmers do not seem willing to release control • Especially when system may do sub-optimal things • Moving Targets • Long stability of Macintosh Desktop paradigm has enabled maturing of tools • We predict a change soon

21. Future Prospects and Visions • Important Trends • Computers becoming a commodity • Ubiquitous Computing • Move to recognition-based interfaces • 3-D interfaces • End-user customization and scripting • Violate assumptions of today’s tools • Assumptions limit what designers can do • Often unrecognized • Implications for future tools

22. Computers Becoming a Commodity • There are no longer “high-end” computers • Computer Science research’s trick of buying faster computers to “time travel” may no longer be possible • Moore’s law continues to operate • New opportunities • Quantitative change makes qualitative change possible • Enables the diversity of platforms • UIs more cinematic • Smooth transitions, animation, visual

23. Ubiquitous Computing • Computation embedded in many kinds of devices • Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms • Next wave: easy communication with radio • E.g., BlueTooth: www.bluetooth.com • Significant Implications for tools • Tools for coordinating multiple, distributed,communicating devices • “Multi-computer” user interfaces • Moving target problem

24. Varying Input and Output • Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels • Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels • Cell phone to Stanford’s wall (3796 x 1436 pixels)

25. Need New Interaction Techniques • Interaction techniques for desktop will not work • No room on small devices • Can’t reach menubar on wall-size devices • Want to run same application on different devices

26. Need for Prototyping Devices • User interface will be in hardware • Rapid design and prototyping needed for hardware • Pragmatics and usability cannot be evaluated from a simulation on a screen

27. Multiple, Distributed, Communicating • Computers more for communication, not for computation • Already true for WWW, email, digital pagers, cell-phones • Computers as intermediaries between people • CSCW • But can’t assume have similar systems • Single person with multiple devices • Room-area networks like BlueTooth or HomeRF • People communicating with themselves • Tools will need to help with data sharing and synchronization

28. Limitations of Today’s Tools for UbiComp • Tools assume a Pointing Device • Hidden reliance on specific characteristics of common devices • Size of display • Many tools cannot handle a different number of mouse buttons • Change to a stylus on a touchpad requires different techniques • Assumptions about the setting • Assume user is sitting and looking at UI • Assume has user’s full attention

29. Move to Recognition-Based Interfaces • Speech, gestures, camera-based vision • Multimodal interaction • User will pick which modality to use • Use multiple modalities at same time • Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. • Need tools to hide these complexities

30. Fundamental Differences ofRecognition-based UIs • Input is uncertain • Recognition can make errors • Requires monitoring, feedback, correction • Interpreting input requires deep knowledge of data • Context of the application • “Move the red truck to here”

31. Implications of Recognition-based UIs • GUI event model no longer works • Do not produce discrete events • Separation of UI from application no longer works • Need a architecture based on accessible application data structures • “Reflection”, “Open Data Model”

32. 3-D Interfaces • Difficult to design the right abstractions for tools • Demise of VRML for Web • Need to settle on the 3-D widgets and interaction techniques that will be standard • Requirement for near-real-time interactivity • Need to hide the mathematics

33. End-user Customization and Scripting • Spreadsheet enables end users to specify their own computation • Visual Basic, other “scripting” languages • Needed in all applications • Threshold for programming is too high • Need “gentle slope systems”

34. Gentle Slope Systems Programming in C Visual Basic Director (v6) HyperCard C Programming MFC C Programming DifficultyofUse xCmds Click andCreate Lingo HyperTalk Goal Basic Sophistication of what can be created

35. More Assumptions of Today’s Tools • Skill and Dexterity of users • Older users • Makes single, fixed library of widgets untenable • Non-overlapping and opaque components • Preclude translucency, magic lens interactions • Fixed libraries of components (widgets) • Creating new widgets is very difficult • New devices will require new interaction techniques • Interactive tools provide freedom of design • Aim for “Mechanism not Policy”

36. Operating Systems Considerations • What is in the OS? • Window Manager? Toolkit? Communication? Scripting facilities? • Need ever increasing services for applications • Need more access to low-level information • E.g., hardware buttons, whether on network • Ideally, API to support competition and research into these components

37. Some Design Guidelines for Future Tools • Many things require further research • Organize around providing rich context • Of application and device state • To inquire about data & methods; “reflection” • Enables EUP, Recognition-Based UIs, data sharing for UbiComp • Rather than event-based

38. More Design Guidelines • Replaceable User Interfaces • Ability to have multiple UIs • Enabled by procedural interface to everything in UI • Enables UbiComp devices, EUP • Aim for low threshold, rather than high ceiling • But cover the right parts of the interface • Predictable for programmers rather than “smart” or automatic • Need for support for evaluation

39. Conclusions • Research in tools necessarily trails innovation in UI design • Due to consolidation on desktop metaphor, significant progress in tools • UI design poised for radical changes • New opportunities and challenges for tools