Part V: Ease of Use Chapter 20: Intelligent User Interfaces for Ubiquitous Computing

Part V: Ease of UseChapter 20: Intelligent User Interfaces for Ubiquitous Computing Rainer Malaka, University of Bremen, Center for Computing Technologies (TZI)

Introduction • User interfaces can be challenging for both their users and their designers: • Even simple things like VCRs or TV-sets are often too difficult to use • Reviews and tests of consumer electronic devices very often rank bad usability even higher than technical aspects • Most modern appliances differ not much in their core functions • Here, we want to extend the question of user interface design in two directions: • the user interface should become more intelligent and adaptive and • We will discuss what “intelligent” means later on • we want more suitable interfaces for Ubiquitous Computing scenarios • An interface is by definition a clear boundary between two entities. In Ubiquitous Computing, we have the problem, that there might not be a clear boundary any more. Computers are no longer visible and at the end, they can disappear from the user’s conscious perception. We will, therefore, face the challenge to build an interface for something that is shapeless. Intelligent User Interfaces:

Introduction • Usability Engineering provides • general criteria that are valid for all computer products • established methods for the design process that can help to meet these goals. In particular: • participatory design and • iterative so-called human-centered approaches • The question on how to make interfaces more intelligent is not trivial: there are multiple approaches to enhance either the intelligence of the system or that of the user • Novel interface approaches follow the idea of embodied interaction • Emphasis has to be put on the situated use of a system and the mental models humans develop in their real-world environment Intelligent User Interfaces:

Building Good User Interfaces • Why do we need good user interfaces? • In general, the design of a good user interface is important for all products • legal constraints demand accessibility • many systems do not differ so much in their technical details and vendors have to diversify their products solely in terms of their “look and feel”. • Without Usability Engineering: • developers just implemented whatever they found useful for themselves • sometimes, the result is technology that is highly functional, but highly inefficient, frustrating or even unusable for many users • State of the Art: • Today usability plays a much bigger role • We have a wealth of methods, tools, and guidelines, which help to develop a good user interface (Dix et al., 1998; Mayhew 1999). There is not one single recipe whose application guarantees 100% success • The essence of usability engineering is to work iteratively in order to achieve the goal of better usability Intelligent User Interfaces:

Usability Goals (ISO 9241) • Safety and security • Good design should not harm users or other people affected by the use of a product. It should also help to avoid errors made by humans in using the system. • Effectiveness • A good user interface supports a user in solving a task effectively, i.e., all aspects of a task can be actually handled. • Efficiency and Functionality • A well designed and usable system should allow for quick and timely work. • Joy and fun • How enjoyable is it to work (or play) with the system? Is it fun or is it a pain to interact with it? • Ease of learning and memorizing • How fast can new users interact with the system and will they remember what they learned. Intelligent User Interfaces:

Usability Goals • There are more potential usability goals • Not all goals can be fulfilled to the same (high) degree: there are classic trade-offs • Examples: • in an interactive game, enjoyability is more important than effectiveness • In a system for firemen efficiency is more important than fun. • Another typical trade-off exists between efficiency and training. One solution can be to provide two modes: an expert mode and a novice mode • All efforts and goals of usability should be measurable • Since most usability criteria depend on the actual use of a system, there is a need to involve users in the design process • The more unknown the application terrain is, the more involvement of users is required, which is of particular importance for Ubiquitous Computing because there is not yet a large set of experience, studies, and guidelines Intelligent User Interfaces:

Models for User-Centered Design • Human-centered design • Design process that involves users • Develop an application iteratively with evaluations in every cycle • In contrast to the classical waterfall model in systems engineering (Royce, 1970): No segmentation into a linear order of clearly separable steps • Best approach when design goals are hard to be formalized • Roots lie in the participatory design idea from Scandinavia (Olson & Blake, 1981). • Multiple models have been proposed replacing the waterfall scheme by cycles or stars, i.e., the design process is open and decisions can be revised depending on user feedback during development (Gould et al., 1991; Hartson & Hix, 1989; Hix & Hartson, 1993) Intelligent User Interfaces:

Star Model for User-Centered Design (Hartson & Hix, 1989) implementation task analysis/ functional analysis evaluation prototyping requirements /specification conceptual design/ formal design Intelligent User Interfaces:

Models for User-Centered Design • Ubiquitous Computing and Human-Centered Design • Iterative development is inevitable since we cannot derive system requirements from interaction goals without user involvement. • UC aims at intuitively usable pervasive IT systems that assist users in their real-world endeavors • Many Ubiquitous Computing prototypes are completely technology-driven. Their developers focus on smart new gadgets, networks and infrastructure but they do not focus their design efforts onto their users. Just for the sake of plausibility some use scenarios and users are added to the design. Such systems will not leave the research labs and they will fail to find their market. • Design models consist of typical design steps as building blocks for good user interface design. They are very generic for interactive systems. Intelligent User Interfaces:

Design Steps for Building User Interfaces • Definition of the context • Consider the context of their envisioned product: including the way the system will be used, is it life-critical or for fun? Used in home or in office environments? • Market situation: What will be user expectations. Who is going to buy the product? • Description of the users • Based on the context definition, analyze each group of directly or indirectly affected users: What are their physical and cognitive abilities, their cultural and social background may affect the way they interact with the system? Special needs may play a role. • Task analysis • Derive a rather formal description of the tasks users want to solve from informal interviews and observations. Most importantly, designers should find out how users actually solve their task currently (not how they think they do it) and how they make use of tools at hand, how they communicate and how their context influences the course of activities. Intelligent User Interfaces:

Design Steps for Building User Interfaces • Requirements/Specification • This step would have been the first step of the classical software development process. For user-centered design, it is now based on a better understanding of the users, their context and their tasks. Specifications can be changed in each iteration based on insights from evaluations. • Conceptual design/formal design • The requirements and specifications are translated into system components. • Prototyping • Instead of “doing it right the first time” we iteratively build prototypes of the system. A prototype can be a very simple design sketch or an almost complete and working system depending on the stage of iterations in the design process. Intelligent User Interfaces:

Design Steps for Building User Interfaces • Evaluations • Evaluations are essential for assessing the progress of the design process and for deriving a better understanding of the tasks, the requirements and thus a better specification and implementation of the system (prototype). • Implementation, Tests, Maintenance • Whenever the iterations got to a stage where the prototype sufficiently fulfills the design goals, the final prototype (product) can be implemented. • Tests and maintenance are as important as in classical system engineering. Moreover, they can help to further improve the system and in particular user feedback after deployment can be used for defining new development cycles. Intelligent User Interfaces:

Evaluations, Alternatives, Prototypes • Where to start? • Do we need an implemented system for the first evaluation? • Fortunately: No! • Techniques for early evaluations: • Design sketches (e.g. on paper or on a blackboard): Users or usability experts evaluate early design ideas. Feedback can help to avoid basic mistakes and facilitates the mutual understanding of the users’ world and the system. • Wizard of Oz experiments: A human operator remote controls all functions of the environment/system and the test users are told they would already interact with the system. Wizard experiments are a cheap and can help to understand how people would interact in a real environment that is enhanced by Ubiquitous Computing technology. • Mock-Ups: A mock-up is a model of the system that already exposes the “look and feel” but does not yet include real functionality of the intended system. Early mock-ups for graphical interfaces can consist of a PowerPoint walkthrough through the system. • Prototypes: In contrast to the mock-up, the prototypes include actual functionalities of the target system. They may iteratively evolve to the final system. Intelligent User Interfaces:

Evaluations, Alternatives, Prototypes • How to proceed? • Evaluation is the core of the above mentioned “star-model”. • Depending on the maturity of the design, the budget and the nature of the system, a great variety of evaluation techniques can be used. • Evaluation methods can be classified according to the following dimensions: • Qualitative vs. quantitative methods • Studies in the field or in the lab • User tests or expert evaluations • System state (sketch, mock-up, prototype …) Intelligent User Interfaces:

Evaluations, Alternatives, Prototypes • When to stop? • The development cycle should not be an endless loop. • In general, the (re-) design-prototype-evaluation cycle can go on forever leading to a continuous increase of usability. • In practice, either the number of cycles is fixed beforehand or certain measures define when the loop has to be stopped and the final design is achieved. • Typically these measures would quantify the usability goals, e.g., • “95% of the test users rate the system as very convenient” • “the task completion rate within 30 min is 98%” • In some cases the stop-criterion is not bound to usability but to other measures such as • “we are out of budget” • “the deadline is next week” Intelligent User Interfaces:

Specific Challenges for UC Interfaces • Since the early days of Ubiquitous Computing, the usability has been in its focus. • Mark Weiser’s idea of Ubiquitous Computing encompasses “invisible” interfaces that are so naturally usable that they literally become invisible for the users conscious perception (Weiser 1999) • “invisible” does not mean “not there”, but rather present without conscious interaction • Examples for such physical objects are our body parts: • We do not have to think consciously about what we do with our arms • We just do the things we want • When we throw a ball, we just throw it and we do not think and plan how to make our hand grasp the ball and our arm swing around in order to accelerate the ball • In this sense, our arm is invisible but also very present Intelligent User Interfaces:

Specific Challenges for UC Interfaces • If we speak of Ubiquitous Computing interfaces that are “invisible” or computers that are “disappearing”, we actually speak of things that are present and “ready-to-hand”. However, the artifacts we interact with might not be consciously realized as computers • This notion goes back to the German philosophers Georg Gadamer and Martin Heidegger who call such interaction with things that we use without conscious awareness things that are “ready-to-hand” or at our “horizon” • In this phenomenologist view, the meaning of the things is actually derived from our interaction with them Intelligent User Interfaces:

Specific Challenges for UC Interfaces • Such a view on interactive artifacts has become popular in Ubiquitous Computing and is closely related to the notion of Embodiment (Dourish, 2001) • Embodied Interaction is a fundamental shift • from the classical positivist approach in computer science, i.e., modeling the real world in simplistic formal computer programs, • to an embodied approach that takes the user in the real world into account • This is relevant for Ubiquitous Computing for multiple reasons: • Ubiquitous Computing applications are to be used in complex real-world settings and their meaning (for the user) evolve in the course of action • Additionally, if things should become natural extensions of our physical abilities, they must be designed such that they do not need conscious interference from their users Intelligent User Interfaces:

Specific Challenges for UC Interfaces • A good example of such a “ubiquitous” technology is present in our homes already: electrical light • Whenever we enter a room that is dark, we just find a switch with our hands next to the door and the light goes on. • Without thinking we turn on the light. • We do not think of cables that conduct electrons. • We do not have to consider how the light bulb works or how they generate electricity at the power plant. • Instead, we have a very simplistic “mental model” on how to interact with the environment Intelligent User Interfaces:

Design Model and Mental Model Designer User System System Image User’s Mental Model Design Model • Humans have a very simplistic model how the thing works • Such “mental models” play are important for usability • Good design is about providing good mappings (Norman, 1998): • The design model must be mapped to the system image • Users must map their understanding/mental model to the system • The system must allow the user to map its image to the user’s model Intelligent User Interfaces:

Support of Mental Models • How can a system image support appropriate mental model? • Answer (with our notion of embodiment in mind): • bring the meaning of things into the things themselves • the meaning of something should be derivable from the interaction with it • or from its appearance that may signal some properties indicating how to use it • Such properties have been named affordances (Norman, 1998) • Many highly usable things that surround us just let us know by their physical appearance how we can use them: A chair for instance does not need a label or instructions on how to sit on it. We just see and know it is a chair and we know what to do with it. • Similarly, affordances have been defined as virtual affordances for computer interfaces and many metaphors on our computer screens signal functionalities, e.g., mouse pointers and scrollbars. Intelligent User Interfaces:

Support of Mental Models • Many Ubiquitous Computing objects include tactile interfaces or smart objects with physical and not just virtual properties, thus affordance becomes more literally a physical property • Consequences from the perspective of embodied interaction for UC: • Support mental models • Respect cognitive economy: humans re-use their mental models • Make things visible and transparent: it should be obvious what is going on • Design for errors: systems must assist users in finding a solution for their task even if something went wrong • Internal and external consistency: Things should work consistently • There are a number of further practical considerations and human factors that play a role for Ubiquitous Computing user interfaces. In the following, we will briefly highlight some of these aspects. Intelligent User Interfaces:

Human factors for UC • In Ubiquitous Computing we know less about the users and their situation than in classical human-computer interaction: • we do not know where the users are, • what tasks they are doing currently, • what other persons may be around. • This makes it very hard to account for some human factors that can have a huge effect on the interface and interaction design • Depending on the time, concurrent tasks etc. • the cognitive load, • stress level, • patience, • and mood may vary extremely. • Thus an interface can, in one situation be well-suited and in the other situation the user is either bored or overloaded Intelligent User Interfaces:

Human factors for UC • Space and Time: • In many Ubiquitous Computing applications, location and time play a crucial role. • Users need the right information at the right time and place. • Example: • In a system that helps a user in navigating her vehicle through a city, the information “turn right” only makes sense at a very well-defined point in space and time. An information delay is not acceptable. • User Attention: • In contrast to desktop applications, user’s are less focused on the device/screen • System message are easily overlooked • Events in the environment can distract the user • Other people and events are not the exception but the normal case in many UC scenarios Intelligent User Interfaces:

Human factors for UC and Adaptivity • Interfaces can try to take the users’ tasks into account and thus adapt their strategy to reach the user’s attention • E.g., when the user is driving a car, the system might interact in a different way than when the user is in a business meeting • However, when the system is literally ubiquitous, the number of tasks and situations the user might be in can be endless! • It is not feasible to model each and every potential situation • Solutions: • The system interface is adaptable to a few distinct modes of interaction • System learn and adapt themselves to behave differently in different situations • This can significantly increase the ease of use • E.g., A mobile phone that automatically adapts to the environment and stays silent in a business meeting, but rings in other situations is rather practical. Intelligent User Interfaces:

Who is in Charge? • Remember: usability goals can be in conflict to each other • More intelligent adaptive systems might lead to conflicts with: • Controllability: Is it the system or the user who controls the situation? • Support of mental models: How can a user still understand a very complex system? • Predictability: Humans want to be able to predict the outcome of their actions. If a system is too adaptive and autonomous, users get lost. • Transparency: If the system adapts to all sort of context factors, its state becomes less transparent. • Learnability: A system that learns and behaves differently in new situations can be hard to understand. • The developer and/or users have to decide, which goal is more important • User-centered design helps to find the right balance Intelligent User Interfaces:

Intelligent and Dumb Interfaces for UC • The term “intelligent user interface” has been debated and is not well-defined • Remark: The term “intelligence” is also not well-defined • What is an intelligent user interface? • In general, a technical solution can be called “intelligent” for two reasons: • (i) there is some built-in intelligent computation that solves some otherwise unsolvable problem; • (ii) Using the system, a user can solve an otherwise unsolvable problem, even though the system itself does not actually do anything intelligent. • Example: Calculating the logarithm of a number is a hard problem for a human • A calculator is a good example for case (i). It solves the problem for the human • An abacus would be an example for (ii). It empowers the user to solve the problem on her own Intelligent User Interfaces:

Intelligent and Dumb Interfaces for UC System System Interaction Interaction b) intelligent user interface (classical AI) a) classical interfaces • The classical approach of artificial intelligence (AI) is a rationalist one: • a system models the knowledge that human experts have and thus emulate human intelligence. • The “intelligence” moves from the user to the system Intelligent User Interfaces:

Intelligent and Dumb Interfaces for UC System Interaction • The classical AI approach is good if expert knowledge is rare and non-experts should also be able to work with a system • The embodied interaction view would rather try to make the interaction more intelligent c) embodied interaction Intelligent User Interfaces:

Intelligent and Dumb Interfaces for UC • New trends in AI focus on embodied intelligence as a property that emerges from the interaction of an intelligent agent with the environment • Even simple and light-weight agents can perform intelligent behavior without full reflective and conscious knowledge of the world • With respect to this definition, all of the above-mentioned material already describes how to build an intelligent interface. Because the processes for designing human-centered systems are just the right techniques for designing intelligent interactive systems, we already defined to a great extend how to build intelligent user interfaces! Intelligent User Interfaces:

Intelligent and Dumb Interfaces for UC System Interaction d) intelligent cooperative interface • Instead of leaving all intelligence to either the system, the user or the interaction, we can also try to get the best of all worlds and combine these techniques to a cooperative system • the system and the user cooperate with their knowledge on solving some tasks supported by intelligent interaction techniques Intelligent User Interfaces:

Techniques for enhancing the system’s intelligence • AI techniques can be used to put more knowledge into the system. • Besides state of the art IT methods such as databases, expert systems, heuristic search and planning, a number of more recent developments have caught a lot of interest by researchers and practitioners in the field: • World knowledge and ontologies: In particular for natural language interaction, ontologies provide resources for better understanding and reasoning. • User Adaptation: customization of systems where a single standardized interface is not appropriate • Context adaptation: context-dependent user interfaces can enhance the usability • Service federation: Integrating a variety of services and providing a single interface can be a great progress towards intelligent user interfaces • In principle all advanced semantic and AI-based methods with relation to user interaction can help to make systems smarter Intelligent User Interfaces:

Techniques for more intelligent interaction • Intelligent methods that aim at more natural interaction such that the interaction takes place on the premises of human communication rather than machine languages: • Multimodal interaction: is on the one hand more natural and on the other hand it also allows for more flexible adaptation in different usage situations • Cross-media adaptation: Intelligent interaction can support media transcoding that presents content on different media adapted to the situation. • Direct interaction: For many activities, direct interaction is superior to other forms of human-computer interaction. • Embodied conversational agents: anthropomorphic interfaces presenting animated characters can in some circumstances be very useful • This highlights some of the most important trends in the field. More ideas have been proposed and there will be more to come. Intelligent User Interfaces:

Techniques for amplifying the user’s intelligence • A really intelligent interface can leave intelligent decisions to the user and does not take away all intelligent work from the user by modeling it in the system • The question is: how can the system support users in acting intelligent? • The answers have been given already when we discussed usability goals: • A system that is easy to learn • where users have control and understand what is going on • where mental models are applicable • Systems that decrease the users’ creativity and enthusiasm: • Leave only minor steps to the user • do not tell about their states and how they got there • do not support appropriate mental models Intelligent User Interfaces:

Conclusions • The basics for building intelligent interfaces are techniques for building good interfaces • Consequently, we first presented an up-to-date introduction to methods of human-centered design • A central aspect of this technique is to iteratively design systems with repeated evaluations and user feedback • This approach is especially important for Ubiquitous Computing systems! • We presented a modern and sometimes controversial view on intelligent user interfaces Intelligent User Interfaces:

Part V: Ease of Use Chapter 20: Intelligent User Interfaces for Ubiquitous Computing