Principles of Language Use for User Centered Design of Software User Interfaces

1. Principles of Language Use for User Centered Design of Software User Interfaces Derek Brock Navy Center for Applied Research in Artificial Intelligence Naval Research Laboratory

2. A different design perspective • Designing a software user interface is like writing a reference book with many chapters and subsections • The Designer is the author • Users are the readers • Interaction starting points in the user interface—menu choices, documents, tool bar icons, etc.—are like chapters and subsections • The author’s goal in each chapter or subsection is to provide readers with: • Clear and useful expositions • Pointers to dependent and related references Derek Brock, Navy Center for Applied Research in Artificial Intelligence

3. What’s the point? • Software is designed by people to be used by people • The purpose of a user interface is to coordinate what the designer thinks users need to understand—and do—to use the software • But few users can read a designer’s mind • Any form of coordination between people requires some form of communication • Successful communication requires— • Collaboration (a willingness to work together) • Meaning and understanding • The use of various kinds of signals • A shared basis for the use of those signals Derek Brock, Navy Center for Applied Research in Artificial Intelligence

4. What goes on when we read a book? • A successful communication takes place— • The author and the reader carry out a kind of collaboration—both have to work at it • The author’s meaning (ideally) becomes the reader’s understanding • A variety of linguistic and nonlinguistic signals are used—text, diagrams, illustrations, etc. • The author and the reader begin with a communally shared basis for the signals used… • …which becomes a personally shared basis when the communication is complete Derek Brock, Navy Center for Applied Research in Artificial Intelligence

5. The same ideas apply to software use • When software use is successful— • The designer and the user carry out a kind of collaboration—again, both have to work at it • The designer’s meaning (ideally) becomes the user’s understanding • A variety of linguistic and nonlinguistic signals are used—labels, icons, actions, sounds, messages, etc. • The user also makes use of input signals—more on this later • The designer and the user begin with a communally shared basis for the signals used… • …which becomes a personally shared basis when the communication is complete Derek Brock, Navy Center for Applied Research in Artificial Intelligence

6. The larger picture • Signal use is really language use • People use language to do things with each other • People’s language use skills are their greatest resource whenever meaning and understanding are involved • The design of human-computer interaction is a representation problem that primarily involves the designer’s meaning and the user’s understanding • The principles of language use are a foundational framework for this design problem Derek Brock, Navy Center for Applied Research in Artificial Intelligence

7. Introducing the study of language use • This is not the study of linguistics • Linguistics is the study of the development and structure of languages, particularly spoken and written languages • Instead, it’s the study of what’s involved in how people use signals to do things together • How do people convey and grasp each other’s intentions? • How do people succeed in coordinating this? • How does this work when people aren’t face-to-face? • What are the cognitive and social processes involved? • How is language used in different conceptual domains of action? Derek Brock, Navy Center for Applied Research in Artificial Intelligence

8. First concepts and terms of art • Language use requires participants and raises the notions of speaker and addressee • In addition to signals, language use always involves speaker’s meaning and addressee’s understanding • Joint actions are atomic instances of language use • They are not composed of disjoint actions that are simply carried out by speaker and addressee independently • But, instead, are the coordinated result of actions that speaker and addressee carry out together, as a duet—even when this must happen across time and space • Joint activities emerge from sequences of joint actions • Interactive joint activities can never be fully planned in advance • This is the proper level for specifying roles, social purposes, goal hierarchies, etc. • Language use always takes place in a setting • Settings shape how language is used Derek Brock, Navy Center for Applied Research in Artificial Intelligence

9. Settings • The notion of a setting in language use combines notions of scene and medium • Scene—where the language is used • Medium—the manner of language use (e.g., spoken, written, recorded, gestural, a mixture, etc.) • Settings range in type, in their properties, and in what they require of people and the roles each person plays • The fundamental paradigms are: • Face-to-face settings—conversation, lecture, wedding… • Written settings—books, letters, movies, recorded music… • A joint activity’s setting determines how joint actions are coordinated and what skills are needed • Face-to-face settings are the most basic (or natural) • Written settings are the most demanding (or contrived) Derek Brock, Navy Center for Applied Research in Artificial Intelligence

10. Joint actions • Complete joint actions involve the coordination of a causally related ordering of cognitive, physical, and perceptual subactions (speakers  addressees) • Proposing an intention considering it (engagement) • Conceptually signaling the intention (meaning) recognizing it (understanding) • Presenting the signal identifying it • Executing the presentation attending to it • All coordinated joint actions between people require these subactions in some form; some examples— • A handshake, paddling a canoe, dancing, making music, etc. • Asking for directions, posting a notice, driving in traffic, etc. • To coordinate these subactions as parallel parts of the whole, people depend on their intuitions of downward evidence and upward completion Derek Brock, Navy Center for Applied Research in Artificial Intelligence

11. But wait… • How are signals used to convey meaning and achieve understanding? • How can a speaker be confident a particular signal will convey his or her intended meaning • How does an addressee converge on the intended meaning represented by the signal • It turns out this is a deep problem for computer science, but something that nature has solved rather well • Meaning and understanding depend on the notion of common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

12. Characterizing common ground informally • Common ground is knowledge people intuitively assume they can use with each other on a presumed basis of shared information • The idea is that when Alan and Zoe go to a movie… • …afterwards, they can presume they had mostly similar experiences of its content… • …and can assume that each will now understand the other’s use of this newly shared information • Shared experience naturally becomes a shared basis of information • In fact, this is exactly how people introduce and then use new ideas with each other • But even more is implied by this notion of a shared basis… Derek Brock, Navy Center for Applied Research in Artificial Intelligence

13. Awareness of a shared basis is the essence of common ground • Verification or inference of similar experiences also builds shared bases • Alan and Zoe don’t even need to see a movie together to use their knowledge of it with each other—they can simply find out if they have both seen it • Common ground acquired in this way often contains many discrepancies—but all common ground contains discrepancies • And note: Alan and Zoe’s separate knowledge of a movie isn’t common ground at all until they have evidence that confirms they’ve both seen it—that their knowledge is actually shared • In general, people can usually account for what they take to be common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

14. Building common ground • Common ground is built in joint activities • People expect each other to reason similarly and to maintain a similar awareness of what they are doing together • People expect each other to be engaged • Common ground in joint activities can be divided into at least three parts: • Knowledge initially taken to be held in common • Knowledge of the activity’s current state • Knowledge of what has conspicuously taken place so far • When common ground is missing, meaning and understanding immediately break down • But people are very good at repairing misunderstandings and filling in missing common ground • Building common ground is a lifelong social and personal process Derek Brock, Navy Center for Applied Research in Artificial Intelligence

15. Using common ground in joint actions: posing coordination problems • People implicitly justify their common ground by using what they take to be shared references to it • Shared references are simply signals that refer to elements of common ground; they function as coordination devices • Joint actions involving meaning and understanding are really coordination problems people pose and agree to solve together • People use coordination devices to pose coordination problems, focus their intent, and to indicate their solution • Most coordination problems in language use are familiar and most are readily solved • To make a coordination problem easy to solve— • It should have a straightforward solution when it is posed • Its coordination devices should fully indicate the solution • The solution should be obvious from what is common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

16. For example… • What does Zoe do if she and Alan are sharing popcorn at the movies and she wants a sip from Alan’s drink? • She can signal Alan either by asking him for a sip or by simply pointing to his drink in a way that draws his attention • By asking or gesturing, Zoe makes a clear and obvious shared reference to her common ground with Alan • Either ploy is an excellent coordination device because— • Zoe has a straightforward solution in mind that Alan can fulfill • Either device explicitly indicates her solution • Her solution is obvious given her common ground with Alan • In agreeing to Zoe’s coordination problem, Alan correspondingly expects that— • Zoe has an intended solution in mind (what he’s supposed to do) • Her signal contains enough information to solve the problem • He can figure out the solution on the basis of what is obvious in their common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

17. More about coordination problems • Codifying the joint premises for solving coordination problems: • Solvability (a straightforward solution exists) • Sufficiency (the solution is fully indicated) • Salience (the solution is obvious from common ground) • Working out coordination problems is a dominant part of people’s language use skills. • These skills have their origins in the perceptual richness of face-to-face settings • In choosing the right shared basis to use as a coordination device, joint salience is usually the most important consideration. • Appealing to this aspect of people’s attention in context greatly increases the likelihood of their converging on the intended solution. Derek Brock, Navy Center for Applied Research in Artificial Intelligence

18. Salient signals • Signals that make salient coordination devices with respect to common ground include: • Actions (including sounds and gestures) • These command perceptual attention • External representations • These are physical aspects of a language use setting that have conceptual meaning(s) for a joint activity; for example— • Playing cards, the elements of a board game, ceremonial objects, etc. • External representations enjoy perceptual immediacy • Conventions of use • Conventions are representations of standing solutions to frequently occurring coordination problems • When a convention is used as a coordination device, addressees usually know how to proceed immediately • Languages are full blown systems of conventions • Conventions of use give written settings their power Derek Brock, Navy Center for Applied Research in Artificial Intelligence

19. Layers—conceptual domains of action • Layers are shifts in the conceptual domain of action (and/or its setting) that shape the focus of common ground for purposes of solving particular kinds of coordination problems in joint activities • Layers depend on people’s abilities to change perspective • Storytelling illustrates the basic paradigm— • Primary layer • A storyteller and an audience use language in a real world setting as themselves • Secondary layer • The characters of the story use language in the story world setting • Note that the audience has no trouble shifting its common ground into the story world • Further layers within the story may arise • Layers can involve play acting, removes in space, time, and reality, persons not present, changes in setting, etc. Derek Brock, Navy Center for Applied Research in Artificial Intelligence

20. Summary of language use principles • Language use takes place in settings (e.g., face-to-face, written) that shape its coordination and its required skills • Language use is a joint activity; it is participatory as opposed to an individual activity • Joint actions are the unit of language use • They involve cognitive, physical, and perceptual levels of subactions • They are coordination problems • Coordination devices are signals people use to pose, focus, and solve coordination problems • The joint premises for solving coordination problems are solvability, sufficiency, and salience • Salient sources for coordination devices include actions, external representations, and conventions of use • Layers are conceptual shifts of perspective in joint activities that shape the focus of common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

21. Language use in human-computer interaction • Speaker and addressee in human-computer interaction are designer and user • Software use is their joint activity • This joint activity fully involves designer’s meaning and user’s understanding • Software use takes place in a written setting and makes use of an important secondary layer • Joint actions are the unit of software use • An entire class of joint actions in software use are called user interactions • User interactions and other joint actions in software use are simply coordination problems • Designing an effective system of coordination devices highlights the problem of building common ground Derek Brock, Navy Center for Applied Research in Artificial Intelligence

22. Thinking about software as awritten setting • As with any written setting, a software user interface is designed and produced in advance of when it is taken up by its user • It is useful to think of a written setting as a type of pre-coordinated joint activity • Conventional written settings build common ground through a static, serial narrative whose domain typically defines a secondary layer of action • The written setting of software use differs from this paradigm in three important ways • It has a non-serial narrative • Its presentation is dynamic (non-static) • Its secondary layer of action places the user in a type of face-to-face interactive setting Derek Brock, Navy Center for Applied Research in Artificial Intelligence

23. Characterizing the first two layers of software use • In the primary layer of software use— • The designer and the user pursue their joint activity (software use) as themselves in a written setting • In effect the user “reads” the designer’s presentation—the display portion of the user interface • In the secondary layer of software use— • The user participates in a face-to-face interaction with the computer (as opposed to the designer) • Particularly because of the change in setting, each layer makes different demands of the user’s language use skills Derek Brock, Navy Center for Applied Research in Artificial Intelligence

24. Characterizing the primary layer of software use in language use terms • The design poses a fixed set of coordination problems • Most software uses a cafeteria paradigm for user interaction • This paradigm implies a non-serial narrative • The design pre-coordinates the joint activity, but the user has opportunistic control of how the software use proceeds • The user interface generalizes to a huge menu of starting points • The presentation uses a language of coordination devices made up of a variety of linguistic and nonlinguistic signals, • These include elements of natural language, visual artifacts, and behaviors (i.e., actions and procedures) • Many elements of this language are (rightly) presumed to be conventions • The presentation serves, in part, as an external representation of the joint activity • Manipulable display elements are generally given specific conceptual meanings in the context of the software use Derek Brock, Navy Center for Applied Research in Artificial Intelligence

25. Building common ground in the primary layer of software use In a cafeteria-style presentation: • Both the designer and the user strive to find and build common ground, but… • The designer’s control over the process of building a complete body of common ground with the user in the planned manner of a serial narrative is sacrificed • The user’s common ground with the designer builds contingently (irregularly) on the basis of— • Opportunistically solving coordination problems posed in the interaction design • And, more generally, opportunistically becoming familiar with the presentation language • Despite the presentation’s non-serial narrative— • Over time, users generally develop a sense of where common ground is missing and will look for it unless the effort proves to be too costly Derek Brock, Navy Center for Applied Research in Artificial Intelligence

26. Designing coordination problems in the primary layer of software use • Coordination problems in a software design deliberately anticipate the user’s domain goals • But addressees’ knowledge in written settings can never fully anticipated • Users would benefit from an account of what the software can and cannot do • All coordination problems posed by a design should honor users’ expectations of salience, sufficiency, and solvability • From a practical standpoint, this is not always possible • Nontrivial designs generally involve multidimensional conceptual dependencies • Opportunistic user control pre-empts the designer’s control over the introduction of critical material • When users are unfamiliar with relevant supporting concepts, they may not know what to do through no fault of their own • Working solution: identify where common ground may be missing and provide users with immediate, context dependent access to it Derek Brock, Navy Center for Applied Research in Artificial Intelligence

27. Other primary layer issues in software use • Serial narratives naturally correspond to the third part of common ground—people’s record of what has openly occurred so far in their joint activity • Serial narratives in hard copy are— • Intuitively indexed; viewed as external representations, they easily represent their current state in this way • Readily open to review when misunderstandings, questions, or lapses of memory arise • These advantages are de-emphasized in the written setting of software use • The presentation may not always represent the current state of the joint activity • External representations in the presentation are generally dynamic (non-static) and actions are evanescent • Cafeteria organization of entry points obfuscates the user’s intuitive ability to index the presentation and/or locate particular functions • Little or no narrative record is kept for the user to consult Derek Brock, Navy Center for Applied Research in Artificial Intelligence

28. Characterizing the secondary layer of software use in language use terms • What we call “human-computer interaction” takes place in the secondary layer of software use • In this layer, the user participates in a direct interaction with the computer (as opposed to the designer) • The interaction is designed to resemble a joint activity in a face-to-face setting… • But this setting is not equivalent with a face-to-face setting between people • The user and the computer use different communication languages… • …and their perceptors are mismatched • External representations in the presentation are technically available to the computer but are generally unused by the system to justify common ground beyond, at most, small numbers of interactions • Joint actions between the user and the computer are designed to closely resemble joint actions between people Derek Brock, Navy Center for Applied Research in Artificial Intelligence

29. Digression—knowledge people accumulate about activities in general Claim: In any interactive activity, the knowledge people accumulate and are able to justify corresponds structurally to the three parts of common ground: • Knowledge initially held about the activity • Knowledge of the activity’s current state • Knowledge of what has conspicuously taken place so far in the activity • This accumulated knowledge of an activity— • Is common ground when the interaction takes place between people and shared informational bases exist to justify it as common ground • Is only an individual’s conception of the activity when the interaction takes place with elements of the environment that accrue no such knowledge • Is part common ground and part individual conception in human-computer interaction Derek Brock, Navy Center for Applied Research in Artificial Intelligence

30. Building common ground in the secondary layer of software use When the user engages the computer through a software user interface: • The interaction is designed to resemble a joint activity in a face-to-face setting • The user actively keeps track of contingent interaction knowledge corresponding to three parts of common ground as well as informational bases to justify it • At most, software keeps track of some of the same knowledge and shares informational bases in limited, ad hoc ways, such as undo mechanisms and other kinds of interaction histories • Contingent interaction knowledge the software fails to be able to justify and use cannot be construed as common ground in the secondary layer’s joint activity • Consequently, some of the user’s most valuable face-to-face language use skills go largely unused Derek Brock, Navy Center for Applied Research in Artificial Intelligence

31. Key points about language use for the design of software user interfaces • The written setting of software use differs from conventional, serially narrated written settings in several critical ways— • Its non-serial narrative structure builds common ground irregularly and introduces problems associated with conceptual dependencies • Dynamic and evanescent aspects of the presentation and its cafeteria organization reduce its hard-copy utility for the user for purposes of reviewing the software use • User interactions and other joint actions in software use are simply coordination problems • Almost all display elements of a user interface should be understood as coordination devices • Users expect all coordination problems posed by a design to honor the joint premises of salience, sufficiency, and solvability • Strategies for the accumulation of common ground with the user in the secondary layer remain ad hoc, at best Derek Brock, Navy Center for Applied Research in Artificial Intelligence

32. Practical language use advice for user centered design • Focus on coordination problems from a primary layer perspective • Identify coordination problems where common ground may be missing for the user • Provide the user with immediate links to context sensitive help; make it easy for the user to return from help to the task • Provide explanatory drop-down labels for icons (“tool tips”) and other external representations; do this even for cases where a terse standing label may be initially insufficient • Give users a simple, point-of-interaction facility for annotating their software use • Provide a hyperlinked index to the software’s set of functions (this can be part of a help mechanism) • Provide a deep undo mechanism • Be aware of the disorienting effect of interface clutter, the value of meaningful organization, the harm of unexplained evanescent behaviors, and the inevitable wrath of the user Derek Brock, Navy Center for Applied Research in Artificial Intelligence

33. Take home messages When the normal function of common ground in a language use setting is impaired, people are forced to work harder cognitively to accomplish their goals Designers of software user interfaces must strive to compensate users for a paradigm that, in primary layer of software use, inherently builds irregular common ground Building true common ground between the user and the computer in the secondary layer of software use awaits the future Derek Brock, Navy Center for Applied Research in Artificial Intelligence

34. Language useful references Brock, D. (2002). A language use perspective on the design of human-computer interaction. Proceedings: Workshop on Cognitive Elements of Effective Collaboration. Arlington, VA: Office of Naval Research, Code 342. Alterman, R. and Garland, A. (2001). Convention in joint activity. Cognitive Science (25)4,2001. Clark, H. H. (1996). Using language. New York, NY: Cambridge University Press. (read this if nothing else!) Hutchins, E. (1996). Cognition in the wild. Cambridge, MA: MIT Press. Norman, D. A. (1992). Turn signals are the facial expressions of automobiles. Reading, MA: Addison-Wesley. Norman, D. A. and Draper, S. W., eds. (1986). User centered system design. Hillsdale, NJ: Lawrence Erlbaum. Winograd, T. and Flores, F. (1986). Understanding computers and cognition. Reading, MA: Addison-Wesley. Derek Brock, Navy Center for Applied Research in Artificial Intelligence