21. Direct Manipulation

1. 21. Direct Manipulation • What is Direct Manipulation? • 1. Original Definitions and Claims • graphical interfaces operated directly using manual actions than typed instructions • continuous representation of the object of interest • physical actions or labeled button presses instead of complex syntax • rapid incremental reversible operations whose impact on the object of interest is immediately visible • usability benefits  learnability; enhanced expert performance; memorability; fewer error messages; better feedback; reduced anxiety; increased control • What makes Manipulation Direct? • Directness = Engagement + Distance • engagement  the perceived locus of control of action within the system • distance  the mental effort required to translate goals into actions at the interface and then evaluate their effects – gulf between goals and actions • Data and Development • Tests Uncritical Comparative Evaluation • measuring the usability of a direct manipulation interface against one or more alternative interfaces • advantages – word processing tasks, file manipulation tasks, database retrieval tasks

2. no differences – file management, drawing, matching concepts and labels • disadvantages – table manipulation, filing and retrieval, browsing Critical Comparative Evaluations • measuring the the usability of several different implementations of a direct manipulation interface against each other and against alternative interfaces Naturalistic Choice Studies • measuring interactional preferences for direct manipulation and alternative methods in a mixed mode interface • direct manipulation was often used to avoid typing long object names in the dialog box, and natural language was used to refer to objects which were not visible on the screen • Mixed Mode Interfaces • how to best mix manual and conversational forms of interaction in hybrid interface design  shift the locus of control from whether to when and how • Theory The Value of Mixed Mode Interaction • mode of interaction metaphors (Hutchins, 1989) • conversation metaphor – the character of utterances in a conversation about the task at hand – limited because users have to learn a new language, maintain a mental model of the world, converse in a very different manner • declaration metaphor – the character of speech acts which magically cause things to happen in the world of interest – limited because it depends on a practice effect in using a conversational interface • model-world metaphor – the character of actions taken in the world of interest – limited because of its directness in collapsing abstract descriptions into concrete actions

3. collaborative manipulation metaphor – the character of carrying out a task with someone elses help limitations with principles • Laurel (1990) – reintroduction of a conversational metaphor within a model-world context  visible intermediary within a model-world -- interface agent defined as a character enacted by a computer, who acts on behalf of the user in a virtual (computer-based) environment • Whittaker (1990) Principle 1 Continuous representation of the object of interest – the problems of locating objects that are not visible – manual search inefficient Principle 3 Rapid incremental reversible operations – leads to inefficiencies in the execution of common compound actions which might be done faster through a single command Principle 3 whose impact is immediately visible – does not allow for processes such as reminders or mail forwarding Further Insights on Directness • the directness of manual interaction (Johnson et al., 1989) with Xerox Star system • seeing and pointing over remembering and typing through the graphical interface • don’t be dogmatic about the desktop metaphor and direct manipulation • the role of icons in representing metaphors (Familant and Detweiler, 1993) • a sign that shares characteristics with the objects to which it refers • icons have lost the shared characteristics with referent objects because they have no real world counterparts (highly abstract) • Desain (1988) equates distance with directness as a measure of interfaces • direct manual interfaces  well-known graphical formalism with real-world actions • direct conversational interfaces  natural language syntax with the jargon vocabulary

4. cognitive directness (least cognitive effort) and social directness (least collaborative effort)  cryptic conversations • Two Philosophies, Two Debates • Separating Directness and Manipulation • how and when to utilize manual forms of interaction at the interface • how by critical tests comparing different implementations of manual interfaces with each other and by discussions of directness  some features of manual and conversational interaction more effective than others • when by uncritical tests comparing manual with conventional interfaces for the same tasks, and by the work on mixed mode interfaces  manual interfaces are not always better than conversational • directness is now equated with distance and manipulation with engagement • re-conceptualization of the space of interfaces • directness philosophy relating to what makes an interface easy to use • manipulation philosophy relating to why manual interaction is preferable to conversational • A New Philosophy of Directness • manual interaction -- addition of real world metaphor principle • A New Philosophy of Manipulation • utilize manipulation selectively!

5. 24. Design of Menus • Menu-Driven Interfaces • a set of options, displayed on the screen, where the selection and execution of one (or more) of the options results in a change in the state of the interface • Designing a Single Menu Panel • Three Types of Comparison Operations • identity searching – a specific target that is literally displayed as one of the options • class-inclusion matching – at the root or other top-level panels of a hierarchical menu • equivalence search – at the leaves of bottom levels of menu systems Identity Matching • Perlman (1984) – alphabetical vs. random order • Card (1982) – alphabetical, categorical, random Equivalence Matching • know the name of the target -- either an alphabetized or a categorized menu. • uncertain about the name of the target -- categorized lists better than alphabetized lists Class-Inclusion Matching • conceptual overlap between the categories • don’t seem to fit well into any of the available categories

6. Aiding the Comparison Operator Adding Descriptors • Lee et al. (1984): effective with limited experience  longer search time and space • Dumais and Landauer (1983): the examples provide very little info beyond that which could be inferred from the category name alone Using Icons • three possible advantages over verbal options • parallel search and no cost associated with a large number of options • categorizations of pictures can be faster than of words • can provide additional info that increases the accuracy of selections • distinctive icons vs. words and representational icons • icons and pictures should be used very selectively • Guidelines for Organizing and Naming the Options on a Single Panel Organization • random, alphabetical, categorical organizations • conventional order • frequency of use – Zipf’s law (1949) – frequency is a negative power function of their rank

7. Naming • precise naming • adding a descriptor – the magnitude of the benefit understood poorly • for large menu-driven interfaces, testing the names on each menu panel with a sample of end users is very costly, but it is the only technique guaranteed to remove all of the clinkers • Organization and Navigation Between Menu Panels • Depth versus Breadth in a Hierarchical Menu Structure Factors Favoring More Breadth • three reasons for considering a system with greater depth • crowding – the amount of available space on a panel • insulation – the opportunity to prompt selections that are likely to be needed and hide those that are unlikely or illegal • funneling – a reduction in the total number of options processed that is achieved by designing a system with more depth and less breadth A General Framework for the Depth-Breadth Tradeoff Lee and MacGregor’s Linear Model • optimal breadth with exhaustive search – 3 to 8 • optimal breadth with self-terminating search – 4 to 13 Paap and Roske-Hofstrand’s Linear Model • in the range of 16 to 36 and sometimes as high as 78

8. Varying Depth • Miller (1981) -- performance is best at the intermediate levels of depth  two levels with eight options is the best -- Snowberry et al. (1983) • Kiger (1984) – 28, 34,82, 16x4, 4x16  performance decreased as depth increased • Varying Breadth Across Levels • Norman and Chin (1988) – constant (4x4x4x4), decreasing (8x8x2x2), increasing (2x2x8x8), convex (2x8x8x2), concave (8x2x2x8) • searching for specific targets  the increasing menu was slightly superior • fuzzy targets – concave>increasing>constant>decreasing>convex • more breadth on the bottom – the least amount of uncertainty across the top three levels  the superiority of the concave menu over the increasing menu • recommendation – allocate the most breadth to the bottom panels • Performance Changes Across Levels of the Hierarchy • Snowberry et al. (1983) – higher error rates at the top  more abstract and ambiguous than thoseat the bottom • Semantics and Syntax • errors are usually caused by labels that are not natural and precise • the semantics of a menu system (the quality of the names for categories and terminal options) are far more important than the syntax (the depth-breadth structure of the menu system)

9. 28. User-Centered Evaluation • Introduction • shift from hypothesis testing and statistical analysis to information gathering for iterative design  shift focus to formative rather than summative evaluation • The Changing View of Evaluation • evaluations – object, process, purpose – subjective or objective • Evaluation Processes • context independent measurements of experience -- usability • evaluations as the result of a process with a purpose in a context focused on an object • Evaluating Software for Usability • Why Evaluate Usability? • usability evaluation – take time and cost money -- function, cost and schedule • usability evaluation activities are not yet structurally integrated into most development processes • characteristics of the user, the work environment, the documentation or on-line assistance • Usability Information Sources • the change from “how good is it” to “identify the problems and solutions” • the end goal is a redesigned system that meets users are able to achieve their goals and are satisfied with the product

10. User-Centered Evaluation • evaluations conducted during system development are generally aimed at providing information for improvements in the existing design – verbal reports, walkthroughs, direct tests of interface alternatives • evaluations conducted on completed systems are often less interested in information about specific details and seek information of a more general nature – surveys, questionnaires, general descriptive studies • Verbal Reports/Think-Aloud Evaluations • anyone can collect them without specific training, and they can be much more easily manipulated than other behavioral measures • as valid as other behavioral measures • considered as an excellent way to gain access to the contents of someone’s short term memory – time is very important • think out loud – mental walk – questions that call for LTM retrieval or that call on highly automated responses will not provide rich verbal reports • difficulties in scoring and analysis -- questionnaires • Surveys and Questionnaires • easiest and least expensive method • questions specific rather then general, actual system experience rather than possible system changes or extensions • selections on a scale with a limited number of points • questionnaires are administered after an experience -- recollections may be distorted

11. Use Data Collection • other measures (such as errors, task completion times, requests for help, general logging data) can be useful • often used in more formal quantitative analysis – empirical data • Design Walkthroughs • using low-fidelity, paper and pencil or storyboard mockups of a system under development • representative end users, programmers, architectures, usability engineers • the team uses a prototype of the system to walk through a series of typical end user tasks • data can be collected in the form of expected problems or recommended design alternatives • the key challenge of the design review is to consider all elements of the software system, including errors conditions and assistance seeking behavior • Theory-based Reviews • can substitute use based evaluations with analysis alone to provide insight into predicted use – KLM, GOMS

12. Use Inspection Based Techniques • whether to use empirical usability testing or inspection methods or both • it is not reliable to obtain user performance data with inspection as compared to testing methods  theory-based techniques more reliable for relative comparisons than for absolute predictions • Karat et al. (1991)  the usability problems that an inspection method missed were relatively severe • Wharton et al. (1992)  inspection methods can identify more potential problems that do not present themselves in a significant way in real use

13. Distributed Cognition • INTRODUCTION • a perspective on all of cognition, encompassing interactions between people and with resources and materials in the environment • the boundaries of the unit of analysis for cognition • the range of mechanisms that may be assumed t participate in cognitive processes • A DISTRIBUTED COGNITION APPROACH • Socially Distributed Cognition • the cognitive properties of societies of individuals – social organization is itself a form of cognitive architecture • social interactions as well as interactions between people and structure in their environments • Embodied Cognition • the organization of mind is an emergent property of interactions among internal (memory, attention, executive function) and external resources (the objects, artifacts, and at-hand materials constantly surrounding us) • Culture and Cognition • the study of cognition is not separable from the study of culture, because agents live in complex cultural environment • distributed cognition returns culture, context, and history to the picture of cognition

14. Ethnography of Distributed Cognitive Systems • cognitive ethnography in 60’s and 70’s focused on the meanings of words – sought in the contents of individual minds – extends to the material and social means of the construction of action and meaning • event-centered ethnography – not only what people know, but how they go about using what they know to do what they do • not enough to know how the mind processes info – necessary to know how the info to be processed is arranged in the material and social world • AN INTEGRATED FRAMEWORK FOR RESEARCH • CONCLUSIONS AND FUTURE DIRECTIONS • HCI exclusively focused on single individuals interacting with applications derived from decompositions of work activities into individual tasks • distributed cognition – understanding interactions among people and technology  ethnographic studies of the phenomena f interest and with natural histories of the representations

15. 23. Ubiquitous Computing • INTRODUCTION • the accomplishments and remaining challenges for three themes • natural interfaces • context-aware computing • automated capture and access for live experiences • everyday computing • COMPUTING WITH NATURAL INTERFAES • natural actions can and should be used as explicit or implicit input to ubicomp system • speech-related interfaces, perceptual interfaces, pen-based or free-form interactions, grasp-able or tangible interfaces • First-Class Natural Data Type • natural interfaces – audio, video, ink, and sensor input • structured gestures, ability to merge independent strokes together as they form letters, words, and other segments of language • Error-Prone Interaction for Recognition-Based Interaction • Error reduction – improving recognition technology in order to eliminate or reduce errors • Error discovery – thresholding of confidence measures, historical statistics, explicit rule specification • Reusable infrastructure for error correction -- toolkits

16. CONTEXT-AWARE COMPUTING • location – GPS-based car navigation systems, hand-held tour guide • individual objects – barcode or identifying tag, vision-based recognition • What is Context? • who, what, where, when, why • Representations of Context • how to represent context • The Ubiquity of Context Sensing – Context Fusion • few truly ubiquitous, single-source context services • assemble context information from a combination of related context services – sensor fusion • Coupling Context-Aware and Natural Interaction – Augmented Reality • “probing the world with a tool” metaphor • AUTOMATED CAPTURE AND ACCESS TO LIVE EXPERIENCES • Challenges in Capture and Access • Capture • wished we had a camera, difficult to find the picture or film • informal setting -- not documented properly; formal meetings – poorly captured • raw streams of info that are captured mainly for the purpose of direct playback • Access • playback in real time – pinpoint a particular topic, summarization of experience • synchronization of multiple captured streams during playback -- foreshadowing

17. TOWARD EVERYDAY COMPUTING • results from considering the consequences of scaling ubicomp with respect to time • they rarely have a clear beginning or end • interruption is expected • multiple activities operate concurrently • time is an important discriminator • associative models of information are needed • Synergy Among Themes • Research Directions in Everyday Computing • Design a continuously present computer interface • wearables – continually worn interface, but limited by the current input and display technologies and are text-based interfaces • Presenting info at different levels of the periphery of human attention • generic peripheral backdrop with no mechanism for the user, or the background task, to move the peripheral info into the foreground of attention • Connecting events in the physical and virtual world • Modifying traditional HCI methods to support designing for informal, peripheral, and opportunistic behavior • ADDITIONAL CHALLENGES FOR UBICOMP • Evaluating Ubicomp Systems • Finding a Human need • reliable system to evaluate

18. build a compelling story, from the end-user’s perspective, on how any system or infrastructure to be built will be used – the basis for evaluating the impact of a system on the everyday life • feasibility study – how a system is being used, what kinds of activities users are engaging in with the system, whether the overall reactions are positive or negative • Evaluating in the Context of Authentic Use • Task-Centric Evaluation Techniques Are Inappropriate • Social Issues for Ubiquitous Computing • who can access and modify the contents – security and encryption schemes • the lack of knowledge of what some computing system is doing – invisibility • control the distribution and use of the info  when and what to capture • privacy • CONCLUDING THOUGHTS

19. 6. Cognitive Models • Advantages for HCI • cognitive models in three main ways in HCI • to help examine the efficacy of different designs to predict task performance • to provide assistance – embedded assistant • to substitute for users • difficulty of connecting the cognitive models to their task environment • A Route to Supporting Models as Users • Artifacts of the Cognitive Modeling Process • produce a cognitive model that performs like a human • can be viewed as producing three artifacts • the cognitive model itself, which simulates the cognitive performance and behavior of a human performing the task • task application or its simulation • mechanism that supports interaction between the model and the task simulation  simulates human perception and action • Role of User Interface Management Systems • tools used to develop user interfaces – a toolkit to support the cognitive modeling process • a tool to create interface • a run-time mechanism that lets the cognitive model interact with the task simulation • a communication mechanism between the cognitive model and the task simulation

20. Cognitive Model Interface Management Systems • extend the cognitive model by adding a simulated eye and a simulated hand for perception and action • link the cognitive model to the simulation • A Functional Model Eye and Hand • most important parts of the CMIMS – visual acuity and the speed of motor movements • Example Cognitive Models That Perform Interactive Tasks • A Simplified Air Traffic Control Model • explore how to create a general eye and investigate what a model would do with an eye • the basic task involves learning how to direct a single aircraft to land at an airport located at the center of the display • a crucial element of the task is that change of heading command must be issued at the appropriate time, which requires that the cognitive model be able to detect when an aircraft is approaching a way marker (appear on the screen as crosses) • Summary • a simple but functional Sim-eye (Soar-IO) could be created using an existing UIMS • the eye is not just the fovea; the periphery is needed even for simple search • Tower of Nottingham Model • Sim-eye and a pair of Sim-hands (grasp, release, rotate) • Summary • the Sim-hand and Sim-eye are generally applicable and that they can be a different cognitive architecture – ACT-R

21. Related Systems • EPIC (Kieras and Meyer, 1997) – used to make accurate predictions of interaction behavior  capabilities and regularities used by Soar and ACT-R/PM • ACT-R/PM is more concerned with detailed psychological predictions, but it is not yet positioned to be a tool that can be widely used to develop user interfaces • Cognitive Models as Users in the New Millennium • Implications for Models • interaction – where to look and what to do with what they see  peripheral vision • after focusing on functional capabilities, the next is to incorporate more experimental regularities to model human performance • Implications for Interfaces • two ways CMIMSs facilitate the improvement of Uis • cognitive models can be used to evaluate user interfaces • the ability to tie more accurate user models to interfaces opens up a range of applications, such as more accurate intelligent assistants to help novices

22. GOMS • GOMS AS COGNITIVE MODELING • Card et al. proposed a framework for building analytic models of human performance with computers – two key components • a general characterization of the human information-processing system, in term of both a system architecture and quantitative parameters of component performance  Model Human Processor (MHP) • the GOMS model – four cognitive components of skilled performance in tasks • strength – its ability to predict the time it takes a skilled user to execute a task based on the composite of actions of retrieving plans from LTM, choosing among alternative available methods, keeping track of what has been done and what needs to be done, and executing the motor movements necessary for the keyboard and mouse  routine cognitive skills can be described as a serial sequence of cognitive actions and motor activities • Card et al. found parameters that were very consistent across tasks: • a keystroke (k) for a midskilled typist -- 280 msec. • a single mental operator (M) from LTM to WM – 1.35 sec • pointing (P) – average 1.1 sec (Fitts’ law) • moving the hand (M) from keyboard to the mouse – 400 msec • Limitations of the GOMS Approach

23. ADVANCES IN MODELING SPECIAL SERIAL COMPONENTS • Motor Movements Keying • depending on the skill level of the typist, the frequency with which the particular key is used, and the predictability and continuity of the text to be typed • average typist – 280 msec per keystroke Moving a Mouse • distance and target size – 1.1 sec (Card et al., 1983), following Fitts’ law An Example of the Application of GOMS and MHP to Design Generation • the time to select an item was far shorter when the menu popped up to the right of the cursor than when the menu appeared below it • Fittsized menus (the target size grows as the distance from the cursor’s starting position increases) vs. putting a virtual border on the top, right, and bottom edges of the pop-up menu  1.9 sec vs. 450 msec Hand Movements • the time needed to move from the space bar of the keyboard until the pointing control begins to move the cursor – 360 msec • Perception • the perceptual processor at 100 msec and a saccade at 230 msec • perception of words includes recognition, some verbal encoding, or retrieval of meaning in addition to simple perception • scanning, storing, and retrieving – 2.3 sec

24. Memory and Cognitive Processes Memory Retrieval • retrieval of well-known units from LTM for placement in WM, ready then to be either executed by a motor processor or further decomposed by subsequent retrieval from LTM – 1350 msec Executing Steps in a Task • the execution of each procedural step – 70 msec Choosing Among Methods • the more choices for a response, the longer the expected response time (Hick’s law) – 620 msec • Predicting Composite Performance From These Parameters • EXTENTION OF THE BASIC FRAMEWORK • Learning and Transfer Time to Learn • Kieras and Polson developed an extension of GOMS they called Cognitive Complexity Theory  provides a basis for making quantitative predictions about the time to learn and the amount of transfer  NGOMSL • time to learn each step – 30 sec; best guess is 25 sec per production Transfer of Training From One System to the Other • the number of productions the two systems share provides a good metric for predictions of the amount of transfer

25. The Analysis of Errors: Forgetting From Working Memory

26. Definitions and a Notation for GOMS Models • Goals • A goal is something that the user tries to accomplish. • a goal description is an action-object pair in the form: <verb noun> • Operators • operators are actions that the user executes • a goal is something to be accomplished, while an operator is just executed • Methods • a method is a sequence of steps that accomplishes a goal • Selection Rules • the purpose of a selection rule is to route control to the appropriate method to accomplish a goal • General Issues in GOMS Task Analysis • Judgment Calls • in order to do a useful task analysis, the analyst must make judgment calls on these issues • how users view the task in terms of their natural goals • how they decompose the task into subtasks • what the natural steps are in the user’s methods

27. Pitfalls in Talking to Users • people have only a very limited awareness of their own goals, strategies, and mental processes in general • what users actually do can differ a lot from what they think they do • what users actually do with a system may not in fact be what they should be doing • Bypassing Complex Processes • the approach presented here is to bypass the analysis of a complex process by simply representing it with a “dummy” or “placeholder” operator Representing Bypass Processes • a bypassed process could be represented just by defining a complex mental operator and using it wherever it is needed I the methods • the user’s task is only to interact with the system in order to carry out this completely described task, referring to the yellow pas as necessary to obtain the required information • Analyze a general Set of Tasks, Not Specific Instances • task scenarios • trace – the list of specific actions that the user would perform for a specific task • the goal of GOMS task analysis is a description of the general methods for accomplishing a set of tasks, not just the method for executing a specific instance of a task

28. A Procedure for Constructing a GOMS Model • top-down, breadth first expansion of methods