The Growth Of Cognitive Modeling in Human Computer Interaction Since GOMS

1. The Growth Of Cognitive Modeling in Human Computer Interaction Since GOMS Judith Reitman Olson and Gary M. Olson The University of Michigan Presenters: Tosin Aiyelokun and Norman Makoto Su

2. Outline • Cognitive Modeling • Introduction to GOMS • GOMS Extensions • Modeling Specific Components • GOMS Limitations • Summary

3. Outline • Cognitive Modeling • Introduction to GOMS • GOMS Extensions • Modeling Specific Components • GOMS Limitations • Summary

4. Cognitive Modeling: Definition • A theory that produces a computational model of how people perform tasks and solve problems by using psychological principles and empirical studies.

5. Philosophy, Logic, Linguistics, Mathematics, Computer Science Formal Analysis Artificial Intelligence Experimental Psychology, Neuroscience Cognitive Modeling: Research Methods

6. Cognitive Modeling: Role • Limits the design space • Answers specific design decisions • Estimates total task time • Estimates training time • Identifies complex, error-prone stages of the design • A means of testing current psychological theories

7. Cognitive Modeling: Human Information Processor (HIP) External World HIP Receptors (perception) Effectors (motor actions) Processor Memory

8. Perceptual Processor -sensory input (audio & visual) -code info symbolically -output into audio and visual image storage (WM buffer) Cognitive Processor -input from sensory buffers -access LTM to determine response -output response into WM Motor Processor -input response from WM -carry out response The Human Processor Model

9. Cognitive Modeling: Applications • GOMS • Today’s presentation • Soar • Integrated architecture for knowledge-based problem solving, learning and interacting with external environments. • ACT-R • Atomic Components of Thoughts - Rational

10. Outline • Cognitive Modeling • Introduction to GOMS • GOMS Extensions • Modeling Specific Components • GOMS Limitations • Summary

11. GOMS: Overview • Formal representation of routine cognitive skill. • A description of knowledge required by an expert user to perform a specific task. • Provides a description of what the user must learn.

12. GOMS: Classification • Provides a predictive, descriptive and prescriptive model • Predictive • Predicts the time it will take user to perform the tasks under analysis • Descriptive • Represents the way a user performs tasks on a system • Prescriptive • Guides the development of training programs and help systems

13. GOMS: Definition • GOMS models user’s behavior in terms of: • Goals • What the user wants to do. • Operators • Specific steps a user is able to take and assigned a specific execution time. • Methods • Well-learned sequences of subgoals and operators that can accomplish a goal. • Selection Rules • Guidelines for deciding between multiple methods.

14. GOMS: A Family of Models • Keystroke-Level Model (KLM) • Card, Moran, and Newell (CMN-GOMS) • Natural GOMS Language (NGOMSL) • Cognitive-Perceptual-Motor GOMS (CPM-GOMS)

15. GOMS: Keystroke-Level Model (KLM) • Simplest GOMS technique • The basis for all other GOMS techniques • Predicts execution time • Requires analyst-supplied methods • Assumes that routine cognitive skills can be decomposed into a serial sequence of basic cognitive operations and motor activities, which are: • K: A keystroke (280 msec) • M: A single mental operator (1350 msec) • P: Pointing to a target on a small display (1100 msec) • H: Moving hands from the keyboard to a mouse (400 msec)

16. KLM Example Top-level Goal: Edit Manuscript (move “quick brown” to before “fox”) Subgoal: Highlight text Operators: Move-mouse Click mouse-button Type characters (keyboard shortcuts) Methods:1. Delete-word-and-retype (retype method) 2. Cut-and-paste-using-keyboard-shortcuts (shortcuts method) 3. Cut-and-paste-using menus (menus method) Selection Rules: If the text to be moved is one or two characters long, use retype method Else, if remember shortcuts, use shortcuts method Else, use the menus method

17. Method Used Cut-and-paste-using-menus 1 2 M=1.35 P=1.10 K=0.20 3 4 5

18. GOMS: Card, Moran, and Newell (CMN-GOMS) • Subgoal invocations and method selection are predicted by the model given the task situation • In program form – analysis is general and executable • Predicts operator sequence and execution time • Based directly on the Model Human Processor

19. CMN-GOMS

20. Outline • Cognitive Modeling • Introduction to GOMS • GOMS Extensions • Modeling Specific Components • GOMS Limitations • Summary

21. Extending GOMS: Grammars • Explicitly represent knowledge a user needs to translate from goals to actions. • Task-Action-Grammar (TAG) by Payne and Green • Model of content knowledge rather than a full system to generate user performance estimation. • However, we can measure by number of rules.

22. Extending GOMS: Grammars • TAG Example for EMACS: • Task[Direction, Unit]  Symbol[Direction] + Letter[unit] • Symbol[forward]  “cntl” • Symbol[backward]  “meta” • Letter[word]  “W” • Letter[character]  “C” • Task: Move one word forward. • Task[forward, word]  Symbol[forward] + Letter[word]  “cntl” + “W”  “cntl-W”

23. Extending GOMS: Production Systems • Like grammars but models a goal stack and working memory. • Tedious to write but can be fed into a program to automatically check for completeness and accuracy. • Can predict errors and learning time behavior.

24. Extending GOMS: Production Systems • Production to see if a closing JOIN statement is needed: Rule 1: (StartUp.SeeifJoinNeeded IF ((GOAL SeeIfJoinNeeded) (NOT(NOTE SeeingIfJoinNeeded TRUE)) THEN ((Add NOTE SeeingIfJoinNeeded TRUE) (Add STEP CountTables))) Rule 2: (CountTables ((DoTask Count NumberOfTables *NumberOfTables) (Add NOTE NumberOfTables *NumberOfTables) (Delete STEP CountTables) (Add Step AddJoinNote))) Insert into Working Memory Delete from Working Memory

25. Extending GOMS: Learning • How to estimate time to learn? • One solution: Soar (UMICH) • From the FAQ: “Soar has also been used for modeling learning in many of these tasks; however, learning adds significant complexity to the structuring of the task…”

26. Extending GOMS:Natural GOMS Language (NGOMSL) • Structured natural language notation • Based directly on the Cognitive Complexity Theory (Kieras and Polson) • Allows GOMS to model working memory (WM) and setup subgoals • Unlike CMN-GOMS, provides quantitative predictions about time to learn each new piece of a task.

28. Extending GOMS: Parallel Processes • Cognitive processes are not always sequential • Clerks imprinting checks often realize an error two checks past • When typing, you often realize an error while typing the next sentence or letters

29. Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) • Predicts a substantially shorter execution time than the other models. • Allocates less time for “prepare for action” type operations. • Allow parallel processes. • Requires analyst-supplied methods. • Uses Critical Path Analysis to investigate parallel processes

30. Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) • Collect-call example1, operator hits a “collect-call” key and says “Thank you” to customer: • You can save time by repositioning the key for faster access in the sequential example, but not in the parallel example. 1Courtesy of Newman, Lemming’s TAO (Toll & Assistance Operator) study

31. Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) • Critical Path: a connected sequence that represents the greatest total time and therefore determines the overall time for a task. • Critical Path1 below is 400 + 280 + 2000 + 280 = 2.96 seconds 1Courtesy of Newman, Lemming’s TAO (Toll & Assistance Operator) study

32. GOMS Family: Summary

33. Outline • Cognitive Modeling • Introduction to GOMS • GOMS Extensions • Modeling Specific Components • GOMS Limitations • Summary

34. Modeling Specific Components • 3 general classes • Memory & Cognition • Motor movements • Perception

35. 7 steps1 of user activities involved in computer-based tasks Goals Intention Evaluation expectation Interpretation Action Specification Perception Mental Activity Physical Activity Execution 1Norman, D. (1986)

36. Goals COGNITION: Execute a mental step Choose among methods Intention Evaluation expectation MEMORY: Retrieve a unit from long term memory Action Specification Mental Activity

37. Memory & Cognition: Memory Retrieval • GOMS provides modeling of Memory Retrieval • Time to retrieve next unit of information • Moving information from long-term memory into working memory 1 @MAX(D2…D12) 2 ≈ 1350 msec

38. Memory & Cognition: Execution of a Mental Step • GOMS allows explicit representation of mental steps of a task (the “Cognitive Processor”): Retrieval of goal Select a method to achieve the goal Find the max value in a column Use the “MAX” formula Retrieval of motor movements necessary to execute the command Execution of each of the chosen commands Type the formula ≈ 70 msec

39. Memory & Cognition: Method Decision • Hick’s Law T = k log2(n+1), k ~ 150 msec n = # of choices • Time to make a decision is roughly proportional to the log of the number of choices • However, determining T is problematic • Spreadsheet task where parameters in a formula are can be indicated via numerous methods • Hick’s law predicts 200 msec • Real time is 2 seconds • → Order of magnitude difference!

40. Memory & Cognition: Method Decision • Choice is a complex task that requires many cognitive steps • Steps differ task to task

41. Goals COGNITION: Execute a mental step Choose among methods Intention Evaluation expectation MEMORY: Retrieve a unit from long term memory Action Specification Mental Activity Physical Activity MOTOR MOVEMENTS: Keystroke Point Move hands Execution

42. Motor: Key Input • Parameters of keyboard input based on • Skill of the typist • Best Typist (120 wpm): 80 msec • Worst Typist: 1200 msec • Predictability & continuity of the text to be typed • Typing random letters: 500 msec

43. Motor: Mouse Movement • Fitts’s Law is a robust predictor of mouse movement • Sometimes distance metric is not clear-cut • Nested menus

44. Motor: Applying Fitts’s Law • Fitts’s law recommends • Larger target sizes • Smaller distances to targets • Usage of corners and edges (they have “infinite” height and width) • Macintosh menus are faster than Windows/Unix style menus because they lie on the screen edge

45. Motor: Applying Fitts’s Law Target size grows as distance from cursor’s position increases Borders for shorter selection time Fittsized Menus

46. Motor: Fisheye Model • Provide local context against a global context • Focuses on screen space versus user’s attention • 3 properties • Focal point • Distance from focus, D • Level of detail, LOD • Degree of Interest • Function to determine whether to display an item or not and its size

47. Motor: Fisheye Menu • Good for browsing tasks • Allows one to present entire menu without having to use hierarchies or scrolling • Longer learning curve • http://www.cs.umd.edu/hcil/fisheyemenu/fisheyemenu-demo.shtml

48. Motor: Hand Movements • Switching between keyboard and mouse • ≈ 360 msec • Differences in times due to distance from home position on keyboard and the size of the targets • Joystick ≈ 260 msec • Arrow keys ≈ 210 msec

49. Goals COGNITION: Execute a mental step Choose among methods Intention Evaluation expectation MEMORY: Retrieve a unit from long term memory PERCEPTION: Perceive Saccade Interpretation Action Specification Perception Mental Activity Physical Activity MOTOR MOVEMENTS: Keystroke Point Move hands Execution

50. Perception • Recognition or perception • Measure the time to respond to stimuli • Responding to lights • Recognizing words • Saccade: fast movement of eye, head, etc. • Measure the time to move and take in information in each jump • Eye jerking around, scanning or moving to the next location