Chapter 5

1. Chapter 5 Human Error? No, Bad Design

2. Today’s Topics • Humans, systems, and error • More on data gathering • Designing for communication and coordination

3. Part 1: Humans, Systems and Error • Root cause analysis for error • Why did Air Force stop at human error? • Do not view the system being designed as one combining human and computer/machine actors (e.g. sociotechnical or cyber-human system) • The five whys method • Humans are good at creative and adaptive behavior. The more mechanistic the activity is required to be, … • Deliberate Violations • Occur in all types of contexts (e.g. driving, officework, security, …) • Sometimes because official procedures are developed for legal or accountability reasons • In many situations, work cannot get done without exceptions

4. Errors: Slips and Mistakes • Slips: the goal is correct but the actions not done correctly • Action based – inaccurate performance of planned action • Memory lapse – forgetting to take a planned action • Mistakes: the goal or plan is wrong • Rule based – wrong plan due to poor planning (“wrong rule”) • Knowledge based – wrong plan due to inaccurate/incomplete knowledge • Memory lapse – forgetting at stage of goals, plans, or evaluation • Examples from elevator • Pressing button below correct one along with correct one • Pressing the 1st floor button when going to the lobby

5. Slips • Slips are more common for experts than novices • Result of more aspects of a task being handled unconsciously • Capture slips • When similar action sequences get confused • Generally a more familiar or recent action sequence replaces another • Example might be finishing a password with more common suffix • More common for experts • Description-similarity slips • Performing an action with a similar item to the one intended • Designers should reduce similar items and reduce number to choose from • Mode errors • Occurs when design has modes where the same user action results in different system actions • Designers should try to avoid modes and make the mode visible when they are necessary • Memory lapse slips • Forgetting where one is in procedure (repeating or not performing steps) • Often caused by interruptions • Designers can show steps and indicate where one is or can create forcing functions enforcing prerequeisits

6. Mistakes • Rule-based mistakes • Situation is misinterpreted so wrong rule is selected • Correct rule is invoked, but it is faulty or did not account for current situation • Correct rule is invoked, but the outcome is incorrectly evaluated • Knowledge-based mistakes • Occurs in situations requiring more general problem solving • Can be due to a mistaken mental model • Designers need to provide clear conceptual models • Systems need to support users when they encounter previously unknown situations • Cooperative problem solving system • Memory-lapse mistakes • Occur when user forgets previously gathered information about situation, or forgets the plan altogether

7. Social Issues and Error • Social, time, and economic pressure lead to bad decisions • Social pressure also can lead to errors not being reported • Adding more responsible people can cause less attention to tasks • Each may assume others are being more responsible • Need to encourage reporting of errors to fix their causes • Jidoka and Poka-Yoke • NASA aviation safety reporting system

8. Detecting Errors is Non-Trivial • Errors are generally preceded by a number of uncommon events that each by itself is not strongly indicative of error • False alarm • Response to correct action interpreted as failure • May occur • When there are conflicting indicators • Due to different interpretations of plan/task • Garden Path • Incorrect action not discovered until later • Losing context of original problem • Machine can interpret action as correct for some alternative path • Users can assume they know the process without machine • Trivial breaches of understanding can become “fatal”

9. Designing for Error • Warnings • Need to be clear from one another and coordinated with one another • Need to gain needed attention without being obnoxious/disabilitating • Computer systems can often add • Different visual cues and locations for different actions/features • Sensibility checks on user actions • Undo and multi-stage undo • Checklists • Common in aviation but not other industries • Design of checklists is still a difficult task • More generally, processes can focus on creating resilience • The Swiss cheese model to minimize likelihood • Notifications when features are lining up for error

10. Part 2: More on Data Gathering • Interviews • Questionnaires • Observation • Choosing and combining techniques

11. Four key issues • Setting goals • Decide how to analyze data once collected • Relationship with participants • Clear and professional • Informed consent when appropriate • Triangulation • Use more than one approach • Pilot studies • Small trial of main study

12. Data recording • Notes, audio, video, photographs • Notes plus photographs • Audio plus photographs • Video

13. Interviews • Unstructured - are not directed by a script. Rich but not replicable. • Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness. • Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.

14. Interview questions • Two types: • ‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’ • ‘open questions’ do not have a predetermined format • Closed questions are easier to analyze • Avoid: • Long questions • Compound sentences - split them into two • Jargon and language that the interviewee may not understand • Leading questions that make assumptions e.g., why do you like …? • Unconscious biases e.g., gender stereotypes

15. Running the interview • Introduction– introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present any informed consent form. • Warm-up– make first questions easy and non-threatening. • Main body – present questions in alogicalorder • A cool-off period – includea few easy questions to defuse tension at the end • Closure – thank interviewee, signal the end, e.g, switch recorder off.

16. Enriching the interview process • Props - devices for prompting interviewee, e.g., a prototype, scenario

17. Questionnaires • Questions can be closed or open • Closed questions are easier to analyze, and may be done by computer • Can be administered to large populations • Paper, email and the web used for dissemination • Sampling can be a problem when the size of a population is unknown as is common online

18. Questionnaire design • The impact of a question can be influenced by question order. • Do you need different versions of the questionnaire for different populations? • Provide clear instructions on how to complete the questionnaire. • Strike a balance between using white space and keeping the questionnaire compact. • Decide on whether phrases will all be positive, all negative or mixed.

19. Question and response format • ‘Yes’ and ‘No’ checkboxes • Checkboxes that offer many options • Rating scales • Likert scales • semantic scales • 3, 5, 7 or more points? • Open-ended responses

20. Encouraging a good response • Make sure purpose of study is clear • Promise anonymity • Ensure questionnaire is well designed • Offer a short version for those who do not have time to complete a long questionnaire • If mailed, include a stamped addressed envelope • Follow-up with emails, phone calls, letters • Provide an incentive • 40% response rate is high, 20% is often acceptable

21. Advantages of online questionnaires • Responses are usually received quickly • No copying and postage costs • Data can be collected in database for analysis • Time required for data analysis is reduced • Errors can be corrected easily

22. Problems with online questionnaires • Sampling is problematic if population size is unknown • Preventing individuals from responding more than once • Individuals have also been known to change questions in email questionnaires

23. Observation • Direct observation in the field • Structuring frameworks • Degree of participation (insider or outsider) • Ethnography • Direct observation in controlled environments • Indirect observation: tracking users’ activities • Diaries • Interaction logging

24. Structuring frameworks to guide observation • - The person. Who? - The place. Where?- The thing. What? • The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?

25. Ethnography (1) • Ethnography is a philosophy with a set of techniques that include participant observation and interviews • Debate about differences between participant observation and ethnography • Ethnographers immerse themselves in the culture that they study • A researcher’s degree of participation can vary along a scale from ‘outside’ to ‘inside’ • Analyzing video and data logs can be time-consuming • Collections of comments, incidents, and artifacts are made

26. Ethnography (2) • Co-operation of people being observed is required • Informants are useful • Data analysis is continuous • Interpretivist technique • Questions get refined as understanding grows • Reports usually contain examples

27. Other Observational Methods • Direct observation in a controlled setting • Think-aloud technique • Indirect observation • Diaries • Interaction logs

28. Choosing and combining techniques • Depends on • The focus of the study • The participants involved • The nature of the technique • The resources available

29. Data Gathering Summary • Three main data gathering methods: interviews, questionnaires, observation • Four key issues of data gathering: goals, triangulation, participant relationship, pilot • Interviews may be structured, semi-structured or unstructured • Questionnaires may be on paper, online or telephone • Observation may be direct or indirect, in the field or in controlled setting • Techniques can be combined depending on study focus, participants, nature of technique and available resources

30. Part 3: Designing for Conversation and Coordination • Conversational mechanisms • Coordination mechanisms • Awareness mechanisms • Examples of technologies designed to extend how people • talk and socialise • work together • play and learn together

31. Conversational mechanisms • Various mechanisms and ‘rules’ are followed when holding a conversation, e.g. mutual greetings A: Hi there B: Hi! C: Hi A: All right? C: Good, how’s it going? A: Fine, how are you? C: OK B: So-so. How’s life treating you?

32. Conversational rules • Sacks et al. (1978) work on conversation analysis describe three basic rules: Rule 1: the current speaker chooses the next speaker by asking an opinion, question, or request Rule 2: another person decides to start speaking Rule 3: the current speaker continues talking

33. Conversational rules • Turn-taking used to coordinate conversation • A: Shall we meet at 8? • B: Um, can we meet a bit later? • A: Shall we meet at 8? • B: Wow, look at him? • A: Yes what a funny hairdo! • B: Um, can we meet a bit later? • Back channeling to signal to continue and following • Uh-uh, umm, ahh

34. More conversational rules • Farewell rituals • Bye then, see you, yer bye, see you later…. • Implicit and explicit cues • e.g., looking at watch, fidgeting with coat and bags • explicitly saying “Oh dear, must go, look at the time, I’m late…”

35. Breakdowns in conversation • When someone says something that is misunderstood: • Speaker will repeat with emphasis: A: “this one?” B: “no, I meant that one!” • Also use tokens: Eh? Quoi? Huh? What?

36. What happens in technology-mediated conversations? • Do same conversational rules apply? • Are there more breakdowns? • How do people repair them for: • Phone? • Email? • IM? • Texting?

37. Synchronous computer-mediated communication • Conversations are supported in real-time through voice and/or typing • Examples include video conferencing, VOIP, MUDs and chat • Benefits include: • Not having to physically face people may increase shy people’s confidence • Allows people to keep abreast of the goings-on in an organization without having to move from their office • Problems: • Difficult to establish eye contact with images of others • People can behave badly when behind the mask of an avatar

38. Asynchronous computer-mediated communication • Communication takes place remotely at different times • Email, newsgroups, texting • Benefits include: • Read any place any time • Flexible as to how to deal with it • Can make saying things easier • Problems include: • FLAMING!!! • Message overload • False expectations as to when people will reply

39. Coordination mechanisms • When a group of people act or interact together they need to coordinate themselves • e.g., playing football, navigating a ship • They use: • verbal and non-verbal communication • schedules, rules, and conventions • shared external representations

40. Verbal and non-verbal communication • Talk is central • Non-verbal also used to emphasize and as substitute • e.g., nods, shakes, winks, glances, gestures and hand-raising • Formal meetings • explicit structures such as agendas, memos, and minutes are employed to coordinate the activity

41. Schedules, rules and conventions • Schedules used to organize regular activities in large organizations • Formal rules, like the writing of monthly reports enable organizations to maintain order and keep track • Conventions, like keeping quiet in a library, are a form of courtesy to others

42. Shared external representations • Common method used to coordinate collaborative activities, • e.g., checklists, tables, to-do lists • They can provide external information on: • who is working on what • when it is being worked on • where it is being worked on • when a piece of work is supposed to be finished • whom it goes to next

43. Collaborative technologies to support coordination • There are a variety of software tools designed to support scheduling, planning and coordinating • e.g., group calendars, electronic schedulers, project management tools, and workflow tools • Need to get balance between human and system control • too much system control and the users will rebel • too little control and the system breaks down

44. Awareness mechanisms • Involves knowing who is around, what is happening, and who is talking with whom • Peripheral awareness • Keeping an eye on things happening in the periphery of vision • Overhearing and overseeing - allows tracking of what others are doing without explicit cues

45. Lo tech awareness mechanism

46. Designing technologies to support awareness • Provide awareness of others who are in different locations • Early example was media spaces “extend the world of desks, chairs, walls and ceilings” (Harrison et al, 1997) • Examples: Clearboard and Portholes

47. Portholes (Xerox PARC) Regularly updated digitized images of people in their offices appeared on everyone’s desktop machines throughout day and night

48. Notification systems • Users notify others as opposed to being constantly monitored • Provide information about shared objects and progress of collaborative tasks • examples: Tickertape, Babble

49. Elvin • Elvin is a distributed awareness system that provides a range of client services (Segall and Arnold, 1997) • It includes Tickertape, one of the first lightweight messaging systems

50. Conversation and Coordination Key Points • Social mechanisms, like turn-taking, conventions, etc., enable us to collaborate and coordinate our activities • Keeping aware of what others are doing and letting others know what you are doing are important aspects of collaborative working and socialising • Many collaborative technologies systems have been built to support collaboration