Monday’s Agenda

1. Monday’s Agenda • Proposal review • Q&A about text contents • A couple of example test questions • Get ready for Wed lab • More on qualitative data collection methods • A few minutes on last week’s lab

2. Three characteristics of the TIGR design and development process that contributed to its success? Three characteristics of the practitioner’s spiral?

3. “In order to succeed…companies have to rid themselves of their Dilbert manifestations of make-work and arcane policies. This freedom from the inanities of corporate life attracts proponents of Agile Methodologies, and scares the begeebers…out of traditionalists. Quite frankly, the Agile approaches scare corporate bureaucrats, at least those that are happy pushing process for process sake...” http://agilemanifesto.org/principles.html

4. Cognitive Task/Work Analysis

5. Use CTA / CWA to develop or evaluate: • Theories • Models of cognitive work • Decision- and work-support software • Training systems, strategies, content • User interfaces

6. Using Interviews for Data Collection

7. “A lot of people look at a specialist like me as a technician. They come to you for a procedure. And there is no doubt that procedures are important, or that the specialized technology we have these days is vital in caring for a patient. But I believe that this technology also has taken us away from the patient’s story.” Falchuk paused. “And once you remove yourself from the patient’s story, you no longer are truly a doctor.” Groopman, J. (2007). How doctors think. Houghton-Mifflin.

8. “Osler essentially said that if you listen to the patient, he is telling you the diagnosis,” Falchuk continued. Groopman, J. (2007). How doctors think. Houghton-Mifflin.

9. Terry Gross, NPR Radio Host “Once I’ve thought through the structure then I’m free to follow wherever the interview leads me but I always have a structure to come back to.” Curran-Hackett, M. (2008). The art of the interview. Writers Digest, February, 56-59.

10. Interview Lessons Learned

11. What are two weaknesses of the think-aloud protocol? What is a cognitive probe in the Critical Decision Method (CDM)? What are two strengths of the CDM? What characteristics contribute to its effectiveness?

12. The Critical Decision Method (CDM)

13. The CDM Protocol Consent Form Collect background data. Ask interviewee (I) to choose a past difficult, stressful, or otherwise memorable event. Ask I to walk you through the event, telling you about it as he/she would tell a story. Record the timeline of the event & write down clarification questions. Review timeline, ask questions, & use probes Debrief Transcribe and ask I to review the transcript.

14. Probe Questions Aka “Cognitive Probes” Prepare these questions in advance. You may use some, none, or all. Help ensure interviewee covers the aspects of their work that you’re studying.

15. Strengths of CDM(If done correctly) • Difficult/stressful/challenging events are: • Memorable, often in detail • What matters • Takes advantage of memory storage structure • Episodic memory • Mostly empirical data; minimal introspection • Any introspecting is grounded in actual events • “What if the situation had changed…”

16. When CDM Doesn’t Work • Many past events are difficult and stressful • One doesn’t stand out as uniquely memorable • Interviewee has an agenda • Systematic comparison of groups

17. Examples of When to Use CDM What is challenging about assessing and triaging emergency phone calls? How many and what cues lead experienced aviators to recognize their situation has changed? How do fireground commanders make tactical and strategic decisions? How do nurses assess patients? What cues to they look for? How do information operations specialists make sense of intelligence data?

18. Applied Cognitive Task Analysis Task Diagram Knowledge Audit Simulation Interview Cognitive Demands Table - Militello, Hutton, & Klein

19. Task Diagram

20. From Militello & Hutton, 1992

23. Coding of Qualitative Data • Helps you find the valid patterns in your data. • Would most E-2 NFOs call back a stray F/A-18? Or just one? Or just less experienced NFOs? Or only in a certain mission type? • Adds rigor to your method. • Forces you to consider every excerpt in your interview / think-aloud protocol / etc. • Not just the excerpts that are consistent with your expectations • At least two people should code. • More than one interpretation considered • Correlation of coders’ code choices is a measure of method reliabilitiy. Can we trust your codes?

25. Observation Devorah Klein’s evaluation.

26. Observation • Goal: Capture authentic worker behavior • "fly on the wall" • Acceptance by workers • Being there with them at 3 a.m. doesn’t hurt • Workers are often suspicious and will show/tell you what they want you to see/think • You represent a possibility for change • Grover’s experience is too common • SOLUTIONS? • Pros and cons on p. 43 of text.

27. Give some structure to your observations by specifying in advance what you will record Instances of multi-tasking Sources of high workload, stress, or frustration Support for apprentices & the less experienced Distractions Procedures and tasks that don’t align with goals Are work procedures strictly defined vs. flexible? Allocation of work to workers vs. automation Decisions made and who had authority to make them Hand-offs between workers

28. Observation: Workspace Analysis Note observations relevant to your research on the map.

29. For each work area, interview workers:

31. Back Up Slides

32. This one was, they shot at us. I pulled the aircraft up and kicked the tail out so the gunner on the side could see where the tracers were coming from. Of course he opens up….the funny things about the guns on the one side is that they traverse the gun. The ejection port is right there flipping….and as he moves the gun like this…of course I am sitting in front of it and the ejection port is knocking things out and they’re hitting the back of your helmet so you’re getting. One of those will go down your back and those things are freaking hot. I mean burning hot. You’re getting pelted in the back of the head with these “dink, dink, dink”. The helicopter is extremely noisy and you can feel the guns go off more than you hear them go off. The aircraft is shaking a little bit, more than usual. And the things are smacking the back of your head as he traversing the gun…

33. This one was, they shot at us. I pulled the aircraft up and kicked the tail out so the gunner on side could see where the tracers were coming from. Of course he opens up….the funny things about the guns on the one side is that they traverse the gun. The ejection port is right there flipping….and as he moves the gun like this…of course I am sitting in front of it and the ejection port is knocking things out and they’re hitting the back of your helmet so you’re getting. One of those will go down your back and those things are freaking hot. I mean burning hot. You’re getting pelted in the back of the head with these “dink, dink, dink”. The helicopter is extremely noisy and you can feel the guns go off more than you hear them go off. The aircraft is shaking a little bit, more than usual. And the things are smacking the back of your head as he traversing the gun …

35. ???