Thinking-Aloud & Protocol Analysis

1. Unit 3.7.1 Macro-Ergonomics: Evaluating human work Thinking-Aloud &Protocol Analysis Margje van de Wiel Department of Experimental Psychology Universiteit Maastricht

2. Overview • Research on cognitive processes underlying cognitive task performance: • Problem solving • Knowledge representation • Focus on the think-aloud method • Collecting think-aloud protocols • Analyzing think-aloud protocols

3. Applications of cognitive psy-chological research methods • Fundamental psychological research • Explorative • Testing theoretical hypotheses • Compare tasks or groups • Cognitive task analysis • Building knowledge based systems • Usability research

4. Usability research • How easy is it for people to set the time and alarm on a clock without a manual? • How easy is it for them with the manual? • Do people understand the manual? • Do they use the manual? • Video, computer program, web site etc. • Compare conditions or groups

5. Problem solving Tower of Hanoi The objective of this game is to move all of the rings to the right most peg.You may only move one ring at a time; you must never allow a large ring to rest on a smaller ring.

6. Problem space (Newell & Simon, 1972) Initial state Inter- mediate states Subgoal state Inter- mediate states Inter- mediate states Subgoal state Inter- mediate states Goal state

7. Research on problem solving • Product evaluation • Observation and registration of behavior (real-life, video, log files) But Gives no direct information on cognitive processes

8. Research on problem solving • Introspection • Direct observation of mind But Participants may give their interpretation of their cognitive processes when they prefer during task performance

9. Research on problem solving • Retrospection But Participants may give their interpretation of their cognitive processes after task performance

10. Research on problem solving • Free recall • Stimulated recall But Participants give their memories of the thoughts they had during problem solving after task performance

11. Instructions • Retrospection • How did you proceed in solving the problem? • What did you think during solving the problem? • Stimulated / Free recall • Report everything you can remember about your thoughts during solving the problem

12. Research on problem solving • Questioning and prompting But Participants may give their interpretation of their cognitive processes when prompted so that natural task performance may be disturbed

13. Research on problem solving • Dialogues in explanation and collaboration But The cognitive processes might be different than in natural task performance

14. Research on problem solving • Thinking aloud • Saying everything that comes to mind Advantages: • Concurrent verbalization • Close to natural task performance • Discourages interpretation and explanation

15. Theoretical framework / verbalization theory (Ericsson & Simon, 1993) • the information processing model LTM STM Task performance Input info Verbal report

16. Theoretical framework STM Task performance Verbal encoding Type of task - modality - verbalizability - instruction Vocalization Verbal report

17. The type of problem and the role of knowledge Well-defined problems Ill-defined problems Weak problem solving methods Strong problem solving methods Knowledge lean task Knowledge rich task Novices Experts

18. Focus on knowledge representation The nature and organization of knowledge is central in problem solving and not the strategies used (e.g. Chi, 1997) • The representation of a problem determines the problem solving process

19. Research on knowledge representation • Same methods as for investigating cognitive processes, but analysis of the knowledge used • Free recall • Report everything you remember about ... • Reflection of the representation of (prior) knowledge, a text or a problem in the mind

20. The think-aloud method • Collecting think-aloud protocols • Transcribing the verbal protocols • Verbal reports as data • Segmenting the transcribed protocols • Analyzing the transcribed protocols

21. Research questions • Gaining insight in cognitive processes and knowledge structures • Explorative • Testing hypotheses • Comparing types of problems • Comparing groups

22. Collecting think-aloud protocols What do you need? • Experimenter • Participant • Problem • Tape-recorder and tape • Instructions

23. Instructions • Make participant feel comfortable • Explain the purpose of the research • Explain the task • Explain thinking aloud • Emphasize that it is no test situation • Emphasize that the data are confidential • Practice problem(s) and talking out loud • Give feedback

24. Transcribing the protocols • Type out the protocols as literally as possible, including • Natural punctuation • Eh, ehs • Thinking pauses; with dots or even better to measure length • Indicate when something is said mumbling so you don’t understand well • Indicate interventions of experimenter

25. Example Tower of Hanoi (Anzai & Simon, 1979) I’m not sure, but first I’ll take 1 from A and place it on B. And I’ll take 2 from A and place it on C. And then, I take 1 from B and place it n C. Experimenter: If you can, tell me where you place it there? Because there was no place else to go, I had to place 1 from B and C. Then, next, I placed 3 from A to B. …..

26. Segmenting the protocol • Use natural pauses indicated by punctuation • Indicate each segment by a line number • No interpretation

27. Example Tower of Hanoi (Anzai & Simon, 1979) 1. I’m not sure, 2. but first I’ll take 1 from A and place it on B. 3. And I’ll take 2 from A and place it on C. 4. And then, I take 1 from B and place it on C. Experimenter: If you can, tell me where you place it there? 5. Because there was no place else to go,.. …..

28. Analyzing verbal protocols (Transcribed) and segmented verbal protocols RAW DATA Interpretation DATA

29. Interpretation • Qualitative analysis • Analyzing qualitative data in an objective and quantifiable way • Compare experimental conditions • Groups • Tasks • Instructions

30. Research question --> qualitative analysis • What comprehension strategies do skilled readers use in reading a text? • Thinking aloud while reading a text • Analysis: What strategies are used? • Exploratory: describe the strategies used by the participants

31. Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) • What strategies do skilled readers use in dealing with degraded text? • Are these comparable to comprehension strategies used in reading normal text? • Thinking aloud while reading text with 3 types of instruction: • Just reading • + ‘Adults skim this type of material first’ • Skim text for 60 sec and report the topic

32. Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) • Analysis: what strategies are used? • Normal comprehension strategies • Rereading • Reading ahead • Summing up • Other comprehension strategies • (Letter generating strategy) • (Determining word type)

33. Analyzing verbal protocols 1 Psychological theory Cognitive task analysis Hypotheses Coding scheme Explore Test Verbal protocols

34. Analyzing verbal protocols 1 Hypotheses Coding scheme Verbal protocols Problem solving <--> Knowledge representation processes (E&S, 93) (Chi, 97)

35. Cognitive task analysis • What knowledge is available in the problem? • What knowledge is necessary for problem solving? • What steps need to be taken to solve the problem? • What is the most efficient problem solving process? • What strategies can be used?

36. Coding scheme • Operationalizing hypotheses in coding categories • Clearly defining coding categories and providing examples from pilot protocols • Categories may differ in level of detail • Segments • Aggregations of segments

37. Analyzing verbal protocols 2 • Code the verbal protocols • Determine interrater reliability (% of correspondences, kappa, correlation) • Depict the results • Seek and interpret patterns in results Eventually • Repeat process

38. Example medical diagnosis • Clinical case

39. Example of a case A 70-year old female is admitted into hospital because of increasing shortness of breath. She has been very tired lately and tolerates her food badly. Sometimes she has chest pain, especially after dinner. Et cetera

40. Example medical diagnosis • Clinical case • Thinking aloud while diagnosing the case • Problem solving process • The nature of the knowledge used

41. Problem solving process Data examination Data exploration Data explanation Hypothesis generation Hypothesis evaluation Discrepancy processing Meta-reasoning Summarization Nature of knowledge used Data interpretation Clinical reasoning Biomedical reasoning Hypotheses (C or B) (Repetitions) Plans for actions Metacognitive remarks Analysis of think-aloud protocol

42. Example medical diagnosis • Clinical case • Thinking aloud while diagnosing the case • Problem solving process • The nature of the knowledge used • Free recall • Case representation

43. Analysis of free recall protocols Proposition analysis • Number of correct propositions • Number of inferences (summarizing two or more propositions)

44. Proposition analysis Proposition = single idea unit Concept Concept Qualifier Causal Conditional Attribute Specification Is a

45. Propositions in the case Female - 70-year old (attribute) - is admitted into hospital (attribute) - (because of) shortness of breath (attribute) - increasing (specification) - very tired (attribute) - lately (specification) - tolerates her food badly (attribute)

46. Other literature • Chi, M.T.H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), 271-315. SGT 3179 • Ericcsson, K. A. & Simon, H. A. (1993). Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press (ch 6 & 7). SL BF 180

47. Other literature • Reitman Olson, J. & Biolsi, K. J. (1991). Techniques for representing expert knowledge. In: K. A. Ericsson & J. Smith (Eds.) Toward a general theory of expertise: Prospects and Limits (pp. 240-285). Cambridge: Cambridge University Press. ME EAM 374 / Margje