David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman,

The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports about Past Events David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman, K. Anders Ericsson, and Laura B. Hassler Expert Performance Research Team, Human Performance Laboratory Learning Systems Institute, The Florida State University, USA www.lsi.fsu.edu/hpl

Using Conventional Interviews to Provide Insights into Cognition during Performance • Conventional interview - common method of obtaining insights into cognitive processes mediating differentially skilled performance • For example, an interview conducted after a military mission to investigate the nature of a decision making during that mission

Problems with Interviews • Key problem with interviews and other self-report measures - tendency by respondents to interpret, explain, summarize thoughts (Nisbett & Wilson, 1977).

Think Aloud Technique - Ericsson and Simon (1993) • A task-concurrent cognitive process-tracing method – used in domains ranging from chess to medical practice (Ericsson & Lehmann, 1996). • Performers think out aloud their thoughts, as if alone, while performing a task • Perform TA exercise

Think aloud technique • Example think aloud: Think aloud while identifying the 4th letter in the alphabet after the letter P. Reports might look like “There’s the P, then there’s err Q, then R, S, and then T, it’s T”. • Thought to elicit verbal reports that reflects contents of WM during task execution • Reduces interpretations, explanations, and summaries of thoughts • Not the performer’s interpreted account of their thoughts but a more direct reflection of their actual thoughts

Constraints on Applicability of Think Aloud Method • Physical constraints (e.g., from rapid respiration) • Environmental constraints (e.g., task is undertaken in water) • Event danger/importance (i.e., unwillingness to verbalize during critical events) • In some domains, neither feasible nor desirable to apply think aloud method

Solution: Immediate Retrospective Report Method • After completing a task, the participant reports immediately on the thoughts experienced during the task • Perform TA and IRR exercise

Immediate retrospective report • Example alphabet task IRR: Experimenter: What was the first thought you remember having while coming up with the answer, and the next, and the next, and so on? Participant: “I saw a P, and then there was a Q, R, S, and finally a T, it was T”.

Limits on IRR Applicability • Limited to those situations in which the researcher can apply the method immediately following task completion • To return to the example of examining mission decision making, access to military personnel might not be possible for some time after the mission

Solution: Delayed Retrospective Report • Our research group has been developing a delayed retrospective report (DRR) method • DRR is identical in essence to the IRR but comprises many more directions for the participant that support the process of recalling thoughts experienced during the event • Incremental approach • participant directed to provide a brief overall description of the event • a timeline around a critical event established that includes clearly recalled “event markers” • participant directed to recall thoughts and actions during each event marker, staring with first, then second, third, and so on • Crucially, like the IRR method, the DRR method directs the participant to: - only report those thoughts and actions that can be recalled with certainty - avoid interpreting, explaining, or summarizing thoughts or actions

Early Attempts to Use DRR with Police–Recall of Thoughts While Performing Under Stress Study Step 1: Initial overall description of event obtained and establishment of sub-events and their timeline • I was dispatched to an attempted armed robbery call. The robbery was attempted in a man’s yard. Two suspects actually tried to rob an individual who…was armed. There was a stand off, no shots were fired, and the suspects fled. • I chose a route that I thought would most likely bring me into contact with the suspect vehicle. I…meet the suspect vehicle on the road, turned around on it, initiated my overhead lights. The suspect vehicle yielded. I exited my vehicle with my shotgun, conducted a felony [vehicle] stop [procedure], and took the driver into custody.

Step One: The Timeline Sub-event markers Car seen Car stopped Radio call

Step 2: Identifying Most Stressful Part of Event • The most stressful part of the whole encounter was from the period that I realized that he was going to yield [his vehicle] to me and the point that I actually exited my vehicle and had my shotgun trained on him • I was at my most vulnerable from the time that he stopped and I stopped behind him because the suspect had an advantage then in that he’s already made up his mind what he’s gonna do and if he decides to exit his vehicle quickly with a handgun, he will probably be able to get shots off at me before I can either respond or extract myself from the situation

Delayed Retrospection of Thoughts and Actions using Timeline Section of transcript: • I: So what was the first thought you remember having as, as the vehicle started to yield? • PO: There’s only one guy in the car. • I: What was your next thought after that?.... • PO: I do absolutely recall thinking “be careful, don’t let the [shotgun] sling get tangled up on say the gear shift or the radio”…and if it does get tangled up…I need to just drop it and immediately draw my handgun and engage him with that. • I: What was the next thought you remember? • PO: I’m, I’m thinking light him up [with the shotgun flashlight] and then go through a felony stop ritual.

Attempts to Validate DRR • Current research in our laboratory with police officers involves studying the relative validity and reliability of the methods described here

Some Context - http://www.wnbc.com/news/4706691/detail.html July 11 2005 • LOS ANGELES – Police shot and killed a man Sunday night when he emerged from his home carrying a young girl, about 17 months old, following an hours-long standoff and opened fire, wounding one officer. The girl was also killed in the gunfire. • [LAPD Chief McDonnell] said. "The officers are taking it very hard," McDonnell said. "Anytime you have a baby killed, it takes its toll."

Police Simulation in Human Performance Laboratory Expert Police Officer “Truman”

DRR Validation – Police Study 12 expert (SWAT, TAC) and 12 novice (cadet) police officers (but I’m not going to report here on expert-novice differences) Initial training in think aloud and IRR techniques with established exercises In a simulator, officers equipped with modified gun, exposed to 20 film clips depicting representative and stressful police scenarios, requiring shoot/no shoot decision Video captures behavior of officer – handles gun, unholsters gun, points and aims gun relative to events occurring on film – all verbal and non-verbal commands For each shot, computer captures initial trigger squeeze time point, full trigger depression time point, and gun shot placement relative to events occurring on screen IRR captures thoughts experienced during scenario

Police Scenario Simulation,Human Performance Laboratory • Plan of equipment Mirror Officer stands here Projector Video of whole body behavior Video record of trigger behavior

Procedure DRR is then performed about thoughts and actions undertaken during two (a priori determined) stressful scenarios, indicated with*

Procedure – triangular comparisons • Compare reports from IRR obtained immediately after certain scenarios with events in the film and recorded response behaviors of officer during those scenarios • Compare IRR reports with those obtained from DRR undertaken some time after all scenarios are completed • Compare DRR reports with events in the film and recorded response behaviors of officer during certain scenarios

Towards the “Damn Konfident” • In terms of the reliability of knowledge continuum proposed by Rainer Martens, the DRR method, when compared to a conventional interview method, should help us move away from feeling like we “Don’t Know” and towards feeling “Damn Konfident” about the findings we obtain from verbal report methods (Martens, 1987). Don’t Know Damn Konfident

Acknowledgements • US Office of Naval Research, Arlington, VA • The Florida State University Center for Research and Creativity • Judd Butler, Stavroula Perdikogiannis, Jason Ramratten, Mike Tuffiash, Ning Zhuang • Pat Thomas Law Enforcement Academy, Quincy, FL

David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman,