Uncovering the Problem-Solving Process:

1. Uncovering the Problem-Solving Process: Cued Retrospective Reporting, Eye Tracking, and Expertise Differences Tamara van Gog, Fred Paas, & Jeroen J. G. van MerriënboerI3CLEPS Workshop/Mini-conference, August 29, 2005

2. Overview Experiment: • Theory • Design • Comparison of 3 verbal methods • The 3 methods & expertise differences • Uncovering expertise-related performance differences through eye movement data • Present limitations and future research • Discussion

3. Theory Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform: • Psychological theory • Expert systems • User-system interaction, But also • Instructional design e.g., design of process-oriented worked examples

4. Theory From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000): + of concurrent reporting (“think aloud”): more information on actions taken + of retrospective reporting: more information on rationale for actions taken and strategies that control the process Needed: A method that combines + & + : Cued retrospective reporting based on a record of eye movements & mouse/keyboard operations?

6. Design Within-subjects, 26 participants, electrical circuits troubleshooting tasks: Seq. Condition + Tasks 1 CR 1+2 CRE 3+4 RR 5+6 CRR 7+8 2 CRE 3+4 CRR 7+8 CR 1+2 RR 5+6 3 RR 5+6 CR 1+2 CRR 7+8 CRE 3+4 4 CRR 7+8 RR 5+6 CRE 3+4 CR 1+2 CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.

7. Comparison of 3 Methods: Hypotheses 1. Concurrent reporting (CR): more ‘action’ info than RR 2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR 3. Cued retrospective reporting (CRR):-> more ‘action’ than RR-> more ‘why’, ‘how’, & ‘metacognitive’ than CR

8. Comparison of 3 Methods: Analyses Segmentation based on speech  sentences / utterances (preceded & followed by a pause) Coding scheme task-oriented main categories: ‘action’ ‘why’ ‘how’ ‘metacognitive’ 20% of protocols scored by 2 raters: kappa = .79 good; proceeded with 1 rater Analyses on nr. of codes on main categories, obtained by summing codes on subcategories

9. Comparison of 3 Methods: Results Friedman Tests with Conover (1999) comparisons CR vs RR:as hypothesized: ‘action’  CR >RRhowever: ‘why’ and ‘how’  CR > RR, and‘metacognitive’ CR = RR CRR vs RR:as hypothesized: ‘action’  CRR >RR‘why’: CRR = RR‘how’ and ‘metacognitive’: CRR > RR

10. Expertise Differences: Explorative 5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency: “highest”: higher performance, lower mental effort, lower time-on-task “lowest”: lower performance, higher mental effort, higher time-on-task • Differences in elicited information? • Differences in preferences/experiences?(open-ended debriefing questions)

11. Expertise Differences: Elicited Information Differences in elicited information? (Mann-Whitney U Tests) CR: ‘how’ and ‘metacognitive’ info: “lowest” > “highest” RR: ‘why’ info: “highest”> “lowest” ‘how’ info: “lowest” > “highest” CRR: ‘action’ and ‘metacognitive’ info: “lowest” > “highest”

12. Expertise Differences: Experience Differences in preferences/experiences? “lowest”: experience: CR  (4/5)preference: CRR > CR & RR (4/5) “highest”:no differential experiences/preferences Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”: • Time-on-task • Cue

13. Studying Expertise-Related Performance Differences: Eye Movement Data 1 Eye fixation data provide insight in the allocation of attention, and hence differ with expertise Research use: provide information about the problem-solving process at a finer grained level than verbal protocols? • (Ultimate) educational use: guiding novices’ attention? 1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology

14. Eye Movement Data: Participants & Procedure Same 5 “lowest” and 5 “highest” expertise participants Data collected in first 3 phases of the process: • Problem orientation (until pushing switch to observe circuit behavior) • Problem formulation and action decision • Action evaluation and next action decision % time spent on phase, mean fixation duration (MFD), and in 1st phase fix. related to faults

15. Task Short-circuit Only 3 Volt

16. Eye Movement Data: Results Phase 1: problem orientation (Mann-Whitney U Tests, 2-tailed, α = .10) % of time: “highest” > “lowest” MFD: “lowest” > “highest” % fixations on battery: “highest” > “lowest” Gaze switches short-circuit: “highest” > “lowest” (NB: only trend)

17. Eye Movement Data: Results Phase 2: problem formulation & action decision (Mann-Whitney U Tests) % of time: “highest” = “lowest” MFD: “highest” = “lowest” MFD First ½: “highest” > “lowest” MFD Second ½: “highest” = “lowest”

18. Eye Movement Data: Results Phase 3: action evaluation & next action decision (Mann-Whitney U Tests) % of time: “highest” > “lowest” MFD: “highest” = “lowest” MFD First ½: “highest” = “lowest” MFD Second ½: “highest” = “lowest”

19. Eye Movement Data: Results MFD over phases (Friedman + Nemenyi post-hoc): n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1

20. Limitations • CRR and fabrication? • Cue: combination of eye movements AND mouse/keyboard operations • Only quantitative analyses of protocols • Eye movement data: distinction of phases • Performance efficiency measure:very relative distinction (lowest and highest within this group of participants) • Small nr of participants in analyses related to expertise differences

21. Future Research • Qualitative differences between CRR and RR? • Cue: different effects with only eye movements OR mouse/keyboard operations? • Cue: technical optimization? • (RR/)CRR: effects of other prompts? • Further study of performance efficiency measure to distinguish expertise levels • Replications with larger N

22. Thank you for your attention! tamara.vangog@ou.nl