Toward quantifying the effect of prior training on task performance MURI Annual Review

1. Toward quantifying the effect of prior training on task performance MURI Annual Review September 26-27, 2006 Bill Raymond

2. Overview • Project goal:Quantify the effects on performance of different training methods for complex military tasks. • Feature decomposition: • Task type • Training method • Performance assessment (context & measures) • Training principles • Planning matrix: • Capture where we know of, and can quantify in terms of performance measures, effects of training method and performance context on task components. • Quantify principles: • Derive performance functions for points in the feature space using empirical data from laboratory tasks. • Generalize performance functions for implementation in IMPRINT modeling tool to simulate training effects on task performance.

3. Decomposition issues • Constraints on decompositions Features must relate to experimental designs • Must be able to describe all experimental tasks. • Task, training, and performance context features can be no finer than experimental manipulations. Features may be different for research and IMPRINT • Can’t control training in the real world as carefully as in the laboratory • Not all experimental results will be major effects. • IMPRINT task categories are already defined. • Planning features should converge to final IMPRINT features, diverging from research features

4. Planning matrix issues • What will the matrix construction provide? • Current and planned research coverage of space • May be used by us or others for future planning • Approximation of final IMPRINT training features • Initial step in determining the generality of performance functions in the space

5. Starting point: Analyzing training and performance • Training variables - during skill learning: • How was the skill taught? • What kind of practice did learners get? • How did practice relate to the way the skill will be used? • Performance context variables - at skill use: • How does expected performance relate to training? • How long has it been since training? • Did learners get refresher training? Pedagogy } Practice } Performance

6. Data entry Data entry Data entry Data entry Data entry Data entry Data entry Data entry Data entry Task, training, and performance matrix IMPRINT task taxons

7. Pedagogy parameters • Method • Instruction (=default) • Demonstration • Simulation • Discovery • Modeling/mimicking • Immersion • Learning location (local = default, remote/distance) • Discussion/Question and answer (no = default, yes) • Individualization (no = default, yes)

8. Data entry (Instruction) Data entry (Instruction) Inst/Discovery Classification Data entry (Instruction) Task by Pedagogy parameters IMPRINT task taxons

9. Practice parameters • Scheduling of trials and sessions • Number • Spacing (massed = default, spaced, expanding/contracting) • Distribution (mixed = default, blocked) • Scope of practiced task (partial, whole = default, whole + supplemental) • Depth of processing(no = default, yes) • Processing mediation(no = default, yes) • Stimulus–response compatibility(yes = default, no) • Time pressure(no = default, yes) • Feedback - presence (no = default, all trials, periodic) • Context of practice • Distractor (no = default, yes) • Secondary activity (no = default, yes)

10. Task by practice Data entry Data entry IMPRINT task taxons Data entry

11. Performance context parameters • Transfer • New context (relative to training) • New task (relative to training) • Delay interval (default = none, time period) • Refresher training (default = no, schedule)

12. Task by performance parameters Data entry Data entry IMPRINT task taxons Data entry Data entry

13. Quantifying training principles • Data Entry used as an example • Consider two principles • Practice  Learning (Power law of practice) • Skill practice - no item repetition • Specific learning - item repetition • Prolonged work  Diminished performance • Quantify effects for each taxon • Cognitive (“Information processing”) • Physical (“Fine motor - discrete”) • …and performance context • Transfer to new items (similarity dimension) • Retention of learned skill (refresher training)

14. Skill practice: Quantifying learning • Skill practice improves performance .5 msec/item • Mean decreases 300 msec in 640 (unique) items • Where does skill practice come from? • Repetition of individual digits (and pairs of digits?) • Cognitive or physical learning? • Individual differences?

15. Skill practice: Origin or learning Pair repetition? • Subjects appear to “chunk” digits 1 & 2, digits 3 & 4 • so they may be learning something about pairs of digits

16. Skill practice: Origin of learning Pair repetition? • Effect of 2-digit chunk practice appears minimal • Skill practice is general facility at number typing

17. Skill practice: Type of learning Physical or cognitive? • Speed improvement occurs on digits 1 and 3 • Learning is cognitive

18. Skill practice: Individual differences • “Chunkers” are 15% slower than “non-chunkers” • Appears to be a strategy choice • Pedagogy - advantage for instruction over “discovery”?

19. Specific learning: Quantifying learning • Repetitious practice improves performance faster initially • Power law of practice

20. General learning functions . . . ? • Performance as a function of number of repetitions • Planned experiment

21. General learning functions New items? Old items? . . . Learning Transfer Retention • Transfer and retention • Planned experiment

22. Prolonged practice • Prolonged work results in an increase in errors • Accuracy rate decline of about 1% over 320 items • Where does the decline in accuracy originate? • Cognitive or physical fatigue?

23. Prolonged practice: Type of performance decline • Two types of errors: • Stimulus adjacency errors: 1234 1244 • Key adjacency errors: 1234  1264 • 90% of errors are of these two types • Origin of errors • Stimulus adjacency = cognitive • Key adjacency =motor phase, which could be motor output planning (cognitive) or motor execution (execution)

24. Prolonged practice: Type of performance decline • Practice results in an increase in key adjacency errors • Accuracy decline occurs during the motor phase (which may be both cognitive and physical)

25. Prolonged practice: Type of performance decline • Feedback eliminates the speed-accuracy trade-off • If feedback is cognitive, then the relevant processes in the motoric phase must be cognitive

26. Summary IMPRINT task taxons