Performance Assessment in Simulation-Based Team Training: Principles from the Science That Matter

1. Performance Assessment in Simulation-Based Team Training: Principles from the Science That Matter Marissa L. Shuffler Eduardo Salas University of Central Florida February 11, 2010 Performance Testing Council Summit, Orlando FL

2. Overview • What do we know about teams and team performance assessment? • What is simulation-based team training (SBT)? • How do we assess performance in team SBT? • Best practices for SBT performance assessment

3. Defining Characteristics of Teams • Two or more individuals • Multiple information sources • Meaningful task interdependencies • Coordination among members • Common, valued goals • Intensive communication • Adaptive mechanisms • Hierarchical structure • Specialized member roles and responsibilities • Task-relevant knowledge

4. Driving Question: How Do We Turn a Team of Experts into an Expert Team?

5. The Conditions and Processes of Team Performance

6. Team Performance Assessment Systems • Requirements of a team performance measurement system (Cannon-Bowers & Salas, 1997): • Must be theoretically based • Models of team performance • Cognition, behavior and affective factors • Must consider multiple levels of measurement • Individual/Team • Process/Outcome

7. Team Performance Assessment Systems • Must be able to describe, evaluate and diagnose performance • Moment to moment changes • Determine causes • Help with remediation • Must support ease of use • Easily adaptable • Useability • Must have validity & good psychometrics • Face • External/Internal • Must facilitate Debriefing/AAR • Feedback

8. What We Know about Team Performance Assessment • We have a good theoretical basis… • 138 models! (Salas, et. al. in press) • We have an idea (not perfect) of what to measure… • Team Type Taxonomies (Sundstrom et al, 2000) • Team Competency Taxonomies (Cannon-Bowers et al, 1995) • Shared Mental Models (Cannon-Bowers et al, 1993) • There are some tools available that work (not perfect)… • Qualitative/Quantitative • Outcome/Process • Taskwork/Teamwork • Used in Complex Simulations

9. Defining Simulation-Based Team Training • What is Simulation? • “Realistic synthetic work/learning environments that can be used for training” (Andrews, 2005) • What is Simulation-Based Training? • “A training strategy that uses several aspects of scientific, theory-based training, incorporating information, demonstration, and practice-based methods” (Salas, Priest, Wilson, & Burke, 2006)

10. Defining Simulation-Based Team Training • What is Team Training? • “A set of tools and methods that, in combination with required competencies and training objectives, form an instructional strategy” (Salas & Cannon-Bowers, 2000)

11. Defining Simulation-Based Team Training • Used to accelerate expertise in team members… • Explicit links are developed and maintained among trainingrequirements, exercise design, observation, and feedback…

12. Defining Simulation-Based Team Training • SBT provides trainees opportunities to demonstrate mastery of learned teamwork skills… • SBT provides opportunities to turn a team of experts into an expert team…

13. SBT Works Best With… • …Clear & Precise Learning Outcomes • …Carefully Crafted Scenarios • …Well Designed Performance Measures • …Robust Observation Protocols

14. Why Measure Performance in Team SBT? • Quality measurement practices guide learning and corrective feedback • Measurement can also ensure learners possess the requisite competencies for effective team performance • Practice opportunities in SBT can be used to detect deficiencies in both individual and team competencies

15. Needed Links in Simulation-Based Team Training Tasks & KSAs LearningOutcome Scenarios &Events Measures PerformanceHistory Feedback

16. Challenges to Assessing Performance in Team SBT • Measuring teamwork is complex! • Multiple people and multiple skills means multiple measures are required • Measuring the critical aspects of performance instead of measuring what is easy • A single snapshot of performance may not be completely representative of long term performance • It is difficult to remove observers or raters from the assessment process • Raters are not always reliable!

17. How to Measure Team Performance in SBT • Two important decisions from a training perspective to guide assessment development: • What meaningful feedback should be given to the team and each individual member? • What further training is required by team or individual members? • Depending on the answers to these questions, certain methods may be more or less useful

18. Methods of Team SBT Assessment • Event based measurement • Generates behavioral checklists that are linked to scenario events and KSAs being trained • Behaviorally Anchored Rating Scales (BARS) • Provides brief descriptions of behaviors as anchors associated with each particular rating

19. Methods of Team SBT Assessment • Behavioral Observation Scales (BOS) • Generally uses a Likert type scale to rate the frequency of certain team processes • Self-Report Measures • Questionnaires administered to each team member individually

20. Best Practices for Team SBT Performance Assessment • Drawing from what we know regarding SBT and team performance assessment, several guidelines have emerged • These guidelines are theoretically, empirically, and practically based • Derived from literature reviews, observations, and interviews (Salas, Rosen, Weaver, Held, & Weissmuller, 2009)

21. Best Practices for Team SBT Performance Assessment • Best Practice 1: Ground measures in theory • Use theory to answer questions about what to measure • Capture aspects of Input Process Output models of team performance

22. Best Practices for Team SBT Performance Assessment • Best Practice 2: Design measures to meet specific learning outcomes • Clearly articulate the purpose of measurement at the beginning of the development process • Design measurement systems to capture information necessary for making decisions about learning outcomes

23. Best Practices for Team SBT Performance Assessment • Best Practice 3: Capture competencies • Explicitly link performance measures to individual and team level competencies targeted for training • Best Practice 4: Measure multiple levels of performance • Performance measures should be sensitive to differences in individual and team performance

24. Best Practices for Team SBT Performance Assessment • Best Practice 5: Link measures to scenario events • Link performance measures to opportunities to perform in the simulation • Affords greater structure & standardization for observers • Best Practice 6: Focus on observable behaviors • Decreases “drift” of observers ratings over time • Provides a fine level of granularity that can help in providing feedback

25. Best Practices for Team SBT Performance Assessment • Best Practice 7: Incorporate multiple measures from different sources • Different sources provide unique information • Must develop a plan for integrating information • Best Practice 8: Capture performance processes in addition to outcomes • Should provide information about HOW teams reached their outcomes so that poor process can be corrected

26. Best Practices for Team SBT Performance Assessment • Best Practice 9: Create diagnostic power • Provide information about the causes of effective and ineffective performance • Best Practice 10: Train observers and structure observation protocols • Establish training to ensure observers are accurately and reliably rating performance • Provide structured protocols to ease the information burden of observation

27. Concluding Remarks • Team performance assessment is not easy, but there is a wealth of research and theory that exists to support assessment tool development…so… Use the science!

28. Thank You! Questions?