Abstract

1. Event 1 Diagnosis Event 2 Intubation/Stabilization Event 3 Arrest End (E1)/Begin (E2) O2 ≤ 85%  Intubate End (E2)/Begin (E3) V-Fib rhythm  Resuscitate Begin (E1) Team enters room Intubation complete/ Consult call made End (E3) Sinus rhythm established O2 reaches 85% OR OR No call within 1 min. O2 below 85% for 1 min. Intubation complete/ Prompt consult call Prompt intubation Figure 1 Mission Analysis Team fully and accurately assesses patient status and all members are informed of situation Complete Skill 5 Very much skill 4 Team assesses patient status, but focuses too exclusively on certain pieces of information or does not communicate information to all members Adequate skill 3 Some skill 2 Team does not reach an overall assessment of patient status, ignores certain aspects of patient status, or fails to communicate any mission information to team members Hardly any skill 1 Designing, Developing and Evaluating Event-Based Team Simulations: Helping Medical Educators Put Theory Into Practice James A. Grand1, Rosemarie Fernandez2, Steve W.J. Kozlowski1, Georgia T. Chao3, Jason L. Huang1, Paul Curran11Michigan State University Department of Psychology, East Lansing, MI, United States 2Wayne State University School of Medicine Department of Emergency Medicine and Office of Student Programs, Detroit, MI, United States 3Michigan State University Department of Management, East Lansing, MI, United States Abstract Step 1: Scenario Design Step 3: Measurement Validation Results • Select a realistic medical scenario that teams might face relative to the purpose of the training context. • Consult hospital records or medical practitioners from various locales (i.e., urban/metropolitan, suburban, community/rural) to generate potential situations and serve as justification for a scenario’s representativeness • Identify important events in the scenario based on objective, reliable indicators. • An event is any substantive team task (e.g., resuscitation, intubation) with a clearly identifiable beginning (change in heart rhythm, lung collapse) and ending (return to normal sinus, stabilization of O2 sats) • Script the ”flow” of events and/or team responses into a standardized event-based scenario. • Scenarios can follow a simple sequential pattern (Figure 1) or a more complex, non-linear pattern depending upon the requirements of the medical tasks at hand • Use subject matter experts to evaluate the content validity of the teamwork measure. • Content validity refers to the representativeness, adequacy and relevance of the items composing a particular measure; content validity ensures that no important material is left out of the assessment and no unnecessary material is included in the assessment • Select individuals who are experts in team evaluation and research • Provide subject matter experts with a brief description of how teams are expected to interact in the scenario, relevant details about the team environment and the overall flow of team activity • Ask subject matter experts to rate the relevance/accuracy of the team process dimensions for each event (Figure 4) • Use subject matter experts to evaluate the content validity of the performance measure. • Select individuals who are experts in the area of relevant medical practice • Provide subject matter experts with a brief description of the medically relevant details of the scenario and each event, and the overall flow of medical activity • Ask subject matter experts to rate the relevance/accuracy of the performance dimensions for each event (Figure 5) • The present procedure produces a systematically developed, highly reliable EBS and accompanying measurement tools for observing and evaluating teamwork and performance. The steps described here result in the creation of: • A detailed script for an event-based medical scenario describing the event triggers and expected flow of the simulation • Measurement tools for evaluating the expected team processes and performance criteria within each event • Validation forms for assessing the adequacy of the team process and performance metrics • An inclusive system for monitoring teams during and after simulations that can be readily adapted to account for unexpected occurrences in the scenario The current effort presents a methodological procedure for constructing reliable, valid and event-based medical simulations as well as the measurement instruments needed to assess the teamwork skills and clinical expertise of medical teams. Illustrative examples are provided from a simulation designed by the authors for evaluating emergency medical teams that demonstrate the utility and practicality of the approach. The benefits and implications of the procedure are explicated, and practical advice to medical educators is offered for employing the methodology across various team training applications in the field of medicine. Introduction Beyond clinical task knowledge, teamwork and communication skills among medical professionals are critical to effective patient safety and care. Some recent reports have even suggested that between 70-90% of adverse events in patient care (e.g., preventable deaths, etc.) can be attributed to failures in communication and/or team-related functioning (Institute of Medicine, 2000; Shojania et al., 2001). Although major advances in simulation technologies have improved the assessment of medical expertise substantially, many current strategies for evaluating and teaching team skills are less well developed or understood. In response, a number of researchers have proposed using event-based simulation (EBS) methodologies to investigate team-related phenomena and performance in the domain of medical education (e.g., Rosen et al., 2008). While undoubtedly a positive step, implementing and taking advantage of EBSs can be difficult for users unfamiliar with the techniques required for proper development, and no standard procedure currently exists for designing EBSs and any accompanying evaluative tools. This study attempts to fill this void by providing a practical methodology for developing EBS training/research platforms, using examples from the authors’ work with emergency medicine teams. Example. A patient is admitted to the emergency room with complaints of respiratory distress and altered mental status (Event 1). After a short time, the patient becomes hypoxic and requires intubation (Event 2). Following intubation, the patient suddenly enters cardiac arrest and the team must resuscitate using standard ACLS protocol (Event 3). Practical Advice & Tips • Don’t expect to design the perfect simulation on your first attempt. The process will almost certainly be an iterative one as you update and revise the simulation’s events and measures to best capture the teamwork and performance criteria of interest. • A good simulation can capture a lot—but it can’t capture everything. Plan to make a series of simulations that require both unique and overlapping areas of proficiency so that you can assess participants’ skills across a variety of situations. • Evaluation is only the first step. Consider how the gathered information from the simulation can best be put to use. For novice teams, it might be most beneficial to focus on providing feedback on performance issues rather than critiquing teamwork behaviors; however, more experienced teams may find direct feedback regarding breakdowns in teamwork more useful. Figure 4 Discussion & Implications Regardless of the pace at which medical simulation technology continues to advance, accurate and valid measurement of learners’ actions and performance will remain a constant and critical need for educators to consider. Given the importance of effective team communication and clinical expertise for improving the overall quality of healthcare, future simulations designed to instruct and train students, residents and other medical specialists must be able to adequately assess both performance and teamwork capabilities. The present effort improves upon previous simulation design recommendations in that it provides a concrete series of steps to constructing EBSs predicated on an evidence-based approach to measurement. In following the outlined procedure, medical educators’ can ensure that the design and development of EBSs in medical practice follow sound methodological principles, which ultimately allows for a comprehensive evaluation and critique of the teamwork and clinical skills required in proper medical practice. Step 2: Mapping and Measure Development • Identify observable teamwork behaviors evoked by scenario events that are indicative of team process dimensions. • Create a dichotomously scored checklist of behaviors that should occur in each event related to effective TEAM functioning (Figure 2) – activities representative of planning, helping, info sharing, etc. • Create behaviorally anchored rating scales that can be used for global ratings of teamwork effectiveness (Figure 3) • Identify observable performance achievements evoked by scenario events that are indicative of medical expertise. • Create a dichotomously scored checklist of objective performance indicators in each event related to effective TASK functioning – derived from physician standards of practice and physiologic data (e.g., correct drugs ordered, correct dosage prescribed, etc.) Team Process andEvent-Based Simulations Figure 5 Step 4: Simulation Assessment Team processes refer to broad categories of functioning that describe the activities of effective teams (e.g., planning, helping, information sharing, etc.). While numerous sets of team processes have been proposed, the present methodology works from a temporally-based taxonomy developed by Marks et al. (2001) because (1) the taxonomy facilitates the creation of an EBS assessment tool by indicating when team processes should occur and (2) empirical evidence exists linking elements of the taxonomy to team effectiveness (LePine et al., 2008). EBS methodologies explicitly structure a task (e.g., resuscitating a patient, attending to a trauma victim, etc.) such that pre-scripted events are included that require specific team responses. Coupled with the Marks et al. (2001) taxonomy, EBSs can thus be designed to elicit a wide range of identifiable teamwork and communication behaviors necessary for effective team and task functioning in a given context during certain periods of time. • Run the simulation with pilot teams to ensure performance data and team process indicators can be reliably captured. • If possible, capture video from multiple views in the simulation room to increase the likelihood that specific behaviors can be readily identified, recorded, and viewed after the simulation is completed (Figure 6) • Readjust events and measurement tools to capture desired teamwork behaviors and performance criteria as necessary. References Institute of Medicine (2000). To err is human: Building a safer health system. Washington, D.C.: Institute of Medicine. LePine, J. A., Piccolo, R. F., Jackson, C. L., Mathieu, J. E., & Saul, J. R. (2008). A meta-analysis of teamwork processes: Tests of a multidimensional model and relationships with team effectiveness criteria. Personnel Psychology, 61, 273-307. Marks, M. A., Mathieu J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26, 356-376. Rosen, M.A., Salas, E. Wu., T.S., Silvestri, S., Lazzara, E.H., Lyons, R., Weaver, S.J., & King, H.B. (2008). Promoting teamwork: An event-based approach to simulation-based teamwork training for emergency medicine residents. Academic Emergency Medicine, 15, 1-9. Shojania, K.G., Duncan, B.W., McDonald, K.M., Wachter, R.M.(2001) (Eds.). Making Health Care Safer: A Critical Analysis of Patient Safety Practices. Rockville, MD: Agency for Healthcare Research and Quality. Figure 6 Figure 2 Figure 3