Research Paper Critical Analysis

1. Research Paper Critical Analysis 10 ways to look at a research paper systematically for critical analysis.

2. NUR 599T Human Factors In Healthcare Engineering Dr. Vincent Duffy Research Paper Analysis Submitted by RuthAnn Smolen, RN, MS Ed, HNP, July 19, 2007

3. STUDY CRITIQUETitle: Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral RatingsAuthors: Gaba, D., Howard, S., Flanagan B., Smith, B., Fish, K., & Botney, R.

4. Research Ideas or Question(Study and Purpose) • Study: “Assessment in clinical performance” Gaba, et.al.) • The purpose is clearly identified in the introduction - to evaluate ratings of performance (technical and behavioral), and variability of ratings made by multiple independent observers viewing videotapes of simulated crises • The use of human patient simulators (HPS) in health-care training is increasing to gain knowledge, teach technical skills, complete drill training, and human factors training for single or multidisciplinary groups

5. 2. Background • Anesthesiology is recognized as the leading medical specialty in addressing patient safety with HPS • Pioneered development of computer screen and mannequin based interactive patient simulators and training plans (Schwid, 1987; Small, Wuetz, Simon, Shapiro, Conn, Setnik, 1999). • A persistent initiative in graduate education is to find more effective methods to assess specialists’ skills • “Techniques are needed to assess anesthesiologists’ performance when responding to critical events” (Gaba, 1998) • Traditionally there has been concern about the potential subjectivity of expert raters.

6. 3. Theoretical Basis/Hypothesis • Helmreich, RI Theory underlying CRM training: Psychological issues in flight crew performance and crew coordination (NASA, 1986) in teams • Behavioral psychologists and learning theories were used originally • In anesthesia, patient safety depends on a complex interaction between a skilled practitioner and technology • A human patient simulator may reduce the risk of learning-curve error in anesthesiologists technical and behavioral clinical performance without subjecting patients to risk

7. 4. Literature Search • 13 reports published between 1980 & 2000; 4 were designed to investigate validity or reliability of HPS systems; overview of existing evidence of the problem is addressed in follow-up searches of today • In this study the literature review is supportive of purpose and results, current for 1998, and thorough • Summaries of other comparative studies are discussed, relevant to the study and includes strengths and weaknesses • Chopra, et al. (pg. 13) • Devitt, et al. (pg. 13) • Law and Sherman (pg. 15)

8. 5. Applicability • The aviation industry’s commitment to simulation and similarities between anesthesia and aviation was a key rationale for developing courses in crisis management and using simulation in anesthesia education • Generalization among various acute events in simulation teaching and evaluation environments across healthcare disciplines is a reality today

9. 6. Methodologies • Raters scored the videotapes of 14 different teams that were managing two scenarios: malignant hyperthermia (MH) and cardiac arrest. Technical performance and crisis management behaviors were rated.

10. 7. Statistical Analyses & Assumptions • Using three observers, the study shows good interrater reliability in scoring technical achievement with less agreement between five raters scoring ACRM behaviors. • The authors analyze the statistics of interrater reliability in depth and note that the scoring system requires further development before it can be used for assess clinical competence for residency/graduation, or board certification

11. 8. Presentation of Results • What do the results really mean? Team performance varied and correlations among interraters were better for technical markers than for behavioral markers

12. 9. Conclusions (are they reasonable?) • The rating system needs refinement before it can be used to assess clinical competence for residency/graduation, or board certification • This study suggests that simulation can be used to measure more complex skills expected in specialty training • Similar to other studies assessing complex technical and behavioral performance multiple encounters are needed to estimate skill effectively and accurately

13. 10. Future Work/Research Direction • Studies emphasize the cognitive and social aspects of designing and operating advanced technologies yet to be developed • The behavioral rating system can be improved – difficulty aggregating a single rating that fluctuated with time • Human Factors and ergonomics link quantitative skills to the integrated systems and make sure design of systems act in accordance with requirement of users; includes interactions with technologists • Questions: • Is there a need for simulation in assessment of clinical performance? • Do current simulators match the fidelity of a real-live clinical setting? Are they realistic? • What are the potential uses of clinical simulation in medical/healthcare education? • What role exists for simulation in healthcare professional training and assessment? • Do collaborative opportunities exist?