Surgeons non-technical skills: Reliability testing of the NOTSS system

1. Surgeons non-technical skills: Reliability testing of the NOTSS system Association for Surgical Education, Washington DC, April 2007 Steven Yule, Rhona Flin, George Youngson University of Aberdeen Simon Paterson-Brown, Nikki Maran Royal Infirmary of Edinburgh David Rowley University of Dundee Project sponsors: Royal College of Surgeons of Edinburgh (RCSEd) NHS Education for Scotland (NES)

3. right hepatic duct Common bile duct Portal vein Right hepatic artery

4. “High technical proficiency cannot guarantee safety” Surgeons’ non-technical skills defined: “Behavioural aspects of performance in the operating theatre which underpin medical expertise, use of equipment and drugs. They are cognitive (i.e. situation awareness) and social (i.e. communication & teamwork) skills”

5. Research on non-technical skills • Adverse events in surgery are primarily caused by failures in perception, judgement, communication and teamwork • Wilson et al (1999) - communication breakdown in 43% of errors • de Leval et al (2000) – cardiac switch operations • Way et al (2003) – 97% of bile duct injuries had perception failures • Gawande et al (2003; 2004) – insurance claims in the US • Good non-technical skills can lead to positive outcomes for the team and patient • Edmondson (2003) – effective leadership • Carthey et al (2003) – behavioural markers in cardiac surgery • Healey et al (2004) – observing teamwork in surgery • Moorthy et al (2005) – team skills in operating theatre simulator

6. NOTSS project background • Rationale • Support safe practice in the operating theatre and surgeons’ professional training • Aim • Identify the essential non-technical skills for surgeons • Develop system for surgeons to rate skills in operating theatre and give structured feedback for training • Establish reliability & usability of system • Scope • Develop an education tool • Intraoperative environment only • For surgeons to observe, rate, and provide feedback

7. Developing the NOTSS system(based on Gordon, 1993, p.12) Phase 1: Task analysis 1. Literature review (Surgery, 2006) 2. Adverse event analyses 3. Cognitive Interviews (n=27 surgeons, n=12 nurses) 4. Observation in theatre 5. Attitude survey (The Surgeon, 2006) • Phase 2: Design and development (Medical Education, 2006) • Develop skills taxonomy • Refine categories and elements (4 panels of surgeons) • Write behavioural markers (n=16 surgeons) • Prototyping and usability testing • Phase 3: System evaluation • Finalize evaluation criteria – reliability and usability • Design evaluation strategy: rwg, in-theatre trials • Collect and analyse data (44 surgeons; c. 100 cases) • Modify system if necessary

8. NOTSS skills taxonomy v1.2(Yule et al, Medical Education, 2006)

9. NOTSS System Structure Surgeons’ Non-Technical Skills Situation Awareness Decision Making Communication and Teamwork Leadership Skill Categories Gathering Information Skill Elements Understanding Information Projecting and anticipating future state Behavioural Markers Good: Keeps anaesthetist informed about procedure (e.g. to expect bleeding) Poor: Waits for a predicted problem to arise before responding

10. NOTSS rating scale 1Poor Performance endangered or potentially endangered patient safety, serious remediation is required 2Marginal Performance indicated cause for concern, considerable improvement is needed 3Acceptable Performance was of a satisfactory standard but could be improved 4Good Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others N/A Not Applicable. Skill was not required or not relevant in this case or scenario

12. NOTSS reliability study Purpose • Test how reliably consultant surgeons use NOTSS to rate behaviours in standardized scenarios Scenario development • Good & poor behavioural examples in 9 scenarios • ‘case mix’ of orthopaedic & general surgical scenarios • Simulator used as patient, surgeons and nurses as actors Participants • n=44 consultant surgeons (mean experience 8.9 yrs (sd 7.5)) • 2-3 hour training session Analysis • Within group agreement (rwg) • Intraclass correlations (ICC) • Reference/ expert ratings

13. NOTSS video scenarios Decision Making Situation Awareness Communication & Teamwork Leadership

14. Reliability study - results *rated separately, not an average of element ratings

16. Critical level

17. Limitations • Novel task for surgeons • Limited training for participants • Calibration would improve reliability • Inter-rater agreement will improve with experience of using the system and non-technical language

18. Current research • Controlled usability trial in the OR, using NOTSS to observe, rate and debrief on performance • Examine the relationship between non-technical and technical skills Contact: s.j.yule@abdn.ac.uk Project website: www.abdn.ac.uk/iprc/notss