Society of Medical Innovation and Technology Amsterdam, Netherlands August 28, 2003

1. Simulators for Training: Assessment, validation and acceptance strategies workshop Richard M. Satava, MD FACS Professor of Surgery University of Washington School of Medicine and Program Manager, Advanced Biomedical Technologies Defense Advanced Research Projects Agency (DARPA) and Special Assistant, Advance Medical Technologies US Army Medical Research and Materiel Command Society of Medical Innovation and Technology Amsterdam, Netherlands August 28, 2003

2. Deficiencies for Simulation No objective measures for technical skills No agreement on what a skill is No classification of types of skills to teach No relation between skills trained and used in OR No standardized method of validating

3. Goals of the Workshop Review current available systems Definition of what is being measured Develop a taxonomy for measurements Match metrics to current systems Develop a core curriculum model Provide validation of the simulation science

4. Definitions (cont.) competent (Fr. competent <competer , to be sufficient; L. competo, to be suitable) Answering all requirements; suitable; fit; adequate; having legal capacity or power; rightfully or lawfully belonging . competencen. State of being competent; adequacy; sufficiency; property or means of subsistence sufficient for furnish the necessaries and conveniences of life. proficientadv. (L. proficiens, -entis, p. pr. of proficere to go forward, make progress; pro forward + facere to make) Well advanced in any branch of knowledge or skill; possessed of considerable acquirements; well-skilled; versed; adept. proficiencyn. The quality of state of being proficient; advance in the acquisition of any art, science, or knowledge; progression in knowledge; improvement; adeptness. exercisen. (F. exercice, L. exercitium, from exercere, exercitum, to drive on, keep, busy, prob. orig., to thrust or drive out of the inclosure;ex out + arcere to shut up, inclose). A task, problem, or other effort performed to develop or maintain fitness or increase skill; an activity having a specified aspect systemn. (Late Latin systma, systmat-, from Greek sustma, from sunistanai, to combine : sun-, syn- + histanai, set up, establish) A group of interacting, interrelated, or interdependent elements forming a complex whole; functionally related group of elements; an organized set of interrelated ideas or principles; An organized and coordinated method; a procedure.

5. Taxonomy ABILITIES (Aptitude) SKILLS TASKS PROCEDURES

6. Taxonomy ABILITIES Psycho motor Tracking Pick and place Translation Aiming (Targeting) Precision Visio-spatial Card rotation Cube comparison Map planning Perception Haptic

7. Taxonomy Skills Instrument handling Bimanual dexterity Transfer/traversal Precision Peg Board, etc Navigation Ligation Foam, Trotters, Bowel (ex vivo), Vascular, Tendon, Laparoscopic Suturing Open, Laparoscopic Knot tying Open, Extra-corporeal, Intra-corporeal Incision Exploration (both visual and haptic) Palpation Cannulation Tissue handling Cutting Blunt Dissection Clamping (clip application) Hemorrhage control (simple) Plaster application

8. Taxonomy Tasks Anastomosis Bowel, Vascular, Laparoscopic Excise Superfacial lesion Deep lesion (e.g. Breast) Closure (especially wound) Tissue extraction Exploration (probing) Camera naviagation Needle IV insertion, Aspiration, Injection, Pericardiocentesis Debridement Dissection, excision, etc Morcellation Energy use Diathermy, scarifying, ablation Stents Implant Prosthesis, mesh, etc Hemorrhage Control Mesh Placement Evacuation

9. Taxonomy Procedures Laparoscopic cholecystectomy Tracheostomy Open, percutaneous Chest tube insertion Diagnostic peritoneal lavage Vein patch Breast biopsy (to be developed) Node dissection Ultrasonic diagnosis Endoscopic* Sinusoscopy, EGD/ERCP, Bronchoscopy Colonoscopy, Arthroscopy Image guided Coronary stent

10. Systems and Exercises 1. ADEPT Advanced Dundee Endoscopic Psychomotor Tester 2. University of Toronto (OSATS) Objective Structured Assessment of Technical Skills 3. MISTELS McGill Inanimate System for Training and Evaluation of Laparoscopic Skills 4. MIST-VR Minimially Invasive Surgery Trainer – Virtual Reality 5. ICSAD Imperial College Skill Assessment Device 6. Rosser Drills Rosser Drills, Yale University 7. PicSor & Fundamental Abilities Pictorial Surface Orientation (PicSoR), Card rotation, Cube comparison, map planning 8. ESSS Endoscopic Sinus Surgery Simulator 9. FSM Fundamental Surgical Manipulations 10. LTS 2000 Laparoscopic Training Simulator – 2000 11. LapSim Laparoscopic Simulator ABILITIES SYSTEM(S) AVAILABLE (see above for number references) Psycho motor Tracking 1,2,3,4,5,10 Pick and place 1,2,3,4,5,10 Translation 1,2,3,4,5,10 Aiming (Targeting) 1,2,3,4,5,10 Precision 1,2,3,4,5,10 Visio-spatial Card rotation 7, Cube comparison 7 Map planning 7 Depth Perception PicSOr 7,8,9 Haptic __________________

11. Skill Levels Level 0 Pre-training (aptitude) Level 1 Basic Level 2 Intermediate Level 3 Advanced

12. Curriculum Template Level 1 Basic Training TASKS Anastomosis Bowel Excise Superfacial lesion, deep Closure (especially wound) Tissue extraction Exploration (probing) Camera naviagation Needle Insertion, Aspiration, Injection Debridement (dissection) Energy use (diathermy, scarifying) PROCEDURES Gown and glove Tracheostomy Chest Tube Insertion Diagnostic Peritoneal Lavage Flexible sigmoidoscopy SKILLS Instrument handling Bimanual dexterity Transfer/traversal Peg Board Navigation Ligation Suturing Knot tying Open Incision Exploration (both visual and haptic) Palpation Cannulation Tissue handling Cutting Blunt Dissection Clamping (clip application) Hemorrhage control (simple) Plaster application

13. Validation When evaluating the proposed systems and tests, must be able to demonstrate 1. Validity 2. Reliability

14. Types of Validity* Face Content Construct Concurrent Predictive *Reber, AS Dictionary of Psychology

15. Face Validity “… experts review the tests to see if they seem appropriate ‘on their face value’…”* EXAMPLE The chosen tasks resemble those that are performed during a surgical task *Reber, AS Dictionary of Psychology

16. Content Validity experts perform “… a detailed examination of the contents of the tests . . . to determine if they are appropriate .. . and situation specific …”* EXAMPLE The tasks for measuring psychomotor skills are actually measuring those skills and not anatomic knowledge *Reber, AS Dictionary of Psychology

17. Construct Validity “… the determination of the degree to which the test captures the hypothetical quality it was designed to measure …”* EXAMPLE The tasks were designed to test the level of a skill, therefore an expert should perform better than a student. *Reber, AS Dictionary of Psychology

18. Concurrent Validity “… the relationship of the new test scores … (and those) whose performance has been evaluated in actual working conditions …”* EXAMPLE The scores on the test corresponds to scores on the current similar or “gold standard” tests under real circumstances *Reber, AS Dictionary of Psychology

19. Predictive Validity “… determining the extent to which the scores on a test are predictive of actual performance …”* EXAMPLE Those who do very well on the tests will do very well in the operating room *Reber, AS Dictionary of Psychology

20. Types of Reliability* Inter-rater reliability Test-retest reliability

21. Inter-rater Reliability … determining the extent to which two different evaluators (raters) score the same test … EXAMPLE Two surgeons evaluate a student performing dissection of the gallbladder and both agree on the same errors, time, etc scores. (p > 0.80)

22. Test-retest Reliability “… reliability of a test by administering it two (or more) times to the same persons and obtaining a (correlation) between the scores on each testing …”* EXAMPLE Students are tested twice on the same test and get equivalent scores each time. (careful - learning curve) *Reber, AS Dictionary of Psychology

23. Definitions abilityn. (Fr. habilite, L.habilitas\, ableness, ABLE). The state or condition of being capable; aptitude; competence; capability; power to do something, physical, mental, legal, etc. Usu. Pl. talents, acquired proficiencies; powers of mind; mental gifts or endowments. skill, n (Icel. skil, Dan skiel, discrimination, discernment). A developed proficiency or dexterity in some art, craft, or the like; deftness in execution or performance; a trade or craft requiring special training for competence or expertness in its practice. taskn (O. Fr. tasque, tasche, a task L. taxare, to tax) A piece of work imposed upon a person by another; a piece of work to be done; that which duty or necessity imposes; an undertaking; a burdensome, difficult or unpleasant chore or duty; a difficult or tedious undertaking. proceduren. (F. proc['e]der. fr. L. procedere, processum, to go before, to proceed; pro forward + cedere to move). A series of steps taken to accomplish an end; a manner of proceeding; a way of performing or effecting something

24. Summary Defined terms for technical skills Developed classification of skills Categorized skills into training levels Developed matrix to choose simulators for curriculum development Proposed a method of validating