DIII-D Safety Program – Incidents and Lessons Learned

1. DIII-D Safety Program – Incidents and Lessons Learned • By • Kellman • Director, DIII-D Operations • Presented at the • DOE Accelerator Safety Workshop • Fermilab, Batavia, IL • September 20-22, 2016

2. Outline • Introduction to the DIII-D Facility • Overview of Safety Program • Incident/Near Miss Statistics and Summary • Review of Recent/recurring Incidents • Proactive Safety initiatives

3. DIII-D System Capabilities Heating and Current Drive (injected power/pulse) • Neutral Beams: 8 sources; 20 MW (80 keV Deuterium) • Electron Cyclotron (Microwaves): 6 gyrotrons@110 GHz; 3.5 MW (5 s) • Fast Wave (RF): 3 MW, 60-100 MHz Coils • Toroidal field – 2.2 T on axis • 18 Poloidal field shaping coils • 6 external coils, 12 internal coils Plasma Control • 17 CPUs, 40 GB real-time CPU network Divertor/First Wall/Conditioning • 3 LHe cryopumps; 15–20,000 l/s • ATJ graphite • 350°C bake, boronization, Helium glow discharge between pulses DIII-D INTRODUCTION

4. Aerial View of DIII-D Facility, Located in San Diego, CA 138kVAC/480VAC Substation 5MW West Substation (Fed from Dunhill) Low & High Pressure Deionized Water Pumping Systems Two RF Transmitter PSs 30kVDC, 125A Helium Gas Storage Five Gyrotron PSs 105kVDC, 90A Diagnostics Lab Eight Neutral Beam PSs 120kVDC, 100A Control Room 12.47kVAC Switchgear (from 138kVAC Line) Dunhill Substation 138KVAC & 69kVAC Substations DIII-D Machine Three Cooling Towers 525MVA Motor Generator DIII-D INTRODUCTION

5. Multiple Hazards Exist at DIII-D • Ionizing radiation (neutrons, gamma rays, X-rays), tritium • Electrical (120V - 138 kV AC, >100 kVDC); high current (>100 kA) • Microwave (110 GHz >1MW), RF (100 MHz >2 MW) • Magnetic fields (<5T) • Lasers (Visible and IR, Class I – IV, pulsed and SS) • Cryogenic (LHe/LN2 – asphyxiation, cold, embrittlement, pressure) • Explosive & toxic gases (D2, diborane, Ar, Ne, Xe, N2, CD4, CF4, and CCl2F2) • High pressure (water < 300 psi, hydrauilc < 5000 psi, compressed gases <2200 psi) • High noise levels • High temperature surfaces (350C) • Confined spaces SAFETY PROGRAM

6. Outline • Introduction to the DIII-D Facility • Overview of Safety Program • Incident/Near Miss Statistics and Summary • Review of Recent/recurring Incidents • Proactive Safety initiatives

7. Safety Program at DIII-D is Based on Integrated Safety Management (ISM) • 7 Guiding Principles • Line Management Responsibility for Safety • Clear Roles and Responsibilities • Knowledge Consistent with Responsibilities • Balanced Priorities • Identification of Safety Standards/Requirements • Hazard Controls Tailored to Work Being Performed • Operations Authorization • 5 Core Functions • Define the Scope of Work • Analyze the Hazards • Develop/Implement Hazard Controls • Perform Work Within Controls • Feedback/Continuous Improvement

8. DIII-D Safety Program Supported by General Atomics Corporate Safety Organizations • DIII-D Operates Under a DOE Cooperative Agreement • Externally regulated by Cal-OSHA and other state and local regulations • Radiological exposure governed by State of California Radiation Control Regulations (Title 17). DIII-D procedures and ALARA program set targets below legal limits. • DIII-D program supported by General Atomics Corporate Environmental Safety & Health Group and Health Physics Group • DIII-D Safety Committee and Energy Group Safety Officer (employee of GA ES&H Group) SAFETY PROGRAM

9. Hazards are Identified for Each Taskand Mitigation Developed • Each task is examined and the hazards are identified • Hazardous Work Authorization (HWA) is prepared by PI • Task and system description • Describes the hazards • Identifies the methods of mitigation, PPE and required training • Lists personnel involved in each subtask • Permits required, Hazardous chemicals involved • Specifies emergency response, if specialized for system • Draft plan for safe operation of system presented at first design review • HWAs are reviewed by DIII-D and GA line and safety management • Reviewed and approved prior to performing work • Specialized subcommittees of DIII-D Safety committee provide review of HWA as needed (electrical, laser, chemical) • Updated annually or after changes SAFETY PROGRAM

10. Training of Personnel Plays a Critical Role in Safety • All new workers entering the DIII-D site to perform work receive a safety check-in sheet and safety indoctrination • DIII-D employees, GA maintenance staff, students, post docs, collaborators, contractors • DIII-D Safety training complements General Atomics safety training • Job classification and function for each person is developed jointly with supervisor based on assigned tasks and responsibilities • Required training is identified for each job classification • Classes, hands-on, and on-line training is available • Supervisor and employee review job classification and required training annually • Web-based program is used to assign, track training, registration • Automatic reminders sent to staff and supervisors SAFETY PROGRAM

11. Outline • Introduction to the DIII-D Facility • Overview of Safety Program • Incident/Near Miss Statistics and Summary • Review of Recent/recurring Incidents • Proactive Safety initiatives

12. DIII-D Safety Statistics – 2011-2015 Incidents and Near Miss Injuries have decreased in past 5 years • Categorizing incidents • Improved investigations • Targeted safety initiatives • Improved safety awareness • Encourage ‘Near Miss’ reporting Reporting and investigation of ‘Near Miss’ events has increased • Decreases the probability of future incidents • Indicates that employees are taking a proactive approach to safety STATISTICS

13. Most Injuries are Due to ‘Low-Tech” Activities • Engineering access controls are in place for high hazard areas, e.g. HV STATISTICS

14. Outline • Introduction to the DIII-D Facility • Overview of Safety Program • Incident/Near Miss Statistics and Summary • Review of Recent/recurring Incidents • Proactive Safety initiatives

15. Case 1 - Faulty Interlock Design Highlighted Systemic Weakness in Interlock Design and Checkout Process • Finding: Laser interlock systems was identified in HWA to prevent exposure to high power CO2 laser and low power HeNe alignment laser, but Interlock system on HeNe laser failed to operate properly and shut laser down when access to the machine hall was granted. • Proximate cause: Interlock system was not designed properly (not fail-safe) and did not turn laser off when power to interlock system was inadvertently disconnected. • Root cause: Inadequate review of Interlock system designs or design changes; systems not tested properly • Corrective Action: • All interlock circuit designs must be reviewed by Electrical Safety Subcommittee - verify proper functionality, fail-safe operation, thorough initial test plan, annual interlock test plan • Any changes to interlock circuitry must be re-reviewed by Electrical Safety subcommittee (or laser subcommittee if appropriate) INVESTIGATIONS

16. Case 2 - Near Miss During Approved Bypass of Access Control Interlock System • Finding: • Exposed buswork was energized while employee was present in area and near buswork • Access control system interlock for area was bypassed; interlock bypass procedure checklist was completed & approved prior to incident • Contributing Factors and Additional Findings: • 2 hazardous sources in area; bypass was for local testing on one system (Audio Amps) • Both hazardous systems were undergoing testing but local testing (Audio Amps) started prior to remote testing involving 2nd system • Poor communication via radios between local testing team and control room to inform test team to leave area when 2nd system was to be energized remotely • Bypass form was used for more than 1 day without reapproval on day of the incident INVESTIGATIONS

17. Case 2 - Near Miss During Approved Bypass of Access Control Interlock System (cont’d) • Corrective Actions: • Bypass procedure revised with the bar set much higher for approving a bypass of a safety interlock. • Three levels of management approval required for safety system bypasses. • Substitution of administrative controls requires examination of all the possible personnel hazard sources and multiple controls • Only one system may be tested in area with an interlock bypass. All other possible hazardous sources must be de-energized and made “safe” by LockOut/TagOut • Bypasses limited to 1 day • Improved training and checklist for implementing procedure • Two separate check lists with different requirements and approvals for bypassing interlocks associated with personnel safety and hardware protection. INVESTIGATIONS

18. Case 3 - 13.8 kV Bus Arc – Explosion and Arc Flash Damaged line reactors • Explosion between MG output and switchgear for DC supplies • Power distribution yard inaccessible during operations • Two plausible failure scenarios were identified • Cable from 13.8 kV Bus to line reactor failed OR • Switchgear from 13.8 kV bus to power supply arced across phases Damaged Switch gear Motor Generator 13.8 kV, 525 MVA Pulsed Motor Generator Breaker and Disconnect INVESTIGATIONS

19. Case 3 - 13.8 kV Bus Arc – Explosion and Arc Flash Insulators to Line reactor Switch gear output cabling Line Reactor INVESTIGATIONS

20. 13.8 kV Bus Arc – Contributing Factors • Aged & deteriorating equipment • General spacing and tracking distance within switchgear cabinet were less than current industry best practices • Insulator length shorter than current standards • Insulator material aged and glazing deteriorated • Preventative Maintenance practices were inadequate • Inspection routine not specific enough • Expertise lost due to retirements • Quantitative measurements were lacking; long term deterioration of performance not recognized • Weather – heavy rain; equipment is outdoors INVESTIGATIONS

21. 13.8 kV Bus Arc – Corrective Action • Corrective Actions • Preventative Maintenance procedures improved • Outside consultant on PM used to review recovery • Bring PM activities up to current standards • NFPA 70B 2013 (Recommended Practice for Electrical Equipment Maintenance)& • ANSI/NETA MTS-2015 (Maintenance Testing Specifications) • Replace aging switchgear– older equipment similar to failed unit are no longer industry standard/best practice • No injuries resulted from failure, despite large release of energy • Engineered access controls • Prevents personnel entry during operation when systems are energized • Emergency response was prompt and appropriate • System was fully de-energized & possibility of unreleased stored energy was evaluated prior to ERT access to damage area INVESTIGATIONS

22. Case 4 - Recurring Injuries to Head and Hand • Frequency of hand and head injuries led to new programs • Systematic search for any sharp or protruding edges, corners on equipment or structures – problems addressed • Both gloves and hard hats were provided and hard hats were required when entering machine hall • Despite availability of PPE, lacerations or hands, fingers, scalp continued • Staff discussions revealed: • Ill-fitting gloves meant people did not use gloves • Hard hats were removed when people were working in tight and constricted areas • Range of better fitting gloves provided to staff who got to choose their preferred type and size • Bumps caps were provided in addition to hard hats • Available at multiple locations around work area with hygienic hair covers to use with community/shared bump caps INVESTIGATIONS

23. Case 5 – Failure to Achieve 100% Safety Orientation and Training for New Staff • Finding: Staff member began working without required safety check-in and orientation, job safety review, or safety training • Cause: Holes in new staff procedure enabled people to begin working without required safety review and orientation • Procedure failed to block all entry paths for new staff (corporate hires, collaborators, students, post-docs, visiting scientist) or staff with changing roles • Corrective action: • Identify all entry paths for new staff • Clarify roles and responsibilities of employees, supervisors, and admins • Identify and improve training of all gatekeepers (administrative staff, facility receptionist, GA and functional supervisors) INVESTIGATIONS

24. Case 6 – HV Line Arc Faults Due to Foreign Objects Bridging Lines INVESTIGATIONS

25. Case 6 – HV Line Arc Faults Due to Foreign Objects Bridging Lines • There have been three incidents since 1978 of birds dropping snakes across HV AC Power lines leading to major power outages with damage to electrical equipment • 2 of 3 incidents resulted in a fire under the line because the grass under the line ignited. • In the 3rd incident, the incident occurred in the substation area where the weeds were cleared and no fire resulted. • Corrective Action: • All areas under overhead transmission lines (on our property) and within substations are maintained clear of all weeds INVESTIGATIONS

26. Summary of Lessons Learned • A common factor in all the incidents is the need to improve the development and implementation of the controls • Increase use and proper design of engineered controls

27. Outline • Introduction to the DIII-D Facility • Overview of Safety Program • Incident/Near Miss Statistics and Summary • Review of Recent/recurring Incidents • Proactive Safety initiatives

28. Tool Training and Use Program (1) • Program challenge and overview: • A wide range of personnel from different institutions with different skill levels build, maintain, and operate equipment. • Tools are available from stock room for checkout and in multiple locations throughout the facility, and generally are not locked. • Provide training in hand and stationary tools (generally power tools, but not exclusively powered) efficiently and effectively. • Provide tool access only to those trained in the safe use of the equipment • Maintain equipment in safe operating condition. INITIATIVES

29. Tool Training and Use Program (2) • Key Attributes of the Program: • Area supervisors are responsible for proper maintenance of equipment in their areas • Experienced and trained individuals are identified as trainers for different pieces of equipment. • Supervisor and lead tech identify required tool training for their staff • Tool use authorization spreadsheets are located in each tool area identifying who is qualified to use each piece of equipment • The following must be completed in order to use power tools: • All tool users must attend initial Tool Safety training class • The Tool Custodian (tool expert assigned to a particular tool) observes and approves the new user for competency and safety • Safety Rules are read and signed for that equipment • A list of all qualified users is posted in all tool areas; warnings against use of tools without being on user list INVESTIGATIONS INITIATIVES

30. A Task Safety Checklist Has Been Developed to Help Promote Daily Safety Discussions • Small, individual work groups discuss task and safety issues each day prior to start of work. • Potential hazards, mitigation and/or PPE are discussed • Lessons learned from previous days work on task are reviewed • Discussion can be led by lead tech, lead engineer, or any member of the workgroup • All members of the group initial the checklist after discussion • Checklists are short and simple INITIATIVES

31. A Task Safety Checklist Has Been Developed to Help Promote Daily Safety Discussions and Review INITIATIVES

32. Summary • DIII-D has a successful system for assuring safe and efficient operations • Hazards identified for each task and mitigation is put in place to protect personnel & equipment • Safety training for all personnel is regularly evaluated, provided on an on-going basis, and tracked • Processes and procedures in place to control work • Lessons learned from investigations of both injuries and ‘near miss’ incidents have significantly improved the safety program • Wide use of engineered controls are key to strong safety record at DIII-D • Most injuries result from common industrial-type hazards – training and safety awareness is key