PEP-II Operational Safety and Management

1. PEP-II Operational Safety and Management Presentation by Roger Erickson B-Factory Operations Review SLAC, April 26, 2006

2. Outline • Accelerator Matrix Management • Communications with Stakeholders • MCC Control Room Staffing • Operator Profile and Training • Accelerator Operations Documentation • Maintenance Management • ROD and Down Time Planning • Conclusions

3. Main Control Center

4. Matrix Management Accelerator programs are managed through a matrix organization structure with links to: • Program Coordinators for PEP-II and Linac Pulsed Programs. • Liaison persons representing each experiment. • System Physicists for each major section of the accelerator facility. • Area Managers – one for each major section, who report to the System Physicists. • Accelerator physicists and engineers responsible for subsystems or commissioning tasks. • Technical support departments with engineering and technician support for on-call maintenance. • Accelerator Division Safety Office. • ES&H Departments, including Radiation Physics and OHP.

5. Communications with Stakeholders • 8:00 Meeting every day, seven days a week, in MCC conference room. • Operations report by EOIC, including run time statistics, progress in carrying out scheduled program, descriptions of problems, etc. • Reports by representatives of major experimental user groups. • Attended by PPA, Accel Div, and Operations management reps, Program Deputy, System Physicists, Area Managers, and representatives of user groups and major technical support departments; typical attendance: 45 persons. • 8:15 Maintenance Meeting, five days each week. • Run by Accelerator Operations Department maintenance managers. • Attended by supervisors of all maintenance groups, Area Managers, and interested physicists and engineers. • Attendance: typically 45 persons. • Accelerator Plans Meeting, weekly. • Review of operational progress, problems, and plans. • Attended by Accel Systems Div and Dept Heads, Program Coordinators, Accelerator Section Heads and ADSO.

6. Control Room Staffing • Program Deputy • Duty rotates among Accelerator Division physicists. • Oversees program for 3 or 4 day period. • Not always present in the control room. • Operations Engineer • 1 EOIC per shift (from a group of 7), on rotating 7-week cycle. • Responsible for safely carrying out the scheduled program. • Exempt position; directs the Accelerator Systems Operators. • Must be present if accelerator is capable of producing beam. • All have experience as accelerator systems operators. • Accelerator Systems Operators Typically 3 operators per shift on rotating 5-week cycle: • Linac and sources. • PEP-II rings. • Secondary program, PPS control, and backup.

7. Typical Operator Profile Typical applicant is a recent graduate with a bachelor’s degree in physics. • 13 have bachelor’s degrees. • 7 have master’s degrees. • 4 have a technical background and extensive SLAC experience. Classifications: • Accelerator Systems Operator 1 • 1-2 years of experience • Accelerator Systems Operator 2 • 2-5 years of experience • Accelerator Systems Operator 3 • 5+ years of experience • Operations Engineer • 5+ years of experience

8. Operator Training Program • Workbook and evaluation for each classification: • Accelerator Systems Operator 1 • Accelerator Systems Operator 2 • Accelerator Systems Operator 3 • Operations Engineer • ES&H Training (including RWT-II and LRCA) • Accelerator Operations Safety Procedures Training (including PPS operation, searching, interlock testing for each zone.) • US Particle Accelerator School • Downtime seminars and special projects

9. Accelerator Operations Documentation

10. AD Safety Documents with Site-Wide Applicability • SLAC Guidelines for Operations • Work Authorization for the Linac, PEP-II, FFTB, and Associated Buildings • Summary of Requirements for Work in SLAC Accelerator Housings • Restart Plan for the Linac, PEP-II, and BABAR • Restart Plan for the FFTB and ESA Accelerator Programs as an Extension of the Linac, PEP-II, and BABAR Restart Plan • Linear Accelerator Facility Safety Assessment Document • PEP-II Safety Assessment Document • BaBar Safety Assessment Document • Policy for Working with Electrical Hazards in SLAC Accelerator Tunnels • Lock and Tag Verification Procedures for Accelerator Magnets in the Linac Complex and PEP-II

11. MCC Key Safe Checkout Log CID PPS Log DRIP/DR PPS Log Linac PPS Log Positron PPS Log BSY PPS Log ESA PPS Log FFTB PPS Log PEP PPS Log PLS Keys Procedures and Log MCC Spare Keys Key Safe Checkout PPS Zone Maps Logging Requirement for Work on BCS Devices VVS Remote Turn-on Procedure Search Procedures Electrical Hazard Test Procedures Entry and Exit Procedures Safety Inspection Checklists PPS Interlock Checklists Beam Containment System Procedures High-Voltage Test Facility Safety Procedures BSOIC Certification Checklists Gun Safety Procedures Laser Safety Procedures Fire Alarm System Reference Manual PEP PPS Overview Safety Configuration Work Control Incident Response Procedures Alarm Response Procedures Warning Response Procedures Accelerator Operations BAS Instructions Accelerator Operations BAS Key and Lock Procedures AcceleratorOperations Safety Documents • 163 controlled documents in the Accelerator Division (incl. non-safety related). • MCC Operations Log is now electronic, web-accessible, and archived.

12. Operator Safety Procedures Operators are trained to carry out certain well-defined tasks, following formally controlled procedures. Examples: • Interlock Checklists • Search Procedures • Entry and Exit Procedures • Safety Inspection Checklists Safety procedures: • undergo a formal review and approval process. • are updated as conditions change. • are available on the AD Documentation web site, although the signed originals are preserved in paper form. Completed checklists and log books are archived.

13. Maintenance Management • Problems that interrupt the scheduled program trigger immediate repairs to restore operations. • 24-hour on-site staff to repair: • Mechanical utilities • Power supplies • Modulators • Cryogenics • On-call staff for: • Electrical power • Magnets and other mechanical systems • RF systems • Vacuum systems • OHP • Extensive call-in lists identify first responders, alternates, and experts for all systems.

14. Opportunistic Maintenance • Opportunistic Approach • Whenever possible, identified maintenance, repair, and installation jobs are pre-planned, but deferred until an opportunity arises. The accelerator systems are not turned off for maintenance on a routine, scheduled basis. • Coordination • Daily 8:15am meeting. • Coordinated by Operations Maintenance Office. • Area Managers are advocates for their areas. • Standby Maintenance List (SML) • Jobs that are planned and can be launched on short notice. • Provides options for EOIC to take advantage of unscheduled interruptions. • Repair Opportunity Day (ROD) • Launched only when required by a driving event. • Carefully planned, and typically involve 100 or more jobs. • ARTEMIS Electronic Database

15. ARTEMIS • ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accelerator maintenance. • Operators report problems as they occur by making entries in ARTEMIS. • Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program. • New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned. • Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down time. • Jobs are tracked using ARTEMIS until they are finally closed.

16. Daily Maintenance Meeting

17. Review of New Problems Sample from an actual maintenance meeting.

18. Standby Maintenance List

19. Repair Opportunity Days • A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations Department. Installation and PM jobs are added as needed. • All ROD jobs are reviewed in the daily maintenance meetings. • The ROD list can be viewed over the internet. • When a “driving event” occurs, a detailed plan can be assembled and launched quickly, using the ROD list. • A typical ROD day includes about 100 distinct jobs.

20. Long Down Time Planning • Long Down Times usually occur once each year and typically last two to three months. • Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database. • A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about four months before the down time. • This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager. • Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required. • Schedule specifies authorizations required (WAF, RSWCF, EWP, etc.). • Detailed (shift by shift) schedule of pre-run testing, safety certifications, and turn-on activities is prepared by the Operations Dept and revised as needed on a real-time basis.

21. Maintenance Statistics • ARTEMIS entries since Jan 1989: 77,328 (as of April 18, 2006) • Total identified problems not yet closed: 456 Many of these involve non-critical problems that were observed, but went away without a clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance. • ARTEMIS entries in one year (since April 18, 2005): • 3166 new problems (8.7 per day). • 450 problems resulted in lost PEP-II run time. • 3296 problems closed (9.0 per day). • Active jobs: • SML: 52 • Next ROD: 32 • Aug-Nov down time: 80

22. Operational Safety Reviews • Linac and PEP-II Re-Start Validation Review (March 2005) • FFTB Re-Start Review (June 2, 2005) • ESA Re-Start Review (July 19, 2005, secondary beams only) • Director’s Integrated Safety Management (ISMS) Review (Sept 2005) • DOE ISMS Review (October 2005) • Injector Test Facility Re-Start Validation Review (October 2005) • ESHAC Review (November 2005) • ESHAC Review (March 2006)

23. New Safety Processes Introduced in FY05 • JHAMs (Job Hazard Analysis and Mitigation) individualized and documented for every employee. • Expanded training requirements for most workers. • AHAs (Area Hazard Analysis) posted for every building. • Arc-flash analysis and warning labels for circuit breakers and contactors (except Cat 0 light switches, etc.) • All hoisting and rigging equipment, including lifting fixtures, straps, etc. must be certified. • Fall protection equipment and training required for tall ladders, raised platforms, rooftop work, etc.

24. Conclusions • PEP-II B-Factory operations are managed by the Accelerator Systems Division in the PPA Directorate. • Control room operations are carried out by trained operators in compliance with documented policies and procedures for all safety-related activities. • Maintenance and installation work is accomplished through a matrix organization, drawing on engineers, technicians, and ES&H and shop services from the Operations Directorate. • We have developed organizational structures, software tools, and documentation that enable us to manage an ambitious accelerator-based science program safely and effectively. • Enhanced policies, procedures, and training programs introduced in the past two years ensure that all our work is carried out with high standards for safety.