Summary for Working Group of Operations, Reliability, Injection, and Instrumentation

1. Summary for Working Group of Operations, Reliability, Injection, and Instrumentation Michael Billing Roger Erickson* Alan Fisher* Shigeki Fukuda Eiji Kikutani Mario Serio Jim Turner Fuhua Wang Uli Wienands * Associates 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

2. Motivation Expectation is when a collider achieves Factory or Super-Factory status ( L ~ 1033 -1036 cgs ), it will be able to operate repeatedly at that level. To achieve this goal we must concern ourselves with Efficient Operations Reliable Hardware and Accelerator Conditions Ability to Detect Faults and Recover A Toolbox of Instrumentation and Supporting Analysis Software (Since the Collider Must Have Beams to Operate), Rapid Injection 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

3. Modus Operandi for This Working Group • Drafted a list of questions under the preceding 5 areas • Discussed the questions • Shared experiences from different laboratories • Series of Talks • Michael Billing Operating experience with CESR • Eiji Kikutani Fill pattern control system of KEKB • Roger Erickson Operational reliability of PEP-II • Alan Fisher Developing the tune tracker for PEP-II • Uli Wienands Trickle (continuous) injection issues with PEP-II • Shigeki Fukuda KEKB-Linac upgrade plan using C-Band system for Super KEKB • Dmitry Teytelman Fault analysis for PEP-II RF • Uli Wienands Online lattice models and beam & Jim Turner measurements With RF & Fdbk Group 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

4. Operations: Scheduling Optimum HEP Data-Taking Time Function of Filling Time, Luminosity Lifetime Determined for Max Integrated Luminosity With slight variations all labs arrive same answer Typical Weekly Machine Studies Periods (shifts/week) • PEP-II: 0.5 (desire for 2-3) • KEKB: 2/2 weeks • CESR-C: 4-6 Regularly scheduled Access Periods (Downtimes) • PEP-II: scheduled only as needed • CESR-C: 1 shift/week • KEKB: 1-4 shifts/2 weeks 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

5. Operations: Personnel • Communications: Operations Meetings • PEP-II: 1 Program Meeting & 1 Fault Reporting Meeting/day • KEKB: 1 Program/Fault Reporting Meeting / day • CESR-C: 1 Program/Fault Reporting Meeting / week 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

6. Fault Recovery • Off-Hours Repairs • Control Room Personnel initiate &/or support repairs • Technical Personnel (e.g. Magnet technician) on-site at SLAC around the clock; but not at KEK or CESR-C • Specialists are On-Call at all times • Escalation Policy: • After accelerators down more than ~ 2 hours, higher level person is notified to coordinate the activities • Person notified is • Hardware Group Coordinator - PEP-II • Operations Deputy - KEKB • Director of Operations - CESR-C 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

7. Fault Recovery • Faults are Logged • All Labs have some form of Electronic Logbook • Some Labs use paper Logbooks also • Electronic Logbooks (e.g @ CESR-C) • Permit automatic mail distribution of logged fault entries • Semi-automatic logging of Beam Loss information • Includes x, y, z FFT’s of position signals • Recording of which element tripped First • Fault record file designations • Other comments • Semi-automatic characterization of conditions (KEKB & CESR-C) (<1 hour/week) • Tunes, orbits, dispersions, betatron phase & coupling, chromaticities, hardware status, et al • Incredibly useful when recovering from faults 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

8. Fault Recovery: Miscellaneous • Spares • Need to have parts on hand • May need to guess & buy a “lifetime supply” of parts • Buy extra spares for hard-to-find parts • Need to do last 2 more often ! • Safety Protection System • Need checking typically 2/year by regulation • Typical system checkout times (wide variation) • KEKB: 4 hours • CESR: 4 hours; CHESS: 4 hours • PEP-II: 6-8 shifts 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

9. Fault Prevention = Reliability • Importance of Record Keeping (Roger Erickson) • All labs account for time usage • For failures causing unscheduled downtime • Account system-by-system • Track Integrated Lost Time • Track Mean-Time-To-Failure • Track Mean-Time-To-Repair • Indicate systems needing attention • Reliability • Obvious problems dealt with • Repair of low priority items • Repairs not made -> hurt reliability or diagnostics??? • When do we upgrade OLD systems? - Cause headaches • Document MTTF, MTTR, Cost of Repairs, downtime to decide 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

10. Fault Detection • Avoid Information OVERLOAD - Many Faults Always Displayed • Make Prudent Choice: • Fix even the low priority problems OR • At least temporarily veto false indicators OR • Raise Trip/Warning level • Operator Training to detect cause of faults is important • Specialists still needed to analyze complex recorded failure records • MGB: Future, even more complex accelerators will find many subtle ways to fail, the accelerator community MUST find a method for automatic analysis of recorded waveforms. • At least automatically find the common failure modes • Could use techniques from HEP? 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

11. Instrumentation: Transient Fault Detection • Triggered counters (KEKB & CESR-C) • Multiple cards with multiple inputs • Detect delays between fault indications from many systems • Need to empirically determine hardware/cable delays • Identify correct source of failure large fraction of the time (but not always) • Transient Recorders (All labs) • ~10 MSamp/sec ADC • 1-several K samples in memory • Signals such as: • x, y positions & currents for beams • RF waveforms: Fwd, Rfl Power, Phases, Tuning Angle • Abort kicker or separator voltages • Feedback system: power CESR-C calls these “Chicken & Egg” cards 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

12. Instrumentation: Transient Recorders Dimitry Teytelman presented a number of analyzed transient signals from PEP-II (unfortunately unable to retrieve some to show) He hasrun several training sessions for operations personnel - teaches how to interpret waveforms - important for accurate diagnosing of problems CESR - Transient data => Horizontal separator trip 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

13. Instrumentation: Transient Recorders CESR Beam Loss record: W2 RF Waveguide Arc Trip N.B. Beam Induced voltage in field & reflected power 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

14. Instrumentation: Slow Transient Recoding • Experience at CESR-C: At times need to record for longer time • 100-200 signals at 10 Hz rate • Write data in memory to file on trigger • Can see vacuum burst propagation • Useful for slow transients 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

15. Instrumentation: Diagnostics for Conditions • Standard Toolbox: Measure • Orbits, tunes, dispersions, beta-functions (via changes in quadrupoles or phase measurements), coupling (global & local) • Feedback Systems Also Provide Diagnostics( John Fox) • Grow-dampmeasurements • DSP data recorder • Bunch-by-bunch:tunes, currents,etc. 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

16. Instrumentation: Diagnostics for Conditions • Beta Measurement from Phase Advance Data (Uli Wienands, Jim Turner) • Drive beam with shaker • Measure phase between BPM’s • Infer Betas • Method adaptable to local • coupling measure- • ments also • Able to repair conditions 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

17. Instrumentqtion: During Collision • Tune tracker (Alan Fisher) • Uses small excitation of non-colliding bunch • Lockin amplifier • Determine tune by phase characteristics -> • Can correct tune change during collisions Test perturbations Feedback on 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

18. Instrumentation: Philosophical View • Automating machine controls & adding feedback systems • Long term effort to improve operational behavior • Observational Toolbox + Analysis Toolbox • These Tools allow • Measuring & correcting accelerator errors • Discovering or confirming intensity dependent behavior • Measuring & improving luminosity • Determining & maintaining injection performance • Like any other tool in the workshop, these tools need maintenance - cost is time & effort e.g. • Calibrate BPM offsets • Test software group controls • Generate test cases for analysis tools • Test accelerator model 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

19. Injection • Routine injection mode is to Top-Off the beams (all labs) • Tuning to keep detector backgrounds low is critical • Tune Injector to keep good beam • Reduce any injection errors (x, y, E), use collimators • Orbits tend to drift • Could standardize magnets more often BUT • Tends to take a long time: • PEP-II (45min), KEB (12 min), CESR-C (4 min) • Could save separate orbit for injection • HEP Detectors are protected • Can reduce REP rate • Can dump beam • Over Filling protection • Dump beam if bunch significantly overfilled (PEP-II) • Stop filling if DCCT /= bunch current monitors (CESR-C) 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

20. Injection: KEK-B Fill Pattern Generation Eiji Kikutani • Method for filling a variety of bunch patterns • Requires 3 processors to communicate • “Uniform filling” & “top-off filling” (BCE) modes • Good human interface • Similar to those at PEP-II & CESR-C 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

21. Super-KEKB Linac Upgrade Shigeki Fukuda • Examined a number of upgrade paths • Some major changes could be • Raising e+ energy 3.5 -> 8 GeV • Adding damping ring(s) for smaller emittance • Increasing charge • Accelerate e+ & e- simultaneously • Major progress: • Designed 5.7 GHz RF structure • Procured RF source & modulator for tests • Successfully accelerated beam - 41 MV/m 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders

22. Continuous, “Trickle” Injection for PEP-II Uli Wienands • Top-Off bunches continuously • Low REP Rate (3Hz) • Small charge • BABAR • Counts triggers for EM Calorimeter • Provides “Beam Loss” • diagnostics • Gates off +/-300 nsec about injected bunch • Tests quite successful; already see 12% increase in int-Lum • Also has been tried at KEKB • Process works • Present current limit is beam induced heating 30th Advanced ICFA Beam Dynamics Workshop on High Luminosity e+ e- Colliders