WELCOME to the MPP Workshop 2013 in Annecy!

1. WELCOME to the MPP Workshop 2013 in Annecy!

2. Past reviews • MP related reviews that were done in the past • External Machine Protection review (June 05 - overall system) • BIC audit summary (Sept 06) • BIC audit sequel summary (June 09) • LBDS audit summary (Feb 07) • LBDS audit sequel summary (June 09) • BLM audit summary (July 08) • BLM audit sequel summary (June 09) • MPS internal review (June 2010) • Collimation system (several reviews) • External LHC Machine Protection Review (September 2010 - are we ready to go beyond 30 MJ?) • Machine Protection Workshop for linear accelerators (June 2012) • Machine Protection Workshop (March 2013)

3. Mandate • Discuss mid-and longer-term improvementsof the MP systems for the LHC + injector complex: • review of the current operational experience with MP systems during the first running period (2009-2013). • understanding the planned changes of MP equipment during LS1 and the consequences/potential limitations for operation after LS1. • identify areas where improvements are required. • ensuring coherence between the different MP systems. • identify misses.

4. Program • IndicoPage • 6 sessions, 2 ½ days, ~ 85 participants • MPS experience (2008-2012) and outlook (D.Wollmann) • Injection and LHC beam dumping system (J.Uythoven) • Beam Diagnostics (B.Dehning) • Collimation and movable devices (S.Redaelli) • Electrical circuit related protection (M.Zerlauth) • Operation after LS1 (J.Wenninger) • Summary and discussion (R.Schmidt) • Executive summary @ CERN in special LMC (date tbc)

5. A bit of general info first…. No (more) coffee break on MON morning! Free WiFi in the whole hotel, consume with moderation Lunch scheduled for 12.30 in the restaurant (seated) Rooms will be ready around 16.00 In case of any problems, please contact Mariane (+41 76 487 87 53)

6. 3 years of MPS in short • Organization and issues • MP protection during commissioning and production • Non-standard operation • Conclusions

7. 3 years of LHC Machine Protection in short 2011 2012 2010 Courtesy B. Todd Courtsey of B.Todd – Evian 2012 • Some 3499 clean beam dumps performed since Oct 2009: • 1582 beam dumps above 450GeV. • In 2012 a majority of dumps with beam energies > 100MJ (reaching max 146MJ). • No beam induced magnet quenches @3.5/4.0TeV. • No serious equipment damaged due to beam (excl. heating, corrector coils of RQX.3L2, ALICE detector). • Reasons and MPS response reasonably well documented, analyzed and validated for all beam dumps > 450GeV.  More details see talk from B. Todd

8. In the end there was reason to be content… >23 fb-1

9. Driving along the edge throughout… • LBDS 12V • Quench detection issues on IPQ, 600A EE • TL collimators • Injecting H9 • False collimator settings (2 x TCTV IR2, 2x IR3) • Roman Pot Controls • BLM High Voltage Cable • OFSU reference problems • BSRT Mirror • MDs and other non-standard machine operation • MKI flashovers • QFB not usable in squeeze due to poor signal • Instrumentation problem in triplet L8 after TS2 • Loss of redundant protection (60A power permits, LHCb dipole , CMS solenoid,…) • ….

10. Mandate and Organization of MP Work • Assure (potentially) dangerous situations are recognized, documented and distributed and actions followed up throughout commissioning and production • MPP(in present form since Sept 2008) for scientific ground work and preparing decisions – ties and certain overlap with COLL WG, LIBD, LBOC, AWG,.. • Will need to increasingly look into HL-LHC and injectors as well • rMPPadded during production run (Sep 2010) to follow up and decide on ‘urgent’ issues and non-standard operation • Very good for ad-hoc discussions (outside large bodies), sometimes had to decide in absence of few key members -> Still need a common line from rMPP • Need for (more) regular meetings? Update of rMPP membership? • Follow-up & responsibility of recommendations/ procedures less obvious (MD documents, checklists,….) • Interplay of MPP and MP3 for performance of magnet powering system to be defined/improved for post LS1?

11. Examples of issues and how we dealt with them • Safety critical issues in MPS systems • Inversion of HTS instrumentation cable on RB.A45found following CL quench on 7th April 2011 due to cooling problem on DFBAI.L5 • Stopped powering for verification of all >6kA circuits for 1.5 days • Common 12V dependency in LBDS system: Found 8th of June 2012 during Lab test (follow up of LBDS review on redundant powering) • LHC coordinator and rMPP informed, ongoing fill dumped + operation stopped for some 5 hours for implementation of watch-dog

12. Examples of issues and how we dealt with them First indication in Pre-op Check Cell 12-34 • Wrong TL-TCDI settings for SPS Q20 optics found on 19th of November 2012 (up to 1.3 σ) • CCC informed and operation stopped for 12 hours to recalculate + revalidate • Missing BLM HV noticed by sanity check on 12th of April 2011 • System reconfigured and validated (using supply and spare cable from LSS), later SIS interlock added

13. Examples of issues and how we dealt with them Missing Orbit reference • Issues in MPS systems resulting in known (and protected) failure cases • E.g. non redundant LBDS powering, MKI flash-over, injection of H9,… • Operation in general continued, further minimizing risk through longer term mitigations • Issues with non-MPS systems with certain potential to become unsafe • E.g. OFSU without reference, BSRA, RP controls • Operation in general continued straight away, short-term mitigations put in place to further minimize risk, e.g. postponed next intensity step, new SIS interlocks and additional sequencer and operator checks

14. Would a MP ‘responsible’ be beneficial? • Major issues were mostly ‘dormant’ failures and appropriate action was taken immediately • Lessons should be learned and commissioning procedures tightened up • Some issues persisted for some time & reaction might vary as function of individuals knowledgeable about it • Double role of EIC/Machine Coordinator to optimize performance and assure protection not always easy to combine • To increase and facilitate support to operation crews: Role of an ‘MP responsible / week’ for post LS1? • Contact (representing rMPP) for operation crews in case of doubts and non-standard operation • Assure necessary documentation, follow-upof operational issues, e.g. analysis of dumps,... • Follow-up of MPS commissioning, operational changes + necessary revalidations

15. Documentation of MPS tests and issues - Commissioning See talk of K.Fuchsberger • Very rigorous and thorough approach to preparation, execution & documentation • Required time was in general allocated /available • Commissioning procedures need to be updated and revised during LS1 • Profit to organize execute as well ‘difficult’ tests (redundant powering,…) • MPS site worked fine – driven by few individuals in parallel to coordination, progress & status of advancement not obvious • Use ACCTEST for more dependable + automatic sequencing, dependency tracking + follow-up

16. From commissioning towards production • Tracking of (relevant) system changes • More coherent approach for follow-up of magnet & beam related MPS issues • Define & enforce minimum (non-negotiable) validation costs of changes, e.g. new setting validation with loss-maps & asynch dumps, critical interventions,… -> Takes ‘responsibility’ off the piquet teams, operation crews, coordinators,… • Automatic tools/dependency models could help to enforce this for well known cases • MP responsible for remaining cases Commissioning mindset should be maintained throughout full running period, suffered towards end increasingly due to scheduling pressure, routine (we have done this before without problem), relatively smooth running, fatigue…. Initial (commissioned) safety level certainly degrades under such circumstances Possibilities to counter-act?

17. Change Management • MPS HW and SW changes tracked during past 2 years in MPS commissioning SharePoint Site, however… • We have to clearly define what is mandatory to be tracked • Requires corresponding culture and level of responsibility of experts (many changes not even known/advertised!) • Even more so for SW deliverables -> Culture of best practice (Good design, Unit Testing, Integration Testing , Functional Testing) • Could a commissioning tool like ACCTEST (in conjunction with a dependency management) be re-used to enforce re/validation process after changes? • Concentrate on few most critical & reoccurring items first, like powering system, settings validations, …

18. During production running • Considered very useful by MP experts, should this be distributed to anyone else? • Regular reflection on past issues, comparison with own fault tracking and reminder for follow-up • Problem of ‘self-validation’ could be improved by true approval circuit • Should be enforced on more regular basis (e.g. did none during Ion running) • Post Mortem DB with operator and MPS expert comments(works well for dumps above injection energy, how to improve for dumps @ injection?!) • Can and need to do more automated validation checks (pre and post-op, IQC, XPOC…), difficulty is vast failure space and too frequent false positives • Regular distribution of MPS check lists during intensity ramp-ups and operation to MP system experts with all dumps above injection energy for comments

19. Machine Developments 1/2 • Machine Developments per definition explore new machine and machine protection territory and hence risks! • MP documents did allow to address and mitigate risks – did preparation phase increase as well the efficiency of the MDs? • Un-intentional losses of beams as (one possible) indicator for ‘efficiency’, e.g. hidden interlocks during change of SBF, (not counting machine unavailability, equipment failures,…) • With detailed MP MD doc ~ 0.38 unintentional dumps/MD • Without doc ~ 0.67 unintentional dumps/MD Courtesy of D.Wollmann

20. Machine Developments 2/2 • Additionalfactorsinfluencing ’efficiency’ • Increasingly flexible plannings towards the end, manydocuments done ‘last minute’, little time for comments and iterations • Operations not sufficientlyawareof MD details and requested machine preparations • Proposal for post LS1: • MD – 4 weeks: • For all MDs up-to-date MD request document (parameter space, interlocks, program..) precondition for being scheduled • MD -2 weeks: • Draft procedure for MD Class C • True EDMS approval circuit, all rMPP members to sign • MD: • EDMS approval precondition to start • Simple additional SIS interlocks to enforce defined parameter space ?! • OP shifters responsible to assure correct execution of MD and eventual checklists

21. Summary LHC Machine Protection Systems have been working well during run 1 Noevidence of major loopholes or uncovered risksin MPS architecture, however a few near-misses have revealed short-comings in designs, commissioning procedures and operation of MPS systems Organization and response of coordinators, OP, (r)MPP, equipment experts to these problems was adequate As a result of many different factors some of the initial rigor appeared to be lost at all levels towards to end of the run Could an MP responsible be an additional asset for operations and help counteracting some of the issues observed? It might become increasingly challenging to maintain systems at their needed level of dependability (additional operational challenges post LS1, allocated resources and time,..)

22. Fin