1 / 40

Machine protection update Rüdiger Schmidt MAC December 2006

Audit of the Beam Interlock System Prototyping interlock systems at CNGS: LHC type Beam Interlock System and Fast Magnet Current change Monitors Software and machine protection Software Interlocks Sequencing Post-Mortem  Critical Settings Management

len
Download Presentation

Machine protection update Rüdiger Schmidt MAC December 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Audit of the Beam Interlock System Prototyping interlock systems at CNGS: LHC type Beam Interlock System and Fast Magnet Current change Monitors Software and machine protection Software Interlocks Sequencing Post-Mortem  Critical Settings Management Compromise between the built-in flexibility of the MP system for the sake of optimum machine performance and efficient commissioning on one hand and the robust and the fail-proof design Machine protection update Rüdiger Schmidt MAC December 2006

  2. Safe operation of the LHC in presence of the energy stored in each beam of up to 362 MJ requires as a key system the Beam Interlock System Failures are detected by beam loss monitors and other beam instruments, by the quench protection system and other hardware related equipment. Beam dump requests are transmitted via the Beam Interlock System to the beam dumping system. There has been a review on the Machine Protection System. One recommendation was to organise more detailed reviews of the most critical systems (Beam Interlock System and Beam Dumping System). The architecture of the LHC machine protection is recalled. The functionality of the Beam Interlock System with respect to machine protection is presented. Main emphasis is on the safety critical aspects of the Beam Interlock System. This event is an audit of the Beam Interlock System. This is the last option for (limited) design changes, before starting series production of the electronics. Introduction to the Audit of the Beam Interlock System – September 2006

  3. Experts: Matthias Werner (DESY); Javier Serrano (AB-CO); Yves Thurel (AB-PO); Philippe Farthouat (PH); Reiner Denz (AT); Stefan Lueders (IT) Members of the interlock team: Bruno Puccio; Benjamin Todd; Rudiger Schmidt; Philippe Nouchi; Markus Zerlauth; Christophe Martin; Arend Dinius; Samir Hamnache Collaborators giving other presentations: Jorg Wenninger; Etienne Carlier Guests: Bernd Dehning; Jan Uythoven; Rossano Giachino; David Belohrad Group Leader: Hermann Schmickler Participants

  4. Organisation • Today (18/9/2006) • presentations • providing the experts with all required material (specifications, layout of the electronics, test results, ...) • showing the prototypes of the electronics • Second day 25/9/2006 • asking questions • discussions of findings • formulating recommendations • presenting the outcome • The week between 18/9 and 25/9 could be used to further understand and discuss details of the interlock system, if the experts consider this to be useful. • We might organise a demonstration of the operating system at the SPS

  5. Audit NOW since..... we have experience from CNGS (10-20% of LHC type hardware is operating) all electronic boards are in the final design phase Not the purpose to propose alternative ideas for interlocking the High Energy CERN accelerators The purpose is to identify critical parts, to understand the function of those parts, to validate their design or propose modifications split the audit in two parts, and leave some time for discussions / investigations in between Possible impact of the audit many comments might not have an impact on the hardware small HW mods: could possibly be still implemented larger HW mods: maybe only for LHC, possibly via an upgrade Some comments....

  6. Programme

  7. Do you consider the requirements for the Beam Interlock System adequately defined? Does the proposed realisation of the Beam Interlock System fulfill the requirements? Do you see any specific risks with the electronics? Do you see any failure modes that should be considered? Are the interfaces between the Beam Interlock Systems and the other systems clearly specified? Would you expect that the Beam Interlock System allows for safe operation (reliability)? Would you expect that the Beam Interlock System allows for efficient operation (availability)? Based on experience elsewhere: what is most critical and where have been surprises? Questions to the experts

  8. Beam Interlock System Audit Report on the Audit held in September 2006 Stefan Lüders (IT/CO) on behalf of the Auditors Reiner Denz (CERN AT/MEL), Philippe Farthouat (CERN PH/ATLAS), Stefan Lüders (CERN IT/CO), Javier Serrano (CERN AB/CO), Yves Thurel (CERN AB/PO), Matthias Werner (DESY)

  9. Scope • This audit is supposed to verify design and implementation of the BIS: • fundamental design decisions • PCB schematics and layouts & VHDL programming • mechanics • interfaces to other systems • Particular focus put on safety relevant aspects: • safe and efficient operation of LHC • sufficiently high reliability and availability • single points of failures AND failure modes leading to blind faults • This audit does not cover • system software running on PowerPC • control aspects & methods for remote diagnostics

  10. 25 cm 6 cm 41012 61012 21012 81012 One slide on the “Why” • 0.1 % of the full LHC beam • 81012protons • σx/y = 1.1mm/0.6mm

  11. CIBU The BIS Architecture • Three ring-type systems: • LHC Beam 1 & Beam 2 • SPS • Four tree-type systems: • LHC injection (Beam 1 & 2) • SPS extraction (BA4 & BA6)

  12. Auditor’s Recommendations • Recommendations have been distributed to all parties involved. • Focus on major points. • Numbers refer to Audit Report.

  13. General Impression • Design and implementation of the BIS is • sound, • complete, • straight-forward, and, • conform to requirement on high inherent level ofsafety, reliability and availability (SIL4) . • BIS as such makes a mature and solid impression. • Requirements have been adequately defined. • The present implementation fulfils completely the requirements.

  14. Documentation • Quite complete set of documentation on EDMS: • incl. drawings for PCB schematics, PCB layout and VHDL code • Additional documents on Bit-Error-Rate of optical link, resistance under EMC, detailed FMECA • Consistent set of up-to-date and finalized documents should be provided. • One of the main actors finishing his thesis soon: • 2. Ensure that all information relevant for the project is properly retrieved. Started: We have begun to update the existing documentation (EDMS). It will be completed in the next future. Done: An engineer has been appointed (former PhD student).

  15. Optical Loop I • Severe denial-of-service due to spurious signals on optical fiber link: • ELED driver amplified power supply ripples on the “high” state after the signal inversion, transmitted by the ELED, and amplified and shaped by the PIN diode receiver circuit. • 13. A solution should be found to avoid this behavior. • 14. The availability of the “old-fashioned” ELED seems to present a potential problem. Started: The Optical board is a small daughter board. We have launched a rigorous re-study of the current Optical board (CIBO). Several solutions are under study. We have checked with the distributor of ELEDs. These components will stay available for at least five years (confirmation received in writing). In case that the board would have to be changed later, this would be a minor investment.

  16. Optical Loop II • Two optical loops at 8.000 and 8.192 MHz: • 4. More separated frequencies like 8.750 and 9.375 MHz should be used. • For hardwired CCC interlock signals, a CIBU is too distant: • 15. Consistent and safe solution should be found for mitigation.The auditors prefer an additional BIC in the CCC. • Started: We have decided to change the frequencies of the 2 optical loops. We have received some quartz: • 37.5 MHz => 9.375 MHz permit loops • 35.5 MHz => 8.875 MHz permit loops • 33.5 MHz => 8.375 MHz permit loops • 31.5 MHz => 7.875 MHz permit loops • 29.5 MHz => 7.375 MHz permit loops • The choice will be done from two of the above values. Started: We intend to install a 17th BIC closed to the CCC.

  17. Testing & Environment • Careful functional testing is essential: • Electrical tests of all PCBs • 17. Power soak test duration should be justified and adjusted • 17. Accelerated thermal aging test of one system • 18. “Walkie-Talkie”-type or RF susceptibility test • BIS depends highly on proper electrical grounding: • 19. Conductivity of unit’s enclosure and earth connection of rack should be tested after installation. Done: We have designed new testers in order to perform electrical tests on the CIBE cards & on the CIBP cards. Will be done Started: We are going to carefully check the grounding and the earth connection of each BIS rack. In addition, we are going to systematically check every cable (with a dedicated cable tester).

  18. CIBUs have never been specified to be radiation tolerant: • 27. Radiation tolerance should be defined and verified. • 27. Persons responsible for BIS users must be made aware of the situation concerning radiation tolerance. Started: We are preparing radiation tests for the CIBU unit (foreseen in January 2007 at PSI). Relevant numbers and threshold will be communicated after the radiation tests.

  19. Components, Xilinx & VHDL • Recommendations have been made on choice of components: • 5. Extra power filters for the 230V mains • 9. Choice of ceramic capacitors • 10. Choice of bi-directional transil suppressor • Xilinx chip will block once the external clock is missing: • 21. Failure modes and corresponding mitigations should be checked. Power filters: better understand motivation from DESY colleagues for having extra filters Done: We have modified the part lists containing large values capacitors. Done: We have checked with the outside company that Transil from Fairchild won’t be mounted on our cards. Not yet started, but planned

  20. A number of questions came up during reviewing the VHDL code: • VHDL code review should be conducted when the final CIBM design is ready. • 23. Storage of VHDL code inside a software repository (e.g. CVS) Critical parts of the code will be written by two different programmers A full review of the VHDL code will be done (experts available inside AB-CO) Is being done

  21. Interfaces I • Substantial amount of effort has been put in high reliability and availability of the BIS: • Dependence on quite a long list of user systems and on proper functioning of the LDBS • LHC safety is only as good as its weakest element: • 35. Clear procedures for testing the full BEAM_PERMIT-signal chain should be defined. Started: We are preparing documents describing the BIS commissioning and the role of the User systems in the test of Beam_Permit chain. We started to address full automatic testing of the beam interlock system WITH the user systems by a software sequencer….. but not as highest priority for LHC initial operation. Beam Dumping System Timing System (Post Mortem) Injection Interlocks Safe Beam Flag Beam Interlock System Powering Interlock Systems FMCM Current 24 Beam Dumping System BLM controllers Access System Vacuum System 4 BPMs Software Interlocks Doors Beam Stoppers Vacuum valves Access Safety Blocks RF Stoppers BLMs (several 1000) QPS (several 1000) PCs (800) AUG UPS Cryo OK Magnets LHC Devices

  22. Interfaces II (Users) • LBDS kicker magnet system is essential for LHC safety: • 29. Similar audit should be conducted for LBDS kicker magnets and their trigger mechanism. • LHC safety relies largely on a small set of monitoring and control systems, e.g. Beam Lifetime Monitors, BLM, FMCM, Powering Interlock, Transverse Feedback System: • 31. Dependencies analysis with regard to LHC safety and of an audit of the major dependencies should be conducted. • 32. Procedures are mandatory to guarantee that BIS user systems obey standard safety rules. • 32. Awareness discussions & training • Other systems are thinking about organising audits… not the responsibility of the interlock team • (at least partial audits have been or are planned) • Being done – long term activity

  23. Interfaces II (Users) • 33. “Walkie-Talkie”-type or RF susceptibility test on (critical) BIS users • 34. Can SOFTWARE_PERMIT sustain high level of reliability and availability of the overall BIS ? Alternatives should be evaluated. Will be done together with the user systems Started: We intend to disable the so-called “Software input”. The Software Interlock System will be connected to the BIS via the standard way (i.e. the CIBU).

  24. Safe Beam Mode • SAFE_BEAM_FLAG allows masking half of the user inputs to the BIC: • No protection mechanism to prevent the exchange of the cables Done: Before the re-initialization of the Beam_Permit loop, we check “connected CIBUs” vs. “Configuration DB” => Swapped cables will be detected.

  25. The SAFE_BEAM_FLAG is distributed by the SMP / GMT: • Safe solutions for the implementation of the SLBR board should be investigated. • No documentation for the implementation and the SLBR. • 38. The distribution of the SAFE_BEAM_FLAG should be consistent with reliability / availability / safety of BIS. Fully agreed – the project is done in collaboration with another section. The distribution of the SafeLHCParameters is being developed. A Functional specification has been written. A local watch-dog will be implemented on the SLBR board. In case of missing data, the Safe_Beam_Flag will be forced to FALSE. In addition, a software monitoring could be implemented for checking this (slow) process.

  26. Summary • Design and implementation of the BIS is sound, complete, straight-forward, and, conform to requirement on high inherent level ofsafety, reliability and availability (SIL4) . • However: • To keep reliability high, functional testing on regular basis is vital. • Worried about the behavior of the optical link electronics(esp. the ELED and ELED driver circuit). • VHDL code should be reviewed separately. • Further electrical and RF susceptibility tests should be conducted. • Concern about the safety/reliability/availability of BLM and kicker system of the LBDS (separate systematic audit / review should be conducted on them). • Finalize documentation.

  27. Some comments – after the audit • Not everyone was convinced that such an audit at this stage makes sense. • All members of the interlock team fully supported it. • The time that we (..the interlock team) invested in the preparation was a few days – not more. • We were pretty sure that we proposed a solid system, and therefore we did not expect findings that would create great difficulties, however, if the experts would have found a major problem, we rather like to know now. • I see many reasons for having an audit, and the (only?) reason of not having it is lack of time for the preparation (I would have been nervous without the audit - I cannot judge all detailed aspects of the system). • The findings at the evening of the first day would have been moderately useful….. but due to the intense work of the experts (many many hours between the first day and the second day a week later), I consider the audit to be very useful for the project. I like to thank the experts for their help

  28. The beam energy extracted from SPS could damage equipment, stringent machine protection is required Beam interlock system, about 10-20% of LHC system, worked successfully Fast Magnet Current change Monitors: few (HERA) systems installed measuring small current change of magnets within less than 1 ms worked very well, performance better than required no false beam dumps upgraded system: prototype unit currently tested at DESY (M.Werner) Software Interlock System became operational during CNGS run Experience from the CNGS run

  29. FMCM • 3 FMCMs are installed on MSE.418, MBSG.4000 and MBG dipole string. • Tested using steep reference changes to trigger FMCM. The trigger threshold is measured using BIC history buffer + read out of current (1 ms step). I (A) I (A) Reference PC current FMCM trigger  0.1% drop ! SC time (ms) J.Wenninger, MPWG talk SC time (ms)

  30. FMCM • 3 FMCMs are installed on MSE.418, MBSG.4000 and MBG dipole string. • Tested using steep reference changes to trigger FMCM. The trigger threshold is measured using BIC history buffer + read out of current (1 ms step). I (A) I (A) Reference PC current FMCM trigger  0.1% drop ! SC time (ms) SC time (ms) J.Wenninger, MPWG talk

  31. FMCM beam tests The tests were repeated with beam by adjusting the step and FMCM trigger to occur ~ nominal extraction time  record max. trajectory excursion – confirms current measurements. In all cases the position change @ target is < 0.5 mm (as specified). • Example : • Max. trajectory excursion due to MSE @ time of FMCM trigger. • Trajectory interlock tolerances : • ±4 mm BPMs no. 1 to 20. • ±0.5 mm last 3 BPMs. Triggers a BPM interlock J.Wenninger, MPWG talk

  32. Summary • The machine protection system for CNGS with 4 BICs and 32 clients was commissioned in ~ 50-60 hours. This time only includes the specified tests (mostly from CCC). It does not include the hardware tests to connect the clients… • All documentations and information for operation is available on a WEB page. • No major problems were encountered during the commissioning and the transition to high intensity operation was smooth. • Almost all interlock systems performed very well, with special mention for the powering interlocks (current surveillance & FMCM) that were quite outstanding ! • The only problem was due to the beam position interlock in the SPS ring – this became a major issue due to the simultaneous out-gassing problems of the SPS beam dump. • I think we must consider installing a more robust (no gains) and faster system, even at the price of a reduced accuracy. • The SafeBeamFlag was introduced for the first time with success. • The next challenge will be operation with CNGS cycles in a super-cycle in October. This is going to be an interesting problem of settings management ! • If the beam dump continues to outgas, operation may be VERY delicate! J.Wenninger, MPWG talk

  33. Post Mortem Required for commissioning of electrical circuits in March 2007 Beam related post mortem being addressed - workshop is being organised in January 2007: http://indico.cern.ch/conferenceDisplay.py?confId=9340 Sequencer Sequencer for hardware commissioning operational Sequencer for beam commissioning in progress Possible merge of the both sequencers Management of Critical settings: good progress, should become operational during first part 2007 Role based access project started (restricting access to some applications) Beam instruments – no news on Fast Beam Current change Monitor, but not required for 2007 MP Commissioning: Working Group started, reporting to LHCCWG (summary report possibly for next MAC) News on other issues

  34. The beam interlock system receives (nearly) all inputs for permitting beam operation Disabling on the level of the beam interlock system is in principle possible, but requires NO BEAM and ACCESS to the hardware disabling a channel is a heavy procedure Some input channels are MASKABLE - only during low intensity beam masking is a light procedure, but requires non-damaging beam in LHC very successfully used for CNGS Many input channels will be commissioned during “hardware commissioning” and “cold checkout”, to be operational for first beam Work ongoing: one accelerator physicist (PhD) studying failure scenarios one reliability engineerr studying consequences for interlock systems The following slides are for illustration of the flexibility, and NOT to define the final phases for the commissioning of the interlocks Flexibility of Machine Protection during commissioning

  35. Beam Interlock Tree maskable and unmaskable inputs Beam Dumping System 2 Timing System (Post Mortem) 1 or 2 Injection Interlocks 2 Safe Beam Flag preliminary Beam Interlock System PIC essential 32 PIC essential + auxiliary 32 WIC 14 FMCM Current 24 BLM aperture limits 16 BLM in arcs 16 Access System 2 Vacuum System 34 Beam Dumping System 2 BPM in IR6 2 BLMs (several 1000) BLMs (some 100) Magnets Power Converters Doors EIS Vacuum valves Access Safety Blocks RF Stoppers Cryo OK QPS (several 1000) Power Converters 864 AUG UPS QPS (several 1000) Power Converters 730 AUG UPS Cryo OK Beam Interlock System Software Interlocks 2 CCC Buttons 2 SLP 2 Experiments 26 Transverse Feedback 2 RF System 4 Beam Aperture Kickers 2 Collimation System 42 FBCM Lifetime 2 BTV 9 LHC Devices Movable Devices BCM BeamLoss Experimental Magnets Collimator Positions Environmental parameters BTV screens Mirrors to be defined

  36. Beam Interlock TreePhase I after Hardware Commissioning Beam Dumping System 2 Timing System (Post Mortem) 1 or 2 Injection Interlocks 2 Safe Beam Flag preliminary Beam Interlock System PIC essential 32 PIC essential + auxiliary 32 WIC 14 FMCM Current 24 BLM aperture limits 16 BLM in arcs 16 Access System 2 Vacuum System 34 Beam Dumping System 2 BPM in IR6 2 BLMs (several 1000) BLMs (some 100) Magnets Power Converters Doors EIS Vacuum valves Access Safety Blocks RF Stoppers Cryo OK QPS (several 1000) Power Converters 864 AUG UPS QPS (several 1000) Power Converters 730 AUG UPS Cryo OK Beam Interlock System Software Interlocks 2 CCC Buttons 2 SLP 2 Experiments 26 Transverse Feedback 2 RF System 4 Beam Aperture Kickers 2 Collimation System 42 FBCM Lifetime 2 BTV 9 LHC Devices Movable Devices BCM BeamLoss Experimental Magnets Collimator Positions Environmental parameters BTV screens Mirrors to be defined

  37. Beam Interlock TreePhase II aftercold checkout Beam Dumping System 2 Timing System (Post Mortem) 1 or 2 Injection Interlocks 2 Safe Beam Flag preliminary Beam Interlock System PIC essential 32 PIC essential + auxiliary 32 WIC 14 FMCM Current 24 BLM aperture limits 16 BLM in arcs 16 Access System 2 Vacuum System 34 Beam Dumping System 2 BPM in IR6 2 BLMs (several 1000) BLMs (some 100) Magnets Power Converters Doors EIS Vacuum valves Access Safety Blocks RF Stoppers Cryo OK QPS (several 1000) Power Converters 864 AUG UPS QPS (several 1000) Power Converters 730 AUG UPS Cryo OK Beam Interlock System Software Interlocks 2 CCC Buttons 2 SLP 2 Experiments 26 Transverse Feedback 2 RF System 4 Beam Aperture Kickers 2 Collimation System 42 FBCM Lifetime 2 BTV 9 LHC Devices Movable Devices BCM BeamLoss Experimental Magnets Collimator Positions Environmental parameters BTV screens Mirrors to be defined

  38. Beam Interlock TreePhase III during very first beam Beam Dumping System 2 Timing System (Post Mortem) 1 or 2 Injection Interlocks 2 Safe Beam Flag preliminary Beam Interlock System PIC essential 32 PIC essential + auxiliary 32 WIC 14 FMCM Current 24 BLM aperture limits 16 BLM in arcs 16 Access System 2 Vacuum System 34 Beam Dumping System 2 BPM in IR6 2 BLMs (several 1000) BLMs (some 100) Magnets Power Converters Doors EIS Vacuum valves Access Safety Blocks RF Stoppers Cryo OK QPS (several 1000) Power Converters 864 AUG UPS QPS (several 1000) Power Converters 730 AUG UPS Cryo OK Beam Interlock System Software Interlocks 2 CCC Buttons 2 SLP 2 Experiments 26 Transverse Feedback 2 RF System 4 Beam Aperture Kickers 2 Collimation System 42 FBCM Lifetime 2 BTV 9 LHC Devices Movable Devices BCM BeamLoss Experimental Magnets Collimator Positions Environmental parameters BTV screens Mirrors to be defined

  39. Beam Interlock TreePhase IV during early beam commissioning Beam Dumping System 2 Timing System (Post Mortem) 1 or 2 Injection Interlocks 2 Safe Beam Flag preliminary Beam Interlock System PIC essential 32 PIC essential + auxiliary 32 WIC 14 FMCM Current 24 BLM aperture limits 16 BLM in arcs 16 Access System 2 Vacuum System 34 Beam Dumping System 2 BPM in IR6 2 BLMs (several 1000) BLMs (some 100) Magnets Power Converters Doors EIS Vacuum valves Access Safety Blocks RF Stoppers Cryo OK QPS (several 1000) Power Converters 864 AUG UPS QPS (several 1000) Power Converters 730 AUG UPS Cryo OK Beam Interlock System Software Interlocks 2 CCC Buttons 2 SLP 2 Experiments 26 Transverse Feedback 2 RF System 4 Beam Aperture Kickers 2 Collimation System 42 FBCM Lifetime 2 BTV 9 LHC Devices Movable Devices BCM BeamLoss Experimental Magnets Collimator Positions Environmental parameters BTV screens Mirrors to be defined

More Related