QPS: "What was learned from HWC"

1. QPS: "What was learned from HWC" MOPS training lesson 6 R. Denz AT-MEI

2. Outline • Present state of the QPS system and planned evolution of the system • Problems observed during hardware commissioning • HC specific problems • Non HC related problems • QPS system faults • QPS faults caused by other systems • QPS controls • Interventions into LHC • Regular maintenance • Interventions in case of non-conform behaviour • QPS piquet • Conclusions QPS web: https://at-mel-pm.web.cern.ch/at-mel-pm/ QPS logbook: http://elogbook.cern.ch/eLogbook/eLogbook.jsp?lgbk=340

3. State of QPS • Present state of the QPS system • Having survived HC, QPS is now basically ready for operation • Transition phase between commissioning and operation • Some issues left over from HC, e.g. circuits not fully commissioned, wrong configurations etc. • Firmware upgrades for some type of controllers • Issues to be closed by the end of the month • Short term evolution • Installation of Remote Reset Units • Rack supervision, power cycle of “lost” systems without access • Upgrade to 7 TeV operation • Following the outcome of HC commissioning and the expected amount of training quenches required to reach 7 TeV the QPS protection system needs to be upgraded for better detection of symmetric quenches • ECR to be issued soon – some more testing required (this week)

4. QPS – HC specific problems • Polarities, layouts, cabling, settings, filters, algorithms etc • Basically all sorted out but proper documentation still to be done • Normally not to be expected to affect LHC exploitation • A few problems still to be fixed • Circuits not yet fully commissioned • Current limit for MCBXH/V powering • Spares management • Updates may be required in case of • Obsolete components, systems, tools (e.g. compilers) • Change of requirements by beam physics (e.g. 600 A circuits) • Increasing luminosity of the LHC – radiation effects

5. QPS system faults - introduction • QPS and EE systems comprise • 6200 quench heater power supplies (100 % needed for pilot beam) • 6400 quench detection systems (97.5% needed for pilot beam) • 32 13 kA energy extraction systems (100 % needed for pilot beam) • 202 600 A energy extraction systems (80 % needed for pilot beam) • 2200 fieldbus couplers (99 % needed for pilot beam) • System reliability • Most of the systems have already reached the bottom of the famous bathtub curve • Still some teething problems • Transition phase from commissioning to exploitation

6. QPS system faults I • Quench heater circuits • To be checked after every trigger / discharge • Powering must be stopped in case of doubt as repeated firing of damaged heaters may affect magnet integrity • Quench heater power supplies • Slow discharge  ST_POWER_PERM (QPS power permit) removed if below 810 V  typically main switch is broken  repair in situ possible • Power supply does not re-charge correctly after a trigger  no QPS power permit  power supply must be replaced

7. QPS system faults II • Detection systems in general • All systems are fully redundant • Loss of communication normally does not affect protection • Detectors are to a very large extent “fail safe”, i.e. faults will activate the interlocks • Main magnet protection systems • MB, MQ protection systems normally very stable • Somewhat higher fault rate in the not so dry part of the machine (sector 3-4) • IPQ, IPD and IT protection systems • No particular faults after HC revealed • 600 A circuit protection • Drifts of detection systems (magnets only)  RESET prior to run • New controller firmware generates regular automatic software resets (only if |I| < 5 A) • Most sensitive systems with respect to noise (EMC)

8. QPS faults related to or caused by other systems • AC powering: power cuts • Several problems observed with UPS system (10 minutes autonomy guaranteed in case of mains loss) • The weak point seems to be the link between the UPS and the client • Power cuts affecting QPS will in any case activate the interlocks and the energy extraction systems • The quench heaters are not fired in case of a total loss of power  unprotected magnet during discharge of the circuit • Re-start of QPS after a power cut may require expert intervention • AC powering: over-voltages other than spikes • One serious event observed in RE88 (alcove)  damage to equipment • Cooling and ventilation • Loss of cooling water  13 kA EE systems only • Humidity in sector 3-4

9. QPS control system I • QPS users and their access rights • Monitor: can observe only • Operator: can access all commands required for LHC operation • Expert: can access all commands – supposed to know what she or he is doing but human error is never excluded • Access to QPS controls only via technical network TN • i.e. terminal server, CCC console, trusted machine on TN • two LOGIN required from home (still possible) • Basic applications • QPS supervision (PVSS based) • QPS expert console • PM viewer • TIMBER (logging data)

10. QPS control system II • Critical commands and actions • RESET • Software reset of a QPS controller and the associated protection systems • Does not fire heaters, interrupts transmission of PM data, may trip powered 600 A circuits • Macro for sending a RESET command prior to start of ramp • TEST MODE • Generates an artificial quench signal • Required for test purposes only – not needed for operation • Operation of Remote Reset Units (not yet installed) • So far only attached to 600 A systems (other systems in preparation) • Accidental activation will stop powering of corresponding circuits

11. QPS control system III • Communications problems – various reasons • QPS controller: only a few circuits related to that controller affected • Power cuts: a group of controllers also of different type is affected • Fieldbus infrastructure (repeaters, cabling etc.) many circuits affected • Gateways (frontend computers): all circuits linked to one or two fieldbus segment affected • RESET of QPS gateways • Only if OK from EIC and QPS expert

12. Interventions into LHC I • General remarks • All QPS and EE equipment is installed inside controlled zones (exception IT protection .L5 I USC55) • Clearance by RP required • Some QPS spare parts, e.g. quench heater power supplies already stored in LHC • MAD only required for material transport and bicycles (access to tunnel and point 7) • Bicycles and trailers soon to be parked in LHC • Regular maintenance • Exchange of broken systems not affecting LHC operation • System updates • According to LHC schedule • R&D ongoing to minimise interventions into the LHC

13. Interventions into LHC II • Interventions in case of non-conform behaviour affecting LHC operation • Exchange of equipment, re-start, reset etc. • Only by QPS experts or explicitly authorised staff • Interventions to be coordinated with LHC operation • Piquet services • General AT-MEI piquet for ELQA, current leads, QPS and EE issues • Team currently being setup: 1st line 162266 2nd line 163162 (not yet operational) • Training of the team to be started soon • Specialist support on best effort

14. Conclusions • QPS and EE systems are built to protect LHC superconducting circuits – second priority is to protect the beam • Triggers of the system are normally not related to faults • Most of QPS and EE systems required to inject pilot beam – all are required for nominal beam • QPS and EE systems are to a very large extent “fail safe” • Faults will prohibit LHC operation but should not cause any further damage • LHC operation just started • Challenging time for all concerned systems, QPS and EE systems are no exception