300 likes | 453 Views
TE-MPE Stand-By Service 2010 – 2011 lessons learnt Gert Jan Coelingh on behalf of TE-MPE Stand-By team,. Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements?
E N D
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Conclusion & Outlook
Team Members • Reiner is responsible of the service, and Gert-Jan is the linkman • Stand-By members: In 2011, 8 members, with specific knowledge: • Giorgio D'Angelo EI ELQA & Heating Syst. C.L • Richard Mompo EI ELQA & Heating Syst. C.L • Grzegorz Seweryn EI ELQA & QPS Heating Syst. C.L • Steven Pemberton EI ELQA & Heating Syst. C.L • Kevin Priestnall CP QPS – Retirement 2011 • Vincent Froidbise CP QPS • Joaquim Mourao CP QPS • Gert-Jan Coelingh CP 600A EE, 13 kA & QPS • Help of Mathieu Favre, Noel Fournier and Bozhidar Panev (CP)
Team Members: Experts • Specialists: support for Stand-By members Very limited specialists (in number ): • Knud Dahlerup-Petersen (EE & QPS) • Reiner Denz (QPS) • Zinur Charifoulline (nQPS informatics). • Jens Steckert (nQPS). • CP section: Other specialists on Stand-By • Current leads specialist all specialists on Stand-By Now under EI responsibility;. • DFB specialists: are part of Stand-By members, Mateusz Bednarek since November 2010!! • MP3 members Stand-By • All Experts are on BEST-EFFORT
MPE Stand-By mandate: • 1. Mission: • To put back in service, as rapid as possible, a faulty installation within the framework (QPS, Energy Extraction Systems and electrical components of the DFBs) during LHC run. • 2. Actions: • Intervene after a call from the Control Centre CERN (CCC), • Diagnostics and if possible solve the problem remotely • Organize the intervention with the operator/EIC CCC (RP) • If needed ask the “Expert” (if reachable) for support • Archive the intervention in the Logbook (QPS Logbook). • 3. Responsibility: • He must assure his own security as well as those he asked helping him and assure the correct way of intervening in the tunnel • The technician is responsible for the intervention. If he cannot find a solution he must inform the group hierarchy.
MPE Expert mandate (1/2) The Expert (based on best effort) : • 1. Mission: • Support the first line intervention in case of complex problems • 2. Actions: • If judged intervene to help the Stand-By technician • Ask another Stand-By team member to join for intervention • 3. Indirectly related actions: • Decrease LHC machine down-time where possible • Implement solutions for problems • Firmware upgrades • Preventive maintenance • DQHDS main switch exchange campaign
MPE Expert mandate (2/2) • 3. Indirectly related actions (cont.): • Not to be underestimated in hours and workload • Example: Arkadiusz Gorzawski developed an interface logging database – 600A EE signals - permits preventive maintenance on breakers with too many closing failures - Currently extended with intervention history per system Direct result = no breaker exchange so far this year HUGE EFFORT not visible in Stand-By statistics
MPE Stand-By organization: • Documentation: • Each equipment specialist prepared a documentation on how to solve a typical problem. • An updated copy is always available in two Stand-By cars • Regular meeting and training: • Weekly meeting on Monday morning to transmit the “Piquet role” and share experience over the past week. • Training are organized in order to be as multidisciplinary as possible. • Two Stand-By cars available, equipped with tooling, docs and spare parts. • Storage place in Build.30, 281 and 936 (in front of CCC) • Yearly planning: On our repository, we have a yearly planning with foreseen vacations, official holidays, night and day “Stand-By” turn-over. • Day calls: transparent to CCC (same number 16.8801), assures intervention during normal hours. This allow night piquet to recover. • Night calls: outside normal hours and w-e, official holidays included.
Intervention Flowchart Emergency call from CCC to Stand-By duty: Stand-By: 16.8801 Information exchange via telephone Stand-By / CCC operator Remote Analysis Difficult problem or Time needed > 3 hours ? Y Problem Solved remotely? Stand-By / CCC Y Call equipment Specialist or Responsible TE/MPE N Fill in QPS Logbook N Stand-By person to analyze max. delay (1hour) End of intervention Repair and restart the equipment Repair and restart the equipment or, feedback situation with CCC. Need more than one Person? Y Assistance via: Stand-By support Operator CCC* N Fill in QPS Logbook End of intervention *(en dehors des heuresouvrablesuniquement )
TE-MPE Stand-By service for the LHCGert Jan Coelingh on behalf of TE-MPE Stand-By team,with inputs from R. Denz Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
Some statistics from 2010/2011: (1/..) • Total number of interventions (Remote / Local) and Total duration 219 hours 140 Interventions 87 Interventions 87 hours
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
SEU – increasing number • 17 treated by Stand-by team so far this year • Symptoms: QPS_OK lost (fill can be continued until beam dump) • continuous sending PM files or loss of communication • Action: • locally power cycle the crate • (and depending on time of intervention) • Update locally the firmware or • change board for one with new firmware) • Specialists actions: upgrade of firmware
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDSmain switch “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
DQHDS main switch • 18 cases treated by Stand-by team so far • 26 treated in 2010 – we are NOT better in number but much better in time of intervention; almost always IN THE SHADOW causing no downtime to LHC operation • 1650 (26%) out of 6200 have been replaced • All DQHDS of all MQs done and regular campaigns on-going • Software interlock adapted to MB. One out of four DQHDS of a MB can be down. No immediate intervention needed. • Why not better in numbers? … • plastic is getting too hard?? We should hurry up • No faults in the new switches
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
« IN THE SHADOW » • 30 out of 87 interventions “IN THE SHADOW” • less downtime for LHC operations • Mainly because • of upgraded Interlocking software in case of a MB DQHDS failure. • SEU can finish the fill before power cycling the crate • Preventive maintenance • Coordination between Equipment Responsible and Machine Coordination in case of longer downtime
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Conclusion & Outlook
Extrapolate 2011 – improvement? 189 interventions = +35% 189 hours = -14% +178% +170% +17% 0% - 41% - 21%
TE-MPE Stand-By Service 2010 – 2011 lessons learntGert Jan Coelingh on behalf of TE-MPE Stand-By team, Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? Outlook Conclusion
Outlook: • Improvement ongoing: • TUNNEL INTERVENTIONS • Upgrades on iQPSfirmware on SEU • Switches DQHDS replacement – 1600 done + 500 4-9 July • Software tools on QPS • Remote restart – done • Upgrade Firmware of new Busbar detector 4-9 July • Interlocks regarding DQHDS – interventions “in-the-shadow” • Potential bottlenecks: • More radiation to equipment SUE • Resources are limited, both experts and technicians • MP3 Stand-By canceled in 201x? Who takes over? • Back-up for Stand-By team based on best effort
Conclusion : • Conclusion so far as in 2010 • Very efficient: < 2 hours/intervention on site, including access time !! • High reliability of the systems • No accident ! • Stand-By service interventions seems to be decrease in numbers of hours • BUT.. • A lot of work behind the scenes to get there
MPE STAND-BY on DFS • All information about the Stand-By service can be found on the DFS folder: • Dfs:\\cern.ch\Departments\TE\Services\MPE_on-call-service
QUESTIONS ? THANK YOU !!! LPCT Local Power Cycle Tool The absence of safety shoes and helmet is not representative for the MPE Stand-By team
1) Routing of faulty pieces from a TE/MPE installation towards a repair workshop - - Who?: Equipment responsible When: Weekly, when lower limit is reached UA (Zone Machine Supervisée) Repair work shop TE/MPE Bat. 30 Bat. 281 Bat. 287 External Who?: MPE on call Technician Radioactif ? Quand?: Uniquement lors d’une intervention When?: Only in cases of an intervention Quand?: Uniquement lors d’une intervention ‘Picomur’ Zonede Transit TE/MPE for non radio active pieces Bat. xx Yes Broken Modules Rack Les piècessontcontrôlées par un technicien DGS/RP RR/UJ (Zone Opérationnellecontrôlée) Who?: MPE on call Technician ‘Picomur’ When?: Only in cases of an intervention Buffer zone for pieces from LHC machine Labo de Réparationdédié aux équipementsfaiblementradioactifs TE-EPC Who?: MPE on call Technician LHC Machine (Zone Opérationnelle contrôlée) When?: Only in cases of an intervention ‘Picomur’ • SécuritéRadiologiqueOpérationnelle: Lignesdirectricesgénérales • Tout matérielquittant les zones de rayonnement CERN doitêtreconsidérécommepotentiellementradioactif, • Tout matérieldoitêtrecontrôlé par DGS/RP, • SeulDGS/RP, estautorisé à déclarer du matérielcomme non radioactif, • En cas de doute, contactez un technicienDGS/RP