prepared by R.A.Barlow (May 2014)

1. TE-ABT-EC Maintenance methods and policy in EC section prepared by R.A.Barlow (May 2014) TE-ABT-EC Maintenance methods and policy Edms nb.

2. Outline TE-ABT-EC Maintenance methods and policy • EC equipment location • Equipment failures • EC environment • Hardware • Software • Maintenance and procedures • Repairs • Documentation • Storage of components • Quality control • CONCLUSION

3. EC equipment location (expert/piquet equipment) Equipment Responsible and Deputy TE-ABT-EC Maintenance methods and policy

4. TE-ABT-EC Maintenance methods and policy

5. TE/ABTSpecialist BE/OPOperation BB BE/CO & EN/ICEControl Infrastructure TE/ABT/EC Electronics& Controls ABT EquipmentPS Complex, SPS & LHCKickers & Septa HV Generator Elements Electronic Modules Power electronics Cabling & associated hardware Software Controls PLC’s & Pxi systems Actuators & Sensors Power distribution Power supplies Control chassis TE-ABT-EC Maintenance methods and policy

6. The EC environment TE-ABT-EC Maintenance methods and policy

7. Electronic design data TE-ABT-EC Maintenance methods and policy • Electronic design is done in-house withthe schematics • PCB routing with exterior companies or TE-MPE-EM • Manufacturing produced outside or by TE-MPE-EM + assembly • AED/EDA numbering system for card versioning • If done by TE-MPE-EM then all documentation goes into EDMS under the EDA numbering scheme • If done by EC section then documentation is stored in EDMS and numbered with AED marking • Sometimes these are mirrored ! (WE wish to keep OUR ‘AED’ numbering system) • Each working manufacturing portfolio contains: Schematic, layout, gerber files, mechanical drawings, drillings file and serigraphy, BOM’s, test procedures, photo’s...

8. Equipment coding of modules TE-ABT-EC Maintenance methods and policy

9. Programmable Logic Controllers TE-ABT-EC Maintenance methods and policy • EC has a vast array of industrial automation in its controls park • PLC distributed systems are based aroundH/W design with associated software • Some chassis design requires custom made electronics coupled to PLC’s H/W • Each PLC chassis is created withtechnical drawing, cabling scheme, connector scheme (See_electrical) and software code (Step7) • In recent years industrial PXi systems (N.I) are becoming an additional resource to our supervisory control systems

10. Power elements/HV elements TE-ABT-EC Maintenance methods and policy • Many semi-conductor switches type (IGBT, GTO, Behlke) • Trigger systems in the medium power range • HV discrete components • HV generators (EC/FPS/SE) • Electro Mechanical elements

11. Power distribution systems TE-ABT-EC Maintenance methods and policy • EC responsible for the design of system power distribution • These are documented systems and undergo approval from HSE unit with outside companies such a DEKRA who come along for conformity testing and place approval seals on the systems. This includes Electrical circuit and Electrical protection checks. • We are currently looking at how to produce these elements on the outside using rigorous procedures and guidelines compatible with expected industry standards.

12. Software maintainability TE-ABT-EC Maintenance methods and policy • PLC codes (Ladder, SCL, STL, Graph7) + WinCC (scada) Stored onto DFS volume, currently looking at other systems for software maintainability (Autosave product) • FPGA ‘s codes (mainly VHDL, sim files, chipscope, etc....)/SVN • C++, Java, other/SVN • Hardware drivers/SVN • PXI systems software/SVN • Fesa equipment classes/SVN

13. Preventive maintenance TE-ABT-EC Maintenance methods and policy • Annual recalibration • Annual rotation of DCPS • Capacitor exchange (ex: Heater crates) • System consolidation • Timing system • Firmware upgrades • Interlock checks • Software release • Piquet weekly inspections of equipment where possible (ex: oil leaks, HV cable degradation, flasher not functioning, etc...) And many more! Preventive maintenance is difficult in areas with difficult access, no clear evaluation of possible cyclic maintenance, lack of data and analysis for overall systems. Also MTBF and MTTR are unknown quantities at the present (only LBDS LHC has undergone reliability studies)

14. Procedures TE-ABT-EC Maintenance methods and policy • Magnet conditioning procedures (SPS MKE4 & 6, MKP, MKD and LHC MKI) • ‘Consignation’ procedures • Electronic module installation procedures • LHC LBDS generator testing + energy scan • Test procedures for electronics card • Procedures for loading BETS energy tables et al ...

15. Procedures – movies ? • The requirements are: • ‘To create a multimedia support that shows and demonstrate some typical intervention carried during piquet time and outside of piquet time on TE/ABT equipment (Kickers & Septa) http://www.youtube.com/watch?v=AwugbEjSxAI TE-ABT-EC Maintenance methods and policy

16. Procedures – JIRA • “ Plan, track, work – smarter and faster ” • ABT-EC has started to use JIRA as an accomplished work and • planning management tool for the following actions: TE-ABT-EC Maintenance methods and policy

17. Interventions/Repairs TE-ABT-EC Maintenance methods and policy • Daytime requires -> Experts • Outside working hours -> Piquet/Experts These events can last 5mins (ex: Remote solving) up to extreme cases such as 48 hours or more (intervention in-situ involving longsome trouble shooting, special equipment handling, heavy maneuvering. Lots of specialists, etc.,,,) And occasionally a new spare part will be required !!!

18. Anatomy of a typical intervention CCC calls ‘online’ monitoring of faulty system Intervene remotely Time Materials Pressure Human factor Unknowns Technical expertise sharing Etc. ... Problem solved Log intervention Go to system location Controls, Hardware, Procedural ? Identify Fault Solve problem Stay around Leave premises Log intervention Call CCC TE-ABT-EC Maintenance methods and policy

19. Surface interventions vs tunnel interventions TE-ABT-EC Maintenance methods and policy Surface equipment interventions • Items and spares are located in-situ (storage cupboards, racks, Faraday cages) • Missing items have to be fetched from 865 in EC section AND other storage areas Tunnel equipment interventions • Items are invariably located in 865 or other storage areas, no spares in UA’s for LHC kicker equipment (except certain power elements)

20. Intervention examples TE-ABT-EC Maintenance methods and policy • LHC LBDS fault: The ratio between DVs1 [v] and DVs2 [v] was out of specification. A possible GTO failure inside the GTO stack. Decision taken to mask the interlock due to built in redundancy. Clearly a procedural intervention requiring permission from various sources and ultimately the chief LHC coordinator on that day • SPS MKE4 heater problem: after analysis of a thyratron related problem a decision was taken to replace the suspected damaged heater module This required locating a spare module, and the removal of the faulty module for future repairs • SPS MKD thyratron switch replacement: heavy maneuvering, specialists (not an evident one man job). This operation is a complex one, requiring transport, crane maneuvering, specialists, potentially dangerous and costly, re-conditioning of system (procedure) and follow up of faulty thyratron switch ‘The technical diversity of the systems under EC responsibility presents a challenge and requires high levels of expertise from individuals. All members of the team cannot always intervene on equal competence levels’

21. Written follow up of intervention TE-ABT-EC Maintenance methods and policy • Usually carried out in the logbook system, operation and EC logbook • Discussed at weekly ‘Piquet’ meetings • Actions follow-up are required

22. Storage of spare equipment TE-ABT-EC Maintenance methods and policy • PLC spare parts critical stock (en-ice-plc, b.864) • Electronic modules/cards (b.865) • Power supplies (b.865) + building • HV elements spares (b.865, b.867, in-situ) • Power distribution components (b.865) • Spares in-situ for surface controls (BA’s) (Electronic modules, power electronics, thyratron switches) • Spare part lists (Excel file) with components and spare location for each installation

23. TE-ABT-EC Maintenance methods and policy

24. Storage of spares at 865 TE-ABT-EC Maintenance methods and policy

25. Documentation types TE-ABT-EC Maintenance methods and policy • Electronic designs schematics, layouts, PCB masks, drilling drawings, mechanical features • Functional specifications and Engineering specifications, internal notes, papers • Minutes of meetings, summaries • Mechanical designs • System layouts, timing schemes, cabling schemes, procedures, stock lists • Many more .... !

26. Documentation Storage TE-ABT-EC Maintenance methods and policy • Much of previous list still stored on DFS volume (should be avoided), some in file holders/no electronic versions. • Ongoing guidelines still to be finalised for the EDMS framework.

27. Pre-installation quality control TE-ABT-EC Maintenance methods and policy • Electronic diagrams reviewed by third party • Surface and visual checks for H/W • Weekly rounds on surface points of equipment controls • Visual inspection of produced PCB’s • Electronic hardware test procedures

28. Recapitulative summary POOR, OK, GOOD, EXCELLENT TE-ABT-EC Maintenance methods and policy

29. To take away today TE-ABT-EC Maintenance methods and policy • Maintenance and methods is a vast complex subject ! • There are no all ‘encompassing’ solution • This leaves room for bigger ideas and concepts • What is OK today will not suffice in the future • There are maintenance methods systems, they are long and tedious (massive legacy of data entry) to put in place and require resources