CO deliverables for HWC – experiences and status after sector 45

1. CO deliverables for HWC –experiences and status after sector 45 Markus Zerlauth, AB-CO-MI for the CO group

2. Outline • HWC Sequencer • Post Mortem Analysis and event builder • Interlocks • Logging and Measurement Service (TIMBER and METER) • Powering Groups of Circuits (PGC) • Improving efficiency of HWC • Working environment – Organization • Circuit Synoptic • SOC naming • Coherency of procedures vs sequences vs MTF • Others • Conclusion

3. Sequencer - Sequences

4. Sequencer - Sequences • Other than in sector 78, first time large scale use of automated tools (sequencer, PMAnalysis, MTF upload,…) • Large commissioning program has been performed this year (>1700 tests run, 1160 successfully) • Proof of principle to perform battery testing of multiple circuits (>70 tests on 60A circuits run in parallel) • Decoupling of Sequence execution and PMA proved to be useful • Several improvements will be considered following feedback from sector 45, notably the exception handling, use of modified safe / MD modes to avoid unnecessary repetition of tests • GUI for ‚battery tests‘ will become more user friendly (for >10 tests launched in //) to allow for efficient throughput • Considerable modifications of the test plan will have to be implement upon request of MPP (mostly reduction of test steps) • Important to have approved and coherent written proceduresfor the next sector, as inconsitent naming of test steps, etc.. has created confusion in the CCC • Still tbd: Inner triplet sequences; ES not yet detailed enough for implementation

5. 3. PNO.2 auto analysis PM system and PM Analysis for HWC Courtesy of PM team

6. Post Mortem Analysis – past, present and future • Re-occurring issue data completness check & related look-up table + overall reliability of tool (migrate to LSA rather than local file) • Ongoing work • Next release will inlcude abort possibility for failed sequences, archive browsing, standardised comments in expert tools, further filter mechanisms • Running on operational machine to improve relaibility • Calculated parameters to be sent to Sequencer for MTF upload (property files) • More calculations (experts are still doing a lot with calculators or Excel macros) • More automatic Pass/Fail calculation for all test steps (expert override always possible), no 100% automation (risk of wrong decision and test repetition) -> user specification needed • QPS expert application and HWC tools available from the PM Browser • Implement merge of PM and Logging data in the applications • Browsing of SCEvents in the PM Browser Courtesy of PM team

7. Event Building for HWC • First prototyping has been done for HWC in collaboration with MA section, based on input of HCC (Mike, Rudiger, Alvaro) • In addition to existing classification, fully automated process will create SCEvent - Classification (Single Circuit Event) • Relating all equipment data belonging to this event (Layout DB) • Identifying the event sequence by retrieving history buffer of interlock systems (Logging DB) • (Pre-)analysis of event based on event sequence and PM data • Event summary for DB upload and first analysisof historic data (Mike) • Promising concept to be further extended and merged with beam PM?

8. PM infrastructure upgrade – Preparing for the LHC ( beam) PO/QPS gateway n FE servers … Single Proliant cs-ccr-pm1 BE server • Redundant and scalable services in CCR and TCR • Redundancy for client connections • Multiple FE servers for primary data storage, BE servers with large disks for complete data image • Transition ongoing now, to be completed before sector 56 • Currently single server single process on cs-ccr-pm1 • Single points of failures due to dependencies in CCR (network, upgrades, etc..) n FE servers … BE server Switch to services of new platform (half day stop) 5-Mar-2008

9. Powering Interlocks • New functionality implemented after sector 78 • Locking of circuits for e.g. nc circuits (new login for EIC and point owner will be provided) • 60A Power Permit (+60A_CRYO_MAINTAIN) • Interface tests (replacing PIC1) done ahead of HWC proven very useful • Few issues detected during (remaining) PIC1 & PIC2 • Polarity issue for one signal of 13kA circuits (understood and solved) • One replacement of an FGC module, one replacement of a current source in the PIC

10. As promised after sector 78, CO-DM has provided new data extraction interface for Logging and Measurement service Very good acceptance (no comments during sector 45?!) Remaining issue is the performance of the HW infrastructure (multiple services are still sharing same physical machine) LHC Logging and Measurement service

11. Evolution of the Logging – Data Rates  CIET  CRYO  QPS We are seriously hitting the disk I/O limits Courtesy of Ronny, Chris

12. 2008 Planned upgrade and migration CTRL CTRL CTRL CTRL 11.4TB usable 2 x quad-core 2.8GHz CPU 8GB RAM Clustered NAS shelf14x146GB FC disks Clustered NAS shelf14x300GB SATA disks Oracle RAC 2 Oracle RAC 3 Oracle RAC 1 LSA Settings DB Measurement DB Logging DB Controls DB HWC Measurement DB E-Logbook CESAR Switch to services of new platform (one day stop) 14-Mar-2008 (?) -> To be announced Courtesy of Ronny, Chris

13. PGC – Powering Groups of Circuits PGC 1: Synchronised ramps to given current levels; SOC driven, using LSA parameters and machine sequencer PGC 2: Synchronised ramps to given energy levels; Use nominal LHC optics; work through momentum, strengths and transfer functions; Looking to pull in realistic transfer functions from FIDEL LL5 Courtesy of Mike, Reyes, Antonio

14. PGC to given energy level Courtesy of Mike, Reyes

15. Improvements • PGC to fixed current level • Coherent values for all required circuits for a single parameter in POWER_CONVERTER_OP_INFO to drive settings generation • don’t worry about orbit correctors • PGC to given energy level • Coherent values for all required circuits for a single parameter in POWER_CONVERTER_OP_INFO so we can check and adjust if anything is out of limits • Some notice required for settings generation • Fixed targets please (2000 A, 4000 A, 5.5 TeV etc.) • Include advanced ramp-down (prepare next cycle) • Roll out FIDEL • Need PLP values on database and set every time Courtesy of Mike, Reyes

16. (Future) tools and ideas to further improve efficiency of HWC

17. Working environment and organization • Standardise the working environment when performing tests (sequencer, circuit synoptics, logbook) • Circuit Synoptics providing all relevant information to start and during the test (will become available on Linux for mid March) • Experts are available to do additional training on existing tools • Fronts should be working one circuit type at similar level of advancement

18. Use of Set of Circuits (SOCs) • Principle of SOCs (Set Of Circuits) is used by many applications (sequencer, FD, Circuit Synoptics, Powering Status pages) and thus a powerful tool • Can help to work more efficiently and help in organisation & daily planningif used with rigorosity • Proposal to use pre-defined SOC names throughout whole commissioning (respecting all known constraints for parallel powering) • S56-A56-RB, S56-A56-RQD/F • S56-A56-600A-F1 • S78-XL8-120A • S56-NC, S81-PO-TESTS, S56-A56-PGC • All circuits are attributed to only one SOC, EIC to manage these pre-defined SOCs • Planning of daily work, follow up and work attribution by SOC rather than circuit • Flexibility to create other SOCs, when working in parallel some naming convention is inevitable

19. Coherence of Procedures – Sequences - MTF • Inconsistent naming, ad-hoc changes, etc.. have created confusion in the past – vital to have coherency in between all three for tracability • Multiple efforts started, changes need to be implemented and approved for next sector • Due to frequent changes and inclusion of online-properties in xml file, MTF is considerably lagging behind and data upload from sector 45 only started yesterday • Test-plan will change once more after 45 (Persistency and quality of data in MTF? Offline data will be uploaded? Data extraction (tools) for MTF?) Courtesy of Alvaro, Mike, Antonio

20. Miscellaneous • Further test automation should be studied, e.g. allowing for automated re-arming of quench detectors and energy extraction switches (after a certain commissioning level) • Use similar scripts provided by QPS expert as for quench simulation for 600A circuits • Signing procedure will be slimmed down and automation increased following MPP, expert and operator input • Stringent and conscious expert signing of tests (no subsequent modification) • Many more detailed issues discussed in SACEC -> Action list • \\cern.ch\dfs\Users\z\zerlauth\Public\SACEC-2008-02-21

21. Conclusions and Outlook • Sector 45 has been a useful eperience to prepare tools for large scale commissioning • Lots of feedback has again been collected, many (priority) issues will be resolved for sector 56 • Multiple infrastructure upgrades will be ongoing in the next weeks to prepare for parallel HWC and first beam (service interruptions) • Confident that tools will allow for the requested througput in coming sectors • Information on changes and new features will be propagated to commisisoning teams to allow for maximum of independecy and efficiency of commissioning teams Thanks a lot - Questions?