1 / 29

CO Support proposal for 2014

CO Support proposal for 2014. Marine Pace Technical Committee -12 Dec 2013. Context. Detailed proposal for the organisation and the process of the CO support in 2014 Main focus on PS Complex as subject to major controls changes & to a new support model.

cree
Download Presentation

CO Support proposal for 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CO Support proposal for 2014 Marine Pace Technical Committee -12 Dec 2013

  2. Context Detailed proposal for the organisation and the process of the CO support in 2014 Main focus on PS Complex as subject to major controls changes & to a new support model

  3. Recap of phases from Eugenia’s presentation COMMISSIONING & EARLY BEAM OPERATION STABLE OPERATION DRY RUNS

  4. CO Support phaseswith respect to re-commissioning schedule 2014 (Latest Official schedule still being discussed EN/GS/OP) CO is involved in many DRY RUNS with EQP GP + OP Aim : TEST as much as possible the controls infrastructure HW TEST period : PHASE 1 : CO DRY RUNS -> all experts required for the dry run will be present

  5. CO Support phaseswith respect to re-commissioning schedule 2014 (Latest Official schedule still being discussed EN/GS/OP) PHASE 2 -> Commissioning & Early Operation phase -> all experts on best effort and selected experts on special shift work

  6. Phase 1: HW Tests CO Dry Runs Involved: CO + EQP Groups + OP Working hours BE/CO support organization -> next slide MCCs are responsible for the dry-runs

  7. Dry-runs Organization Claude’s courtesy • 1 Dry run expands over one week • Several dry-runs per week • Groups involved: CO + EQP Groups + OP • Each dry-run deals with a system (ex: Matrix 10 MHz)

  8. Dry-runs Insertion in OP Schedule - example List of DRY RUNS Many in parallel

  9. Dry-runs Support Team Claude’s courtesy 5 working days Period MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY AM PM AM PM AM PM AM PM AM PM Preparation 1 BE-CO Test 1 Debriefing/Debugging/Consolidation Preparation 2 BE-CO Test 2 Debriefing A specific team of 5 experts is assigned to each dry-run by the MCC CO experts for all systems involved are available on-call

  10. Next Steps The dry Run planning (duration, date, content) will be consolidated by Claude (as planning officer) with input from MCCs Then will be discussed with EQP groups + OP + Katy Foraz Thefinal agreed planning will be incorporated in the 2014 OP Schedule Milestone: these steps must be completed by end January 2014

  11. Next Steps for the MCC • Prepare for each dry-run : • List of pre-requisites defined as Jira issues • List of checks defined as Jira issues • Wiki page centralizing all information • List of experts to be present during the whole dry-run and those to be available on-call.

  12. Phase 2: Cold Checkout Can be during night time • 24-hour planning of checks defined by OP • CO INTENSE validation phase • Experts on best effort • Experts PRESENT in the CCC (full availability expected) during day time • Some experts placed under special shift work conditions • Organization under MCC responsibility

  13. Phase 2: Beam Set up & Early Operations phase BEAM SETUP & OPERATIONS • CO is considered operational by OP but is NOT • Experts on Best Effort but high availability is expected • Requested experts placed under special shift work conditions • Organization under EXM responsibility

  14. Special Shift work for phase 2 • Motivation for shift work: speed up CO issues resolution • Inspired from BI Commissioners organization at LHC startup • Role of expert on shift • Offer support to OP on CO problems to diagnose and solve the problem • Report feedback on DIAG tools as will be heavily used • If time left during shift, work on DIAG tools • Will be a natural trigger to improve diagnostics tools to avoid constant calls to CO experts • Organization • 3 shifts per day • Max 2 shifts / week per person and typically only one out of normal working hours • Compensation: experts on shift are required to take the morning off after an evening shift and the complete day off after a night shift

  15. Special Shift work Start CCO Lin2/PSB CO system STABLE • Current estimate of total shift period: • Max 3 months to cover all accelerators • To be adjusted according to real time requirements • AD case (starting later, will benefit from debugging done on previous accelerators -> no shift work may be required)

  16. Special Shift work : status • The detailed organization will be presented to the group early 2014

  17. Phase 3: Beam Operations Beam OPERATION • Controls stabilized • Upgrades only for justified requests and under OP authorization • BEST EFFORT support ONLY

  18. Support model in stable phase • BE/CO support is based on Best Effort specialists teams • For all CO Hardware and Software • We reach a unique model for all accelerators PSX SPS LHC • Based on 2014 interventions statistics, we’ll evaluate and tune our support strategy • Previous ‘Piquet’ suppression motivated by : • 1st line FE support transfer to EQP Groups • New HW -> less failures (teething problems solved in previous phase) • Many problems reported to the PIQUET were outside its scope of responsibility (wrong timing config….) • Pre-requisites for Best Effort mode to perform well: • The responsibility transfer of first line support for FEs from CO to EQP is effective • Strong diagnostics infrastructure -> Vito’s talk

  19. First line support for Front ends 1/2 • Responsibility is transferred to EQP Groups • Except for pure BE/CO systems (LTIM, OASIS, …) • Policy agreed between CO an EQP GP • Special case of legacy systems (ex. CPS/PSB BLM) • Responsibility share between CO and EQP GP • Case by case study to be done • To minimize intervention time, the EQP group specialist will have access to centralized stock of BE/CO HW modules to perform the HW change if (s)he is already on site • BE-BI is on board • Training of EQP experts on procedure to extract HW modules from CO stock • CCC will call directly the FRONT END specialist in the EQP GP

  20. First line support for Front ends 2/2 • CO requirements • Every FE is described in the CCDB with its responsible group and its operational support team • Information is propagated from CCDB to DIAMON and the OP WEB Support page ‘webPiquet’: • DIAMON should display the operational support team when clicking on the front end • WEBPIQUET should display the phone number(s) related to the operational support team • Actions on MCC • Ensure interfaces and data propagation are available • Check that operational support team is filled

  21. BEST EFFORT Expert Teams – not exhaustive nor correct list Teams likely often contacted > hence the importance of good diagnostics OP EQP GP FRONT-END (Ex: TE/EPC FEC) EQP GP FRONT-END (Ex: TE/EPC FEC) EQP GP FRONT-END (Ex: TE/EPC FEC) BECO BEST EFFORT SERVICES INCA /LSA . SIS Logging CONSOLE + SERVER ... . FRONT END SYSTEMS TIMING OASIS FEC OS … DIAMON/LASER …. .. CMW/JAPC DRIVER FRONT END INSTALLATION HW SPECIALISTS • Organization of expert teams under discussion in CO • Team membership • Generic phone number • Anonymous phone number … OTHERSERVICES. ..

  22. Scenario for a problem • A problem manifests in a high-level application or on DIAMON screen • OP will have improved error reporting from application and core diagnostic tools to identify the source of problem • Possible recovery solutions will be available via DIAMON (repair, restart process, reboot FE, etc. ) to be done by OP • Requisite: DIAMON will display the right status of all FEs at all times • If no recovery was possible by available actions, OP will call the best effort expert from the OP Web support list & record the intervention • In case of a FE problem: EQP GP specialist is called and runs his expert diagnostics • if it turns to be a CO FE problem, he will ask the CCC to call a CO specialistfrom the Best Effort team identified as concerned.

  23. Intervention Recording • The recording of each problem and intervention should be done by OP using eLogbook • This is critical to ensure the compensation of the experts on best effort when they are called by the CCC • The best effort specialist will ensure the information is updated at the end of his own intervention • A technical report of the intervention will be written by the specialist • This will be used by the second expert and ease the hand over • This means extensions on e-logbook & Jira • e-logbook: Creation of FAULT sub-categories under ‘ CONTROLS FAULTS’ category, basically one per BEST EFFORT system • e-logbook: Write access mode from home • …JIRA issues automatic creation: to be generalized to all issues & promoted to LHC & SPS • OP agreed on all of these, design to be finalized for elogbook + JIRA • CO recorded interventions will be monitored by EXM to ensure any persisting problems are properly followed up and resolved inside the group

  24. Perspectives 2014 …… for LHC !

  25. Early 2015 schedule  “Nominal" 25 ns Physics from July’15 RECOMMISSIONING WITH BEAM b* = 19 m vdM LHCf Work in progress!

  26. Preliminary Planning – Commissioning starts in 2014 LBDS reliability run:      CCC: 6 weeks June/July From April: Every 2nd week: dry run October: Every week: dry run + sector test preparation Beginning of Nov: Sector Test January: Machine Check-Out Feb + Mar: Beam Commissioning LBDS Prep +LBDS Prep Dry Run +LBDS Prep Dry Run Dry Run LBDS Dry Run Dry Run Dry Run Dry Run Sector Test Dry Run, Settings, Cycle Check-out Beam comm. 2014

  27. Additional slides

  28. OP SUPPORT Web page webPiquet Additional slides

  29. Tentative Re-commissioning Schedule(still being discussed EN/GS/OP)

More Related