1 / 13

Post-LS1 Shift Scheme and Shift Organization

Post-LS1 Shift Scheme and Shift Organization. Recap: 2012 shift scheme. 2012 shift crew consisted of 9 (detector operations) + 1 ( ADC, distributed computing ) shifters . Detector shifters: Muon, ID + BCM, Calorimeter + Forward (Lucid, less ALFA/ZDC) Trigger Run Control

kami
Download Presentation

Post-LS1 Shift Scheme and Shift Organization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Post-LS1 Shift Scheme and Shift Organization S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  2. Recap: 2012 shift scheme • 2012 shift crew consisted of 9 (detector operations) + 1 (ADC, distributed computing) shifters • Detector shifters: Muon, ID + BCM, Calorimeter + Forward (Lucid, less ALFA/ZDC) • Trigger • Run Control • Data Quality, including luminosity monitoring • Slimos and Shift leader • Distributed Computing, including database monitoring also for ONLINE S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  3. Shift organization 1 Questions asked to systems as part of the questionnaire • Were your shifters required to do a minimum number of shifts in a certain period of time? If yes, what was the requirement? • Did you have problems to enforce or meet the requirement? • Were institutes of your system generally ok with the imposed requirement? • Do you think a minimum requirement on shifts helped with good shift and shifter quality and competence? Answers: • Most systems had a requirement on a minimum number of shifts, varying from simply asking shifts should be done in blocks (Trigger), to 2 (DQ) - 6 (LAR, SCT, PIX) blocks in xxx weeks, with xxx typically 2-3 months. The exception was run control with no requirement on a minimum number of shifts. • The general answer was NO, with 2 exceptions: Muons („yes“) and LAR („sometimes“) • General answers from essentially all systems was “yes”. S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  4. Shift organization 2 4. Do you think a minimum requirement on shifts helped with good shift and shifter quality and competence? • The unequivocal answers from ALL systems was “yes, definitely” !! A look at the reality(thanks to Wainer for extracting the info from OTP) – non-detector desks Fraction of shifters having done 1-5 shifts is significant ! S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  5. Shift organization 3 Detector desks LAR req. 6 blocks TIL req. 1 block ? LUCID req. 3 blocks Situation even less pronounced for detector desks ... A remark: Official ATLAS rule, still stated in the shift booking pages, is min. 20 shifts per person doing shift work .... S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  6. Shift organisation 4 Conclusions: • Definitely should require a certain minimum number og shifts after post-LS1 from shifters, and enforce it –subdetector communities seem all ok with it • Many systems remarked tools to check/enforce shift requirements are lacking, maybe we should see globally (OTP) what can be done for it. Gaps between shifts for the same shift person: many people followed the rule of doing shifts within a 2-3 month period, but there is also a significant tail to many many weeks (the plot is somewhat biased due to the different number of shifts done on average per system) S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  7. Post-LS1 shift scheme 1 Questions asked to systems: • DCS operations: Move to a shift crew where all DCS monitoring and DCS actions are being handled by a common DCS shifter for all sub-detectors. • DAQ operations: In a merged shift crew, the DAQ shifter will in charge to handle ANY stopless removal, recovery, the busy monitoring of all sub-detectors, these tasks no longer being done by subdetectorshifters.Would there be tools missing to allow such? • Separating DAQ from DCS operations: • Do you see typical issues in your subsystem where DAQ and DCS interfere, that would be less visible if not monitored by the same shifter? • Do you fear having separate shifters handling DCS and DAQ/Run Control will delay taking the correct action in case of a major problem? • What is the minimum number of shifters in the control room you think one could arrive at to safely and efficiently run ATLAS? • Do you see compelling reasons which absolutely require to keep sub-detector specific shifters? If yes, which ones? • What tools do you think are most missing in order to reduce the number of shifters? S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  8. Post-LS1 shift scheme 2 5. Do you see compelling reasons which absolutely require to keep sub-detector specific shifters? If yes, which ones? The answer to this question was „No“ or for some non-detector systems, no answer which we took as „no/not applicable, eg DQ, with the following exceptions/caveats: DAQ: We do not see major technical reasons. However, as discussed above, a lot of effort will have to be put into standardizing procedures, including e.g. calibrations. It is not clear this feasible in basically one year when a lot of effort will be put into having ATLAS running again. One should probably consider how quickly we want to get there and be prepared to invest the needed amount of resources. Accounting and responsibility are to be considered. Who will be the responsible of a merged desk? There seems to be wider implications on the shift model used so far. • Lumi: A: competence –or lack thereof – over too broad a range of concepts, jargon, issues, actions. LAR (*): No if the shifters are of the proper level of qualification. (*) LAR advocats strongly a professional shifter scheme with people on shifts for ~ 6months • Muons:Yes. (At least the first few months of Run II, we would like to keep the Muon shifter since we need to optimize our tools and recovery procedures. We can reduce the shifter only if we reach the stable operation state.) Comment: this sounds more like what an expert would do in the intial running phase, which will be the case for all systems ... S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  9. Post-LS1 shift scheme 3 1./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter Systems generally seem to agree IF one goes for a recduced/merged shift crew, this is the natural way to organise it .... There is strong objection against such a scheme by LAR: „In a direction of reducing the ACR Crew, it is important to raise the level of expertise of the crew with a detector global vision meaning a global vision connecting Physics, data flow/data quality and detector behavior. This really does not go in the direction of having a central DCS crew but more to have a reduced expert crew in all the above aspects ( so several run manager/shift leader experts). So no central DCS shifters.” S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  10. Post-LS1 shift scheme 4 1./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter • Systems generally (LAR excepted (*))had no concerns if DCS actions would be executed in a merged scheme by somebody not from their own system. • PIX actions are currently restricted to experts, which is planned to remain so. • Lumi made the comment that luminosity monitoring while via the DCS infrastructure, should in a merged scheme not be under the DCS shifter, but rather coupled to data quality or shift leader, since it frequently plays a role in communication with the LHC/CCC. • All systems declared they would see the DCS shift task then open to anybody, with proper training, not requiring specific detector experience etc. • In general, systems did not see too many tools in the area of DCS missing to hand all DCS operations to a common shifter. • Handing all DAQ operations, incl. Removals and recoveries to a common shifter. Some systems stressed this would require standardized removal/recovery mechanisms, and could only work if the recovery does not require detector specific knowledge. Better BUSY monitoring was mentioned (L1Calo). Trigger rate checking would have to be automatised, which is alread planned. Some trigger monitoring as L1Calo TT maps would have to be seen how to handle. S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  11. Post-LS1 shift scheme 5 1./2./3. A shift scheme were all DCS operations is in the hand of a common DCS shifter, all DAQ operations in the hand of a common run control shifter • Systems are generally not concerned about issues where DAQ and DCS interplay, and which potentially could be less visible if not monitored by the same shifter. • PIX answered some revision in the handling of the pre-amps off may be needed, which at present result in a harmless DCS NOT_READY state whenever the run is stopped and the ATLAS partition goes below CONNECTED. • Systems are generally not concerned that having separate shifters handle DAQ and DCS operations may delay taking correct actions in case of a major problem. (we had thought to eg a rack or crate trip leading to a stopless removal of a large part of the detector). Example answers given were • Systems are fully separate. • Any relevant DAQmessages related to DCS are displayed in MRS (TIL) • Not really, since for major issues an SMS/email is issued to experts anyway (TIL) S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  12. Post-LS1 shift scheme 6 4. What is the minimum number of shifters in the control room you think one could arrive at to safely and efficiently run ATLAS? (note: we exclude the ADC shifter here which is separate) S. Zimmermann, ATLAS Post-LS1 Operations Workshop

  13. Outlook Discussion S. Zimmermann, ATLAS Post-LS1 Operations Workshop

More Related