Inner Detector Combined Shift

1. Inner Detector Combined Shift • Tools and Procedures • version 0.8 • September 6th, 2011

2. Table Of Contents • ID General FSM Panel • IDE FSM Panel • ID General Alarm Panel • DAQ Tools • ID Operations & Problems Database • Checklists & Elog • Special Actions: Warm start/stop, beam dump • Online Histogram Presenter • Data Quality Monitoring Display

3. ID General FSM Panel

4. FSM Panel Explained LHC Handshake Status Fast Navigation Alarm Summary ID FSM Tree FSM Main Panel FSM Help Secondary Panel

5. ID FSM Main Panel ID Env. Background Conditions SCT SCT Automatic Turn On Pixel TRT BCM/BLM & Post mortem Injection permit

6. ID Environment Evaporative Cooling Status Compressor Configuration Liquid level Environmental Cooling Status TEH Status Thermal Enclosure Heater (TEH) Trip Recovery Button

7. SCT Endcap C Status Barrel Status Endcap A Status Pix SCT Infrastructure Status Snapshot shows normal status in STANDBY Should ALL be GREEN when SCT is ON

8. Pixel Barrel Layers C-Side Disks A-Side Disks Pixel Infrastructure Snapshot shows normal status in STANDBY Should ALL be GREEN when PIX is ON

9. TRT ALL indicators should ALWAYS be green Endcap C Endcap A Barrel C Barrel A TRT Infrastructure Gas Gain Stabilization System (GSSS) Status

10. Beam Conditions(see later slides for descriptions) BLM, RAD Mon SCT Auto-ON Injection Permit Status Ready For Inj. Status Inj. Permit Mask Configuration BCM post mortem counter State 6: normal State 5: PM analysis)

11. Backgrounds Green: OK Red: Too high (or no data) Pixel Occupancies SCT Occupancies BCM Rates Background monitoring active? Stable beam yes/no?

12. IDEnvironment FSMReached via IDE buttonUseful for more detailed look at ID environment & beam conditions, eg in preparation for warm start

13. IP1 Collimator Check Panel(IDE FSM) Absorbers Background rates Beam pipe pressure Tertiary collimators Beam parameters

14. Cleaning Collimator Panel(IDE FSM)

15. Beam Rates Panel(IDE FSM) LUCID, ZDC rates Beam pipe pressure, BCM rates, LHC params SCT/TRT occupancies

16. Example - Beam Rates Lumi optimization SQUEEZE RAMP Collapse of the separation bumps → collisions

17. IDG Alarm Screen Click On Alarm for details, help, and logging Sort According to Severity

18. IDG Alarm Help Right click on alarm to see options Insert Directly to elog Get Alarm Details Open Alarm Help (Twiki page) All alarms shall be logged Errors and Fatals shall be followed by a call to the expert

19. IDG Alarm Help (Opened by selecting ‘Alarm Help’ in options list) Alarm explanation along with actions to be taken. Please report any missing help pages!

20. DAQ Panel Busy monitor ATLAS partition OHP DQMD

21. DAQ Tools • Shifter Assistant • Alerts the shifter to specific DAQ error conditions that require shifter action • Eg ROD busy or disabled etc • Complements IGUI panel indicators • Includes instructions for the shifter • IGUI Panels • Indicates DAQ related alerts and warnings • Provides shifter tools to take the appropriate action • MRS • Warnings/Errors/Fatals from DAQ applications • Eg missing essential files, application crashes etc • Usually require expert actions, but shifter should monitor MRS for ERRORs or FATALs

22. Shifter Assistant Tool New alert Expand if new alert http://pc-tdq-mon-61:8081/alerts/Detectors (Link available soon from desktop)

23. Shifter Assistant Tool Read the messages and mark them as read Info & Warnings can be ignored Errors and Fatals shall be followed by a call to the expert

24. More details in SA

25. ATLAS DAQ IGUI Loading IGUI Panels

26. IGUI Panels • The IDPanel under the ID tab of the TDAQ IGUI provides: • Visible indications of any SCT and TRT DAQ-related alerts or warnings that the ID shifter should react to • In principle, the same alerts/warnings are displayed on the Shifter Assistant • Tools to react to those alerts of warnings • The IDPanel should be open and visible at all times on the ID shifter desktop • At the start of the shift, the shifter should open IDPanel (if not already open) by ticking ‘IDShifterGui’ in the list of tabs shown when clicking the ‘Load Panels’ button at the top of the TDAQ IGUI • For info, the IDPanel can be closed by unticking ‘IDShifterGui’ in the same list, but the shifter should leave the panel open

27. Panel Structure • To the left of the panel is an expandable tree indicating the DAQ status for both SCT and TRT • Green: OK, Yellow: WARNING, Red: ALARM • The ‘ID DAQ Status’ icon colour indicates the combined status for both SCT and TRT • The ‘TRT’ and ‘SCT’ icons in the tree indicate the overall status of TRT and SCT respectively • If yellow or red, click on the sub-tree component to see the origin of the warning/alarm and the tools to resolve the problem

28. Overall SCT Control and Status panel, visible by clicking on the ‘SCT’ icon in the tree

29. SCT Control & Status Panel : 1 Control Mode: Enables recovery tools Only one TDAQ IGUI has control mode enabled (if someone opens anotherIGUI and takes control mode, this IGUI willhave control mode disabled) Module Recovery: Automatic recovery of link errors Should be enabled by default If Control Mode is disabled, modulerecovery gets disabled too. Global Reconfiguration: Time between automatic global reconfigurationof the SCT to counteract chip configurationcorruption by SEUs Set to 30minutes by default

30. SCT Control & Status Panel : 2 Disabled ROD count across full SCT Green: 0 Yellow: 1 (take action!) Red: >1 (take action!) Link Error count across full SCT Green: <50 Yellow: <100 (call expert!) Red: >100 (take action!) Busy ROD count across full SCT Green: 0 Red: >0 (take action!) If any of these 3 indicators are not green, click on the crate icons which are non-green to perform actions

31. ROD status in first column (Grayed out slots mean no ROD present in that slot) Green: ROD is Enabled, and not BUSY Yellow: Disabled ACTION: Right mouse click on disabled ROD and select‘Invoke Stopless recovery’ Then recover modules (next slide) • Red: ROD is Enabled, but it is BUSY • ACTION: • Ask RC shift to invoke stopless removal • If RC shifter clicked NO by mistake, right mouse click on the red RODand select ‘Invoke Stopless Removal’

32. ROD percent busy Useful to see if 100% busy or not

33. Link Errors per ROD Should be ideally 0 If non-zero, module recovery is attemptedautomatically (if enabled in control&status panel) Green: <10 Yellow <20 Red >20 Yellow: ACTION: Data ok, but call SCT DAQ/DCS expert on 162034 during normal hours, make elog • Red: • ACTION: • Right mouse click and select ’Recover all modules’ • If no improvement: • call SCT DAQ/DCS expert on 162034 • invoke stopless ROD removal, then stopless ROD recovery, then again select ‘Recover all modules’

34. TRT Panel Visible by clicking on TRT Recovery

35. Synchronization: - If the # of sync problems is > 0, an automatic resync is performed (no shifter action necessary) - If prompted by the Shifter Assistant or an expert, click the Request Resync button to manually request a resync

36. ROD Recovery: - If the # of disabled RODs is > 0, click the Recover Disabled RODs button to attempt to recover these. - Run Control will have to acknowledge a popup window for this procedure. - If this doesn't work, call the TRT DAQ expert on 160531

37. Specific ROD Recovery: - If instructed by the shifter assistant or an expert, you can force a ROD recovery. - Type the ROD address into the text box and click Request ROD Recovery. - Run Control will have to acknowledge a popup window for this procedure. - If problems occur, call the TRT DAQ expert on 160531

38. MRS Enter subscription string to filter for SCT, TRT, PIX and BCM SCT|TRT|PIX|BCM (? To be confirmed) Call experts for ERRORs or FATALs

39. Pixel Rod Mon Not supposed to be an ID shifter tool

40. ID Operations and Problems DB Web interface opened via ID menu on desktop (keep it open for easy/quick reference)

41. Searching Prob. DB Specify System Specify Category

42. Searching Prob. DB Click on DB entry to get details Enter String to search DB

43. Searching Prob. DB Details with explanation of DB entry and solution prescribed

44. Checklists and Procedures • Elog • Start of Shift • Warm Start • Warm Stop • Beam Dump • During Shift • End of Shift

45. Elog Maintain a log throughout your shift of what happened, and upload it at the end of the shift. A new elog interface will be available “real soon” for use by the ID shifter: - uploads your entries directly as you type them, line by line, and appends them to the shift elog summary - useful for experts and run coordinators to see the status of your shift in ‘real time’ - will only append (not delete/change previous lines)

46. Start of Shift • Arrive to Shift 15 Minutes before Start of Shift to get summary from previous shifter! • Login to • Elog • DCS ID FSM Panel • DCS Alarm Screens • Run Com Tool (not yet possible) • Introduce yourself to the shift leader (if they haven’t done so already). • Read latest elog messages from last 24 hours. Keep elog regularly updated • Ensure that OHP, DQMD, SA, and IGUI tools are up and running with the proper configurations. • Ensure that individual ID desks are running their proper tools as well. • Check the alarm screen for any alerts. • Check the DAQ status (IGUI ID panel and SA) • Check the status of the pixel infrastructure (pixel igui, checked by SA) • Check the FSM Status (for silicon should be READY/OK during data taking, otherwise in STANDBY/OK, TRT should always be READY/OK).

47. During Shift • Monitor and acknowledge any alerts from the SA tool, and take action as needed • Monitor the IGUI ID panel and take action as needed • Log any alarms in the FSM. Acknowledge them once they have been logged and WENT. Take any necessary actions needed (use Alarm Help and/or Problems DB tool). • Check the trend in evaporative cool plant liquid level (~ 1kg / 8 hours). • Check ROS occupancy is non-zero in FSM. If it is zero and beam is present, then ask the Run Control Shifter to restart the SCT ROS Monitoring. • During data taking, check hourly • DQ checklists • Number of disabled pixel modules, especially b-layer (OHP) • Check Pixel Module Errors (i.e. LVL1ID errors) (Pixel Rod Mon / DQMD)

48. Start of Run It is important to be specially vigilant at the beginning of a run. Problems most likely occur at this point. Be aware of: • FATAL / ERROR messages from the Pixels, SCT, TRT, or BCM appearing in the MRS • Check BUSY status using the BUSY status display. If no triggers are coming through one of the systems could be busy. • Check the IGUI ID panel to ensure all is green • Stop-less removal could happen at this stage, the run control shifter should tell you if an ID System goes busy and may disable one or more of the RODs. Follow the instructions in the operations database if this happens. • Ensure that all the Pixel RODs are allocated by the Central DAQ. If not an ERROR should appear in the SA

49. During a run Keep checking the SA and IGUI ID panel. If the run control shifter did not tell you he/she disabled an SCT ROD, you would see a big increase in link errors. Check SCT ROS Monitoring every hour. Look for errors and anomalous values in the results Continue to check for errors in MRS Watch out for busy of any of the detectors.If the run control shifters asks to disable a ROD always say yes. Document everything throughout your shift and upload a single e-log entry at the end.

50. Before beam injection Before and during beam injection the Pixel and SCT need to be in the STANDBY state, ie for Pixel preamplifiers and HV off and for SCT reduced HV (50V instead of 150V) Make sure that neither Pixels nor SCT maskes the stable beam flag / "Injection Permits" If SCT is not already in STANDBY, Issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. If you can't get all branches to be in the STANDBY state, call the SCT DCS on-call expert (162034). Check that Pixel is in STANDBY and gives injection permit. If not please call the pixel on-call (160032).