DAQ tutorial

1. DAQ tutorial Carlos Solans Expert training 6th September 2010

2. ATLAS TDAQ • The ATLAS TDAQ is the Trigger and Data Acquisition (TDAQ) software used in the operation of the ATLAS detector. • The aim of these slides is to allow you to become a TDAQ expert for Tile. • These slides are divided into hardware, software and common issues. • There are many topics which are not covered. • Operations expert manual is available (to be updated): • https://atlasop.cern.ch/atlas-point1/twiki/bin/view/Main/TileCalorimeterOperationManualExpert Carlos Solans

3. Data Flow • Electric signals in the detector are stored in pipeline memories in the front-end every bunch crossing (BC) at 40 MHz • The Central Trigger Processor (CTP) of the Level 1 (L1) Trigger selects 100 kHz interesting events defining Regions Of Interest (ROI). • The back-end electronics of the detectors read-out the interesting events through the RODs. Energy computation and zero suppression are applied. • Data is transmitted through read-out links to the Read-Out Buffers of the Level 2 (L2) trigger. • Data Flow Manager executes L2 algorithms based on L1 ROIs. L2 trigger selects 2 kHz interesting events. • Data Flow Manager executes Level 3 (Event filter) algorithms on the data collected from all the detector based on L2 trigger selection at 200 Hz. • Full events are stored in a byte-stream format. Raw Data Objects (RDOs) in Offline jargon. Carlos Solans

4. Tile read-out • ATLAS is located in Point 1 of the LHC ring. The read-out electronics are located in the USA15 cavern of Point 1. • Divided into four TTC (Timing Trigger and Control) partitions (one per barrel). • A TTC partition is the minimal structure that can be operated with the TDAQ. • Each TTC partition contains it’s own TTC and ROD units. • There are 64x2 TTC links to “write” to the detector and 64x2 ROD links to “read” from the detector per partition. USA15 Back-end read-out electronics for the Tile Calorimeter Carlos Solans

5. Back-end electronics • The back-end electronics for each TTC partition are split in two VME crates. • Both crates are controlled by a Single Board Computer (SBC) also called ROD Crate Controller (RCC). • To communicate with the TTC/ROD modules one has to login to the TTC/ROD SBC. • The TTC modules are: LTP, TTCvi, TTCex, TTCpr, RODBusy and LTPI. The ROD modules are: ROD and TBM.The following slides contain an introduction to each module. • VME crates are monitored by DCS. It is also possible to power cycle a crate from the command line: tile_vme_crate. Back-end electronics SBC names in point 1 SBC names in the lab Carlos Solans

6. SBC • The SBC is the VME crate controller for each TTC and ROD crates. • It is a Concurrent Technologies (CCT) VP110, Pentium 3, 512 MB RAM, dual ethernet PC. • Equipped with VME bus interface that allows mapping of all modules connected to the crate. • It is remotely booted and loads a Scientific Linux CERN (SLC) version specifically tuned for TDAQ. • It loads TDAQ drivers at start-up: service drivers_tdaq • Specific post boot scripts can be configured (bwm-post-boot) Carlos Solans

7. TTC Modules: LTP / LTPI • The purpose of the Local Trigger Processor (LTP) is to receive TTC signals from the CTP through the Link-in cable and inject them into the front-end electronics through NIM output cables. • It can be used in stand-alone mode by using local timing and trigger signal sources or by internal signal generation (It has a pattern generator) • It receives a busy signal coming from the RODBusy Module and propagates it to the CTP. • The LTP command line tool is: menuRCDLtp • Base Address 0xF00000 • It is controlled through centrally provided library. • The Local Trigger Processor Interface (LTPI) is an interface between CTP and LTP to allow combined running between various subdetectorsin parallel, as opposed to ATLAS global run. • The LTPI comamnd line tool is: menuRCDLtpi • Base Address 0xE00000 Link-in Ttype Input/output TTC signals: L1A, TT1, TT2, TT3,… Busy Carlos Solans

8. TTC Modules: TTCvi / TTCex • The TTCvi is the module used to configure the front-end electronics. • It provides the two signals (A and B channels) to the TTCex module which contains the electrical to optical converter. • The A channel is used to transmit the Level-1 Accept (L1A) signal. • The B channel is used to transmit framed and formatted commands and data. • Base address is: 0xA00000 • TTCex uses TTCvi clock to send TTC commands through 10 outputs. • TTCoc is a 1 to 8 splitter to distribute the signals to all the drawers in a sector • 1 Sector = 8 Modules = 1 ROD Ttype Signals in Signals out Clock to TTCex TTCvi 10 Optical outputs 1 per sector, 1 ROD+TTCpr,1 LASER Input channels A and B Clock from TTcvi TTCex Carlos Solans

9. TTC Modules: TTCpr / RODBusy • The TTCpr is a mezzanine card in the PMC form factor, pluged in the SBC of the TTC crate, used to make available the Tile TTC information in the TDAQ. • It asserts the busy signal after a TTC event has been received, which is only cleared by software after the Event ID, BCID and Ttype have been read out from it. Only used in calibration runs. • The RODBusy has 16 busy inputs which can be masked and monitored. It is possible to know the number of consecutive BC units an input line has been busy. • It produces the sum of all busy lines as a busy out signal which is connected to the LTP. A VME interrupt can be generated after a programmed timeout. • It receives the TTL busy signal from the TBM in input 0 for all run types and TTCpr in input 15 for calibration runs. • Hardware address in point1 is 0xB00000 • Debug tool is: menuRODBusy TTC in Busy out TTCpr Busy inputs 0-15 Busy output RODBusy Carlos Solans

10. ROD Modules: ROD/TBM • The Read Out Driver (ROD) is the last element of the Level 1 trigger. It is located between the front-end electronics and the Read-Out System of the Level 2 trigger. • ROD is controlled and monitored from the SBC through the VME bus. RODs are installed in slots 6,8,10,12,14,16,18,20. Geographical mapping is masked through the TileROD library. • The Data comes in and out optical links on the board. There is no VME read-out of the data. • There are 4 Processing units slots, only two are populated. • Conditions data is loaded into the ROD memory for the DSP processing. • TTC information is received through the backplane from the TBM and distributed to all DSPs. • Busy signal is sent out through the backplane to the TBM which makes a logical OR of all busy signals and is connected via a TTL output to the RODBusy module. Processing units VME bus Data input links ROD motherboard RCC TBM ROD ROD ROD ROD ROD ROD ROD ROD Commands Config & control Read-out Monitoring Monitoring Data ROD crate operation summary Carlos Solans

11. ROD Modules: ROD (II) • Data comes in through the optical links, it is deserialized in the G-Links and trasferred to the Staging FPGAs. 2 Stagings are routed to one Processing Unit (PU) which processes data from 4 Front-End Boards (FEBs). Data from the PUs is routed to the Output Controller FPGAs (OCs) and serialized throught the back-plane to the ATLAS wide standard S-Link cards. • XOFF signal propagated back from the ROB input (ROBin) cards is interpreted as a busy signal from the ROS. No data can be sent out if XOFF is asserted. • Busy signal from the ROD is distributed through the VME backplane to the TBM. The ROD is the only component in the TDAQ that can assert the busy signal. Carlos Solans

12. ROD Modules: ROD (III), DSP • Each PU is equiped with two Input FPGAs (InFPGAs), two Digital Signal Processors (DSPs) and one Output FPGA (OutFPGA). • There is no EPROM to keep the InFPGA or DSP code on power down, so, the devices have to be re-programmed always after the power up of the crate. These files are found in $TILE_DSP_PATH. • Tile_InFPGA.dat/Tile_DSP.ldr is a symbolic link to the current InFPGA/DSP code • It performs a copy task and/or optimal filtering on the data. Optionally it can perform iterations, muon tagging and total transverse energy computation and histogramming if optimal filtering is enabled. • If TTC synchronization is requested (always), the BCID of each data event is matched to the BCID of the TTC event received in the ROD. If there is no match, data is sent out anyway with a flag. • Conditions data required per drawer are: Optimal Filtering Constants (OFC), bad channel list, muon ID thresholds and Fragment 1 conditions. Data out OutFPGA InFPGA DSP Data in Processing Unit Carlos Solans

13. Hardware summary • There are only 2 links between Tile and the CTP, LB and EB. There will be only two busy inputs to the CTP. • LB is a daisy chain of LBA and LBC and EB a daisy chain of EBA and EBC TTC partitions. CTP Front-end and TBM L1A L1A L1A Busy L1A Control LTPI LTP TTCvi TTCex TTCpr RODBusy RCC Busy Busy ROD crate Front-end Data Data Busy TTC crate Data Control ROS OC DSP ROD TBM RCC XOFF L1A Carlos Solans

14. TDAQ software • A partition is the software element that defines the set of software and hardware that has to be executed at a time. There can be several partitions defined. For example, Tile has a partition named Tile to take calibration runs and the ATLAS partition is the partition that groups all sub-systems together. • The TDAQ software can operate only one partition at a time and partitions cannot be nested. Therefore, the sub-systems include their partitions as another type called segment to the main partition. • The description of the software is done through he configuration database. Also called the OKS database because of the package that implements it, or the XML databases because the back-end storage is in XML. • The TDAQ software is distributes in releases, which are a collection of CMT packages. There is an AFS installation and a point 1 installation. • /afs/cern.ch/atlas/project/tdaq/cmt • /atlas/sw/tdaq Carlos Solans

15. Tile online release • The software used for the operation of Tile that uses the TDAQ infrastructure is grouped in the Tile online release. Numbering convention follows TDAQ numbering and adds a minor release number: • tdaq-02-00-03  tile-2.0.3.6 • Current architecture is: i686-slc5-gcc43 • As the TDAQ release, the Tile online release is a set of CMT packages built together and distributed in AFS and Point 1. • /afs/cern.ch/user/t/tiledaq/public/tilercd • /det/tile/rcd • AFS installation of the software is used for development and Lab setups at CERN and IFIC. Patches and installed version of the software is found in Point 1. Patches is group (ATLAS) writable path and preferred to the release path. • Minor releases are built on demand, especially when the changes involve changing of symbols of the compiled binaries. • The usage of the Tile online software is recommended to the direct usage of the TDAQ release as it includes Tile specific tools. • /afs/cern.ch/user/t/tiledaq/public/tilercd/bin/setup.sh tile-2.0.3.4 • /det/tile/scripts/tdaq-02-00-03/setup.sh Carlos Solans

16. Setup the environment in the lab At CERN use tilerod account for development and debugging in Lab@175. Use valtical01 as main computer to start the partition. Other available computers are valtical02, valtical13, valtical03, valtical19. [lxplus240] ~ > ssh -XC tilerod@valtical01 Scientific Linux CERN SLC release 4.8 (Beryllium) tilerod@valtical01's password: tilerod@valtical01 > cd tdaq-02-00-03/ tilerod@valtical01 > source setup_daq.sh Setting up Tile Online SW release "tile-2.0.3.6" Setting up CMT v1r20p20081118 Setting up TDAQ Common SW release "tdaq-common-01-14-00" Setting up DQM Common SW release "dqm-common-00-10-00" Setting up DAQ SW release "tdaq-02-00-03" i686-slc5-gcc43-opt SVNROOT=svn+ssh://svn.cern.ch/reps/atlastdaq Setting up HLT SW release 15.4.0 tilerod@valtical01 > Carlos Solans

17. Setup the environment at Point 1 • To login to Point 1 you have to access through the gateway from CERN GPN. Access from home is forbidden, you must login to lxplus first. • If the access is free, you can login to any point 1 machine. Recommended: pc-til-scr-02/03 • If the access is restricted (while beam) you must follow instructions. • Actual Tile online release used is set by setup_daq.minimal.sh which is executed by the setup script. [lxplus240] ~ > solans$ ssh -XC solans@atlasgw solans@atlasgw's password: Check login authorization service... … Request authorization from TDAQ Access Manager service...GRANTED! Disk quotas for user solans (uid 19014): … STEP 1 - Hostname: pc-til-scr-02 Access granted inside Point 1 ! Proceed to next step... Detected OS for target machine: linux STEP 2 - Command [terminal]: STEP 3 - Username [solans]: >> Environment: lab32, 40, 513 >> Based on "SLC5_32 (Boron)" >> BWM project v7.14 built on 220110-09:52 for kernel 2.6.18-164.11.1.el5 solans@pc-til-scr-02 ~ > source /det/tile/scripts/tdaq-02-00-03/setup_daq.sh Setting up TDAQ Common SW release "tdaq-common-01-14-00" Setting up DQM Common SW release "dqm-common-00-10-00" Setting up DAQ SW release "tdaq-02-00-03" Setting up partition "Tile" solans@pc-til-scr-02 ~ > Carlos Solans

18. CMT • CMT is the ATLAS wide package distribution standard. We distribute source code, not compiled files. • A CMT package is nothing else but a folder structure containing a requirements file inside a directory called cmt. It can be compiled and installed through simple commands (cmt config, make, make inst) issued from the cmt folder. • The requirements file specifies with a simple language the name of the package, the author, the contents (applications, libraries, java binaries, … ) to be built, the linking options and the packages it depends on. • http://isscvs.cern.ch/cgi-bin/viewcvs-all.cgi/detectors/Tile/TileDDC/cmt/requirements?revision=1.12&root=atlastdaq&view=markup • All packages must use a policy package, in the case of Tile this is the DataflowPolicy package. • To use CMT you must setup your environment appropriately. The setup of TDAQ and Tile releases do it for you. • CMT packages for TDAQ and Tile releases are stored in a software repository. TDAQ uses SVN, Tile still uses CVS. • https://svnweb.cern.ch/trac/atlastdaq/browser • http://isscvs.cern.ch/cgi-bin/viewcvs-all.cgi/detectors/Tile/?root=atlastdaq Carlos Solans

19. Supervision System • The supervision system is responsible for starting and stopping the TDAQ applications. If the applications implement the TDAQ FSM, therefore called run control applications, the transition commands are also issued by the supervision system, which will watch their run control state. Nearly all TDAQ applications in Tile are and run control applications: RCD, MinBias and Athena PT. • The TDAQ applications are organized in a hierarchical tree, the so called, run control tree. • Applications are grouped in segments. There are five major Tile segments, one for each TTC partition and one for Athena monitoring. • Applications handle resources. For example, this is how the RODs in a crate are described. • Resources can be grouped in resource sets. This is how the resources describing the ROD output links (TileReadoutChannel ) and the ROS input links (RobinDataChannel) are grouped together to describe a read-out link. Chop of the Integrated Graphical User Interface (IGUI) of the TDAQ running the ATLAS partition. Run control tree, commands and state are shown. Carlos Solans

20. Tile run control tree 4 barrel segments + Monitoring segments Carlos Solans

21. Tile run control tree Carlos Solans

22. Configuration Plug-ins TileConfig ROSDBConfig TileTTCprTriggerIn TileDigiTTCModule RCD/ROS core (IOManager) Trigger Plug-ins Module Plug-ins TileTBMTriggerIn TileCal_RODModule EmulatedDataOut TCPDataOut Output Plug-ins RCD • ROD Crate DAQ (RCD) is the base application executed in the SBCs to operate the crates. It’s behaviour can be adapted to Tile’s specific needs through plug-ins. • There are module, trigger, configuration and output plug-ins. • The design functionality, used by the ROS is: According to the configuration read by the configuration plug-in, the trigger plug-in will poll the module plug-in for data and push it to the output plug-in. • In the case of Tile, module plug-ins are only used for configuration and monitoring purposes, as the data-flow is transparent to the VME. A few modules are described as trigger plug-ins for hierarchical reasons and for calibration runs. Output plug-ins are only used for calibration runs. • It implements all the run control state commands. These are propagated to the plug-ins. • Each RCD application is described in the configuration database. Tile back-end hardware operation is basically handled by RCD. There is one RCD application running in each TTC and ROD crate. Carlos Solans

23. Most important module plug-ins • The TileDigiTTCModule plug-in is the central part of the TTC configuration. Uses a Tile specific TTCvi VME library (ttcvi) to communicate with the drawers through 3in1 commands. • Plays key role together with TileDigiTTCModuleDataChannel in calibration runs for the CIS injection steering. • Located in the package TileModules. • The TileCal_RODModule is the central part of the ROD configuration. Uses Tile specific ROD VME library (TileROD) to control the ROD and it’s devices. • At Configure, boots the DSPs and allocates OFC plugin. At Prepare for run, configures the DSPs for next run, reads condition data from OFC plugin and loads it to the DPSs. • Located in the package TileCalModules. Carlos Solans

24. Conditions data reading • ROD module implements OFC plugin to load conditions data • TileOfcCoolPlugin reads conditions data from COOL • TileOfcShameModule is an RCD module that has no hardware associated, used to connect only once to the DB. It loads all conditions data to the shared memory region of the SBC at prepare for run • A TileOfcShamePlugin is used by the ROD modules to read the conditions data from the shared memory at prepare for run (after it has been read by the TileOfcShameModule) • A common failure is the following: • Previous shifter run partition under his user and shutdown without terminate. The shared memory is still allocated under this user • The conditions data chenges • Next run produces a non-fatal error on the shared memroy creation by TileOfcShameModule which is ignored by shifter. Data for current run contains wrong set of conditions. • Use standard linux commands to clear the shame (ipcs, ipcrm) or reboot the SBC. Carlos Solans

25. IGUI • The Integrated Graphical User Interface (IGUI) is the main user interface with the TDAQ. It controls one partition at a time. • It is possible to extend with user panels that are added as tabs to the display. • However, the number of display tokens is limited. You might not be able to open it in key moments. • Message pane is annoying. Select only FATAL messages to reduce the number of callbacks. • You can bring it up in point 1 after you setup the environment with the following command: Partition name Click to load panels Tabs Messages • solans@pc-til-scr-02 ~ > Igui_start -p ATLAS -d oksconfig:combined/partitions/ATLAS.data.xml Carlos Solans

26. Run status web interface Run, counter, trigger and beam info The run status can be followed via web through altasop.cern.ch Click on systems  DAQ Busy info (note Tile) Rates Carlos Solans

27. Read application logs 1. Click on Load Panels 2. Select PmgGui panel 4. Select application 5. Click on out log 3. Select host Carlos Solans

28. Read application logs (II) • Accessible from a web interface:https://atlasop.cern.ch/partlogs/<computer>/<partition>/ Carlos Solans

29. Tile panel Click on the Tile tab to open the Tile panel You can also open the Tile panel without openning IGUI solans@pc-til-scr-02 ~ > tile_igui_panel Carlos Solans

30. Tile panel Click on the DDC tab to view the DCS information The information displayed in this panel is read from IS. The application that puts the values in IS is TileDcs2Daq. There is one TileXXX_Dcs2Daq application in each TTC partition. For detailed information refer to DCS panels. Carlos Solans

31. Tile panel Tile tab LBA LBA Crate panel Crate • You can see • which ROD is busy • Free • Busy • partition busy • masked modules Press Ritmo to get more information Carlos Solans

32. Ritmo panel TTC FEB1 DSP DATA FEB2 BUSY DSP input/output block diagram Ritmo stands for “ROD Information for Tile Monitoring” It displays a selection of DSP counters per ROD 8 super-drawers, 2 Pus, 4 DSPs Events in, out, discarded. TTC events in and out, busy counts and total events. And information concerning the last event processed Carlos Solans

33. Globals panel Tile tab Globals Raw data and/or Reconstructed magnitudes Select iterations or no iterations The Globals panel allows you to select reconstruction settings for all Tile Select units Sometimes the globals does not display the correct values. Check the values in IS. Carlos Solans

34. IS Start IS monitor • The Information Service (IS) is the infrastructure service that allows exchange of information between applications. • There are a number of IS servers per partition. Tile DAQ uses TileParams IS server to store information about ROD and DSP configuration (TileCal_DSPConfig), DSP processing status (Ritmo), super-drawer status… IGUI toolbar • solans@pc-til-scr-02 ~ > is_monitor Click to view details Click to select partition Carlos Solans

35. Read IS from the command line Check the DSP settings tilerod@valtical01 > is_ls -p TilePartition -n TileParams -R TileCal_DSPConfig -v -D Server "TileParams" contains 1 object(s): TileParams.TileCal_DSPConfig <11/2/10 21:43:39.242010> <TileCal_IS_DSPConfig> 7 attribute(s): /* DSP processing format: 0=RAW, 1=RAW+RECO, 2=RECO */ 1 /* Enable 1 iteration */ 1 /* Enable Missing Et */ 0 /* Muon tagging algorithm */ 0 /* Enable Online Histogramming at DSP level */ 0 /* Enable TTC synchronization at DSP level */ 1 /* Number of samples for physics */ 7 Check the OFC plugin settings tilerod@valtical01 > is_ls -p TilePartition -n TileParams -R TileOfcCoolPlugin -v -D Server "TileParams" contains 1 object(s): TileParams.TileOfcCoolPlugin <11/2/10 21:43:39.242778> <TileCal_IS_OfcCool> 4 attribute(s): /* Optimal Filtering type: 0=none, 1=OF1, 2=OF2 */ 2 /* Energy Units: 0=ADC, 1=pC, 2=CspC, 3=MeV */ 1 /* Database connection string */ sqlite://schema=/work/TicalOnline/conditions/ 15.5.3/data/tileSqlite.db;dbname=COMP200 /* Path where to store debug output of the constants */ /det/tile/logs/conditions Carlos Solans

36. Read IS from the web https://atlasop.cern.ch/atlas-point1/tdaq/web_is/app/is.html Carlos Solans

37. Minimum Bias applications • Minimum bias applications are run control applications but do not use the RCD. They implement the run control states and are included in the run control tree under their appropriate segment. • They use the Tile Read-out Buffers (RB) to read data from the drawers through the integrator output (CAN bus cable). And also use the TTCvi module to send commands to the drawers. • Common problems are related to multiple access to the TTCvi. Including in rare cases the reset of the TTCvi settings after configure of the RCD. • The TTCvi command line tool is: menuTtcvi Carlos Solans

38. Stopless recovery • The stopless recovery is a mechanism to continue running in case a resource goes busy without stopping the run. The reason is because it takes to long to re-configure the run and thus it this is not a reasonable action to solve a problem. The aim is to improve the data taking efficiency. • For a given partition a resource set tree is built at the start of run which monitors the busy tree. If the resource that goes busy does not react on it, its top level resource will take action. The two actions of the recovery are resource disabling and enabling. • After the run is stop, the actions taken by the stopless recovery are dismissed and the next run starts with clean configuration from the database settings. • The triggering for the stopless recovery disabling is a hardware error. In case of Tile this is issued by the ROD RCD module based on the percentage of time a DSP is busy within a predefined number of seconds. • If busy level is [90,100[ % a warning is raised. Note the nominal busy is 20% when running at 85 kHz (Should be investigated further) • If busy level is 100 % an action hardware error is issued for whole PU (4 drawers = 1 read-out link). There is a logica OR of the busy of two DSPs. • There is still no reporting of a faulty drawer. • The hardware error is received by the expert system and displayed to the shifter for acknowledgement. If hardware is disabled, a disable command is issued to all applications controlling the resources linked with the faulty one. • ROD RCD receives command and distributes it to ROD modules. Corresponding ROD module stops processing in whole PU and masks the busy signal going out. Carlos Solans

39. Hardware failure error ROD busy is detected by Tile tab Event rate stopped Pop-up reported to control IGUI Error reported to all IGUIs (control + spies) Carlos Solans

40. Stopless recovery (II) • The resource re-enabling after the disabling is shifter operated. The first step is to RESYNC the “faulty” hardware. • To recover a read-out link, a run control command has to be sent to the ROD RCD. This will make the ROD module send a hardware recovered issue for the disabled resource. The run control distributes the enable command for that resource to all linked resources and enables both the ROS and the ROD part. The ROD will configure the masked DSPs and start the processing in them again. • rc_sendcommand -p $partition -n $app USER RESYNC $hw • To recover a super-drawer, a similar sun control command has to be issued to the TTC RCD. This will re-configure the given drawer. Note in this case, it is required to hold the trigger. • rc_sendcommand -p $partition -n $app USER RESYNC $drawer_index ROD is 100% busy Operator tries to resynch disabled HW Send hwError for PU ROD sends hwRecovered Operator disables hw Hw recovered? yes yes Stop DSP Mask busy Expert system enables HW Removal Clear and start DSP Un mask busy Recovery Shifter operated Carlos Solans

41. Stopless recovery (III) • The bookkeeping of disabled resources is done in the Resources IS server and in COOL. There is a specific folder (TileDrawerModule, TileCal_RODModule, RobinDataChannel) for the Tile resources in a central place: • /TDAQ/EnabledResources/$partition/$detector/$folder • Not always it is acceptable to automatically disable a resource. When the percentage of the detector affected is beyond 3%, strong suggestion must be made to run control to restart the run or investigate the problem further. • The only way to investigate a busy problem is to do it while the resource is busy and no action has been taken. • Common failures are related to TTC operations while ROD configuration. The symptoms are always the same: ROD busy. • Try to gather as much information as you can before the run is stopped or busy is masked. What can go busy in Tile Tile Read-out resources Carlos Solans

42. Stopless recovery (IV) Find out which are the disabled resources solans@pc-til-scr-02 ~ > is_ls -p ATLAS -n Resources -R ".*Disabled.*Tile.*" Server "Resources" contains 1 object(s): Resources.StoplessRecoveryDisabled:TileEBA_ROD5_Chan2_EBA33-EBA36_ROL08 Send command for recovery solans@pc-til-scr-02 ~ > rc_sendcommand -p ATLAS -n TileEBA_RODRCD USER RESYNC TileEBA_ROD5_Chan2_EBA33-EBA36_ROL08 The result of the recovery is sent to the MRS as a warning: Enable HW = TileEBA_ROD5_Chan2_EBA33-EBA36_ROL08 by TileCal_RODModule_5 It can happen that the hardware goes busy again. If this happens, try to find out what is the cause before trying to recover the hardware once more. Carlos Solans

43. Drawer reconfiguration • One of the common actions is to reconfigure a super-drawer after a power cycle in the middle of a run • Please, coordinate action with Shift Leader • Alternatively, you can instruct shifter to execute similar command from GUI started from DAQPanel  TILE • /det/tile/scripts/bin/tilereconfig.py solans@pc-til-scr-02 ~ > rc_sendcommand -p ATLAS -n TileEBA_TTCRCD USER RESYNC TileEBA05 Carlos Solans

44. Configuration database • One cannot overemphasize the importance of the configuration database, but we will need a dedicated tutorial to learn how to use the oks_data_editor. Please use it in the lab. This is the link to the old documentation: • http://atlas-onlsw.web.cern.ch/Atlas-onlsw/components/configdb/docs/oks-ug/2.0/html/OksDocumentation-11.html • The oks data editor is the graphical user interface for the user to the configuration database. The interface starts loading the database from a given file (usually a partition file) which includes many other files (schema and data) that may include even more. • oks_data_editor tile/partitions/Tile.data.xml • Once the self contained database is loaded, a list of classes is presented. All Tile configuration is described as objects of a class in the list. Carlos Solans

45. Tile read-out database view Carlos Solans

46. Changing the Tile release • The version of the Tile online release used is controlled by the configuration database. • Three SW_Repository objects define the location of the • Tile online software: Tile_Online_SW • Tile patches area: Tile_Online_Patches • Tile external software: Tile_External_SW • InstallationPath attribute has to be modified in all three to change the software release. Carlos Solans

47. OKS web interface • https://atlasop.cern.ch/atlas-point1/tdaq/web_is/app/oks.html Carlos Solans

48. Busy presenter solans@pc-til-scr-02 ~> /det/ctp/scripts/busy_display ATLAS The maximum Level 1 rate is dominated by the final trigger veto. Maximum rate for different processing types The Busy is the mechanism to lower the Level 1 rate dynamically. Tile inputs EBA LBA LBC EBC If you need to change the globals settings for a single run, values should be saved to IS. Only save values to DB when you want to keep settings from run to run. Globals manipulation should be done in initial state from the RunControl desk. LB EB Busy signal to CTP

49. Monitoring • There are three different levels of monitoring • ROD: Hardware based monitoring (DSP) • ROS: Monitoring of simple quantities, reduced granularity (GNAM) • EF: Sophisticated monitoring (Athena) • There are two main monitoring displays (OHP and DQMD) Carlos Solans

50. OHP web interface • https://atlasop.cern.ch/atlas-point1/tdaq/web_is/ohp/Tile.html Carlos Solans