300 likes | 490 Views
DATE the DAQ s/w for ALICE (Birmingham, Budapest, CERN, Istanbul, Mexico, Split, Zagreb collaboration) and its possible application at NA49-future. ALICE Experiment. Computing Center. DAQ. 10Gb Ethernet. Counting rooms. DDLs. Machine: LHC @ CERN, Point 2 ALICE detectors: 17
E N D
DATE the DAQ s/w for ALICE(Birmingham, Budapest, CERN, Istanbul, Mexico, Split, Zagreb collaboration)and its possible application at NA49-future NA49-future Meeting, January 26, 2007
ALICE Experiment Computing Center DAQ 10Gb Ethernet Counting rooms DDLs Machine: LHC @ CERN, Point 2 ALICE detectors: 17 Online Systems: ECS, DAQ, TRG, HLT, DCS Readout: ~400 DDLs @ 2125 Mbps Storage rate: max. 1.25 GB/s Events: 2.5 MB (p-p), 87 MB (Pb-Pb) DAQ Software: DATE V5 Cavern NA49-future Meeting, January 26, 2007
ALICE DAQ architecture PDS TDS TDS Rare/All CTP L0, L1a, L2 BUSY BUSY LTU LTU DDL H-RORC L0, L1a, L2 HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment Sub-event Event File 10 DDLs 10 D-RORC 10 HLT LDC 123 DDLs 262 DDLs 329 D-RORC 175 Detector LDC LDC LDC LDC LDC LDC Load Bal. Event Building Network EDM 50 GDC 25 TDS GDC GDC GDC DS DS GDC 5 DSS Storage Network NA49-future Meeting, January 26, 2007
ALICE online logical model Detector ALICE Terminology Digitizers Front-end Pipeline/Buffer Trigger Level 0,1 Decision Decision Detector DataLink (DDL) DAQ Read-Out Receiver Card (D-RORC) Readout Buffer Trigger Level 2 Decision Decision Data transfer High-Level Trigger Subevent Buffer Local Data Concentrator (LDC) Decision Decision Data Global DataCollector (GDC) Event Building Event Buffer Transient Data Storage (TDS) Permanent Data Storage (PDS) Storage NA49-future Meeting, January 26, 2007
ALICE Detector Data Link NA49-future Meeting, January 26, 2007
DDL architecture Front-End Read-Out Standard detector/DAQ interface Source Interface Unit Forward Channel (Raw data) Backward Channel (Pedestals, control) Detector Data Link (DDL) : • Source Interface Unit • Transmission media • Destination Interface Unit LDC Destination Interface Unit DAQ Read-out Receiver Card (D-RORC) 100 MHz 64-bit PCI card PCI Bus NA49-future Meeting, January 26, 2007
DAQ Read-Out Receiver Cards (D-RORC) • PCI-X adapter for 2 DDL • PCI master:autonomous DMA NA49-future Meeting, January 26, 2007
Readout System Performance • Motherboard with dual Xeon CPUs @ 2.4 GHz • Six PCI-X slots, 4 bus segments (3+1+1+1), 2 controllers • Linux OS • ALICE Data-Acquisition software (DATE) NA49-future Meeting, January 26, 2007
Performance: 6 D-RORCs #4 PCI #6 #2 #3 PCI #5 #2 PCI #4 PCI #3 Controller #1 Segment #1 PCI #2 PCI #1 • Testing the fully populated PC using data source internal to PCI interface • Interoperability test • Measure the maximal input bandwidth Aggregate Bandwidth [MB/s] Normalized Bandwidth [MB/s/Ch] NA49-future Meeting, January 26, 2007
DAQ software • DAQ software • DDL software we can freely use • DAQ framework (DATE) does we need it? • Performance Monitoring (AFFAIR) • Data quality monitoring (MOOD) • Information dissemination • Documentation: 4 DAQ sw packages + ECS: fully documented. User’s guide released and printed • Regular DAQ trainings (70 people in last 3 years) • Linux • Linux SLC4 now. • Process defined to produce, distribute, install new versions • Code management system: CVS. Release packaging and distribution: RPM (~20 MB) • Automatic installation of Linux and DATE on DAQ nodes NA49-future Meeting, January 26, 2007
Memory Management’s Principle RORC Ready FIFO data length transfer status Page aligned continuous user memory outside Linux’s memory space, reachable for DMA and user as well. (physmem module necessary) Free blocks for events or event fragments Maximum block size: 2Mbyte Firmware Free FIFO start address block size Index of Ready FIFO Transfer status possible values: ffffffff unloaded (set by sw) 00000000 loaded, no DTSTW (set by fw) else loaded, DTSTW (set by fw) Free FIFO / Ready FIFO depth: 128 NA49-future Meeting, January 26, 2007
The Free FIFO D-RORC PC memory bank Firmware Free FIFO page address page address page address PC CPU readout Allocation of free pages NA49-future Meeting, January 26, 2007
Direct Memory Access D-RORC PC memory bank DDL Firmware PC CPU No involvement NA49-future Meeting, January 26, 2007
The Ready FIFO D-RORC PC memory bank page status page status page status address address address DDL Firmware Ready FIFO PC CPU readout Delivery of filled pages NA49-future Meeting, January 26, 2007
DDL Software • All functions accessible as interactive commands or API • Script-based interpreter for sequence of operations: • Sending command to the FEE • Reading FEE status • printing the status • comparing the status • polling the status • Downloading data into the FEE from a file • Reading data from the FEE • writing data into a file • comparing data with data in a file • TPC configuration: < 0.3 s • DDL performance • Parallelism FERO DDL D-RORC LDC define pedestal_addr 0x1FFF define enable_pedestal 0x2C reset SIU write_command enable_pedestal write_block pedestal_addr pedestal.hex %x read_and_check_block pedestal_addr pedestal.hex %x NA49-future Meeting, January 26, 2007
DATE Run Control • Control • Configuration • Main parameters • Display • SMI State of all nodes • Main counters of all nodes • Compatible with ALICE ECS NA49-future Meeting, January 26, 2007
DATE Configuration Database • Database content • DATE RolesActors of DATE system:LDCs, GDCs… • TriggerTrigger masks • DetectorsFront-end equipment of LDCs • Event building controlEvent building rules • BanksMemory banks to operate DATE NA49-future Meeting, January 26, 2007
DATE Infologger (1/2) • Online view • Online selection of visible fields • Online selection of message displayed NA49-future Meeting, January 26, 2007
DATE Infologger (2/2) • Offline view • Queries in the message archives • Selection of visible fields and of message displayed NA49-future Meeting, January 26, 2007
Data quality monitoring: MOOD • MOOD: Monitoring Of Online Data • DATE + ROOT environments • MOOD framework • Interfaces to detector code • Applications: • Raw data integrity • Detector performance NA49-future Meeting, January 26, 2007
Performance monitoring: AFFAIR • Individual nodes view • CPU usage • Input/Ouput • Node status • System view / Aggregated performances • Event building bandwidth • Event numbers NA49-future Meeting, January 26, 2007
Event Building Switch • ALICE baseline: • TCP/IP over switched Ethernet • Computing Data Challenges • Event-Building Switch • CERN frame contracts • Needs of IT and experiments • Good prices • On site maintenance (company+IT) • Force 10 Model 1200 NA49-future Meeting, January 26, 2007
Qualification of event-building switch NA49-future Meeting, January 26, 2007
Storage Arrays Performance • Local disk buffer at Pt2 • Storage network: • Fibre Channel currently FC 2G or 4G(Brocade,QLogic) • 16 ports switches with 4 uplinks for interconnect NA49-future Meeting, January 26, 2007
Storage Arrays Performance • Transient Data Storage • Storage arrays(e.g. Infortrend A16F 2 FC 2G ports, 16 SATA II HD) • Performance for a RAID 5 set of 5 disks NA49-future Meeting, January 26, 2007
DDL and DATE for NA49-future? (1/2) • Data transfer • DDL and D-RORC produced and used for longtime • Match speed requirements • Few (< 5) PCs are enough for the task • We have the knowledge how to implement to NA49-future • About 1 man-year of work • Problems with sub-detectors using VME • In ALICE exist VME boards where DDL SIU cards are attached, but interfaces are specific to the given ALICE sub-detector • Or the design VME – DDL interface is necessary for TOF and CAMACs • Interconnect with the trigger system • Trigger “box” has to be designed NA49-future Meeting, January 26, 2007
DDL and DATE for NA49-future? (2/2) • DAQ software • Software (DATE V5) released and documented.Process defined to produce, distribute, install new versions. • Everything (ECS, monitoring, etc.) included • Can we get it from ALICE? • They give it only with support • They can not support us during this year(integration with 17 ALICE sub-detectors) • Modifications necessary if TOF and CAMACs are used without the DDL • Is it shooting at sparrows with a high caliber gun ? • DAQ fabric hardware • Event building switch is necessary • Simple storage network has to be added NA49-future Meeting, January 26, 2007
Thank you NA49-future Meeting, January 26, 2007
Detector Data Link (DDL) • Detector readout: fast data transfer to PC memory • Electronics configuration: pedestals download • Interface and data-transfer detector/DAQ • Radiation tolerant card All components are radiation tolerant including FPGA (ACTEL) • 10 year doses in ALICE (at TPC inner radius): total ionising dose 1.6 krad neutron fluence 3.9·1011 cm-2 charged hadron fluence 8·109 cm-2 • ACTEL ProASIC adopted as baseline NA49-future Meeting, January 26, 2007
DATE V5 Software ALICE Data Acquisition and Test Environment (DATE): software framework for the ALICE DAQ system • Latest DATE version V5: • DATE V5.x kits:RPM package (~20 MB), User’s Guide (~470 pages) • Platforms:IA32 family, SLC4 using a 2.6 kernel • Development:~20 packages, mostly in C, using CVS • Auxiliary Software:DIM, SMI, MySQL, Tcl/Tk, libshift, ROOT, VMEbus driver • Testing:reference system, on-line data challenges, test beams DATE V5 Packages NA49-future Meeting, January 26, 2007