First release of D ata A cquisition B ackbone C ore

1. First release ofData Acquisition Backbone Core J. Adamczewski-Musch, H.G. Essel, N. Kurz, S. Linev GSI Darmstadt, Germany Experiment Electronics: Data processing group

2. FAIR accelerators Atomic Physics SIS 100 SIS18 Upgrade p-linac SIS 300 SIS100 SIS300 p-linac HADES & CBM PANDA SuperFRS Unilac Antiproton Prod. Target HESR HESR High Energy Exp. Super- FRS Plasma Phys. Low Energy Exp. p-bar target Atomic Phys. FLAIR Accelerator RESR RESR CR CR Experiment NESR Exp. NESR FAIR baseline* *FAIR Monthly, April 2009

3. Transition Radiation Detectors Electro- magnetic Calorimeter Ring Imaging Cherenkov Detector Silicon Tracking System Projectile Spectator Detector (Calorimeter) Micro Vertex Detector Resistive Plate Chambers (TOF) Dipole magnet CBM experiment at FAIR

4. CBM DAQ Conventional DAQ Detector FEE buffer L1trigger L1trigger L1trigger L1trigger Readoutbuffer Switch FLES HLT HLT HLT HLT HLT HLT HLT Processorfarm Storage DAQ architectures Self-triggered Front-endall hits shipped to DAQ.Data push architecture Fast links Readout buffer outside radiation area. Many Gbyte storage easily possible. Allows L1 decision times of 10-100 ms High-throughputEvent building First event selectiondone in processor farm.

5. InfiniBand performance studies > 500 MBytes/s bidirectional + 25% thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz

6. Hardware tests nXYTER, 128-channel self-triggered readout chip with few ns time resolution, DEFNI collaboration CBM Readout controller (ROC), Kirchhoff Institut für Physik, Heidelberg

7. Future DAQ requirements • Flexible – adopt different kinds of soft- and hardware • Compact – use only necessary code • Scalable – from small detector tests to 100-nodes clusters • Performing – low framework overhead • Multiprocessing & multithreading

8. Thread1 Thread2 Thread3 Thread4 Multithreading void MainLoop() { while (IsWorking()) { … writeSomething(); … } } void MainLoop() { while (IsWorking()) { … doSomething(); … } } runs runs Deadlock! void MainLoop() { while (IsWorking()) { … calcSomething(); … } } void MainLoop() { while (IsWorking()) { … readSomething(); … } } runs runs

9. Thread1 Thread2 Thread3 Thread4 Events queues Events queues Events queues Events queues Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor • Events • Timeouts • Commands • Events • Timeouts • Commands • Events • Timeouts • Commands • Events • Timeouts • Commands Multithreading in DABC runs runs ev1 runs runs ev3 ev2

10. Memory management dabc::Buffer dabc::MemoryPool References dabc::Buffer References Memory blocks dabc::Buffer References Multiple references on resources (blocks) Copy of reference without copy of data Zero copy transfer: PCI/DMA → buffer → IB/DMA

11. dabc::Module dabc::PoolHandle dabc::Parameter dabc::Timer dabc::Port dabc::Port dabc::Port User code - modules • dabc::Module provides: • I/O ports for communications • Pools handles to request memory • Timers for timeouts processing • Configuration & monitoring parameters

12. UserModule “Pool” “Output0” “Input” “Output1” Two kind of modules: sync & async Constructors mainly the same CreatePoolHandle("Pool", 2048, 1); CreateInput("Input", Pool(), 5); CreateOutput("Output0", Pool(), 5); CreateOutput("Output1", Pool(), 5); fCnt = 0; Explicit loop in ModuleSync Event processing in ModuleAsync void UserModule::MainLoop() { while (ModuleWorking()) { dabc::Buffer* buf = Recv(Input()); if (fCnt++ % 2 == 0) Send(Output(0), buf); else Send(Output(1), buf); } } void UserModule::ProcessIOEvent(dabc::Port*) { while (Input()->CanRecv() && Output(fCnt % 2)->CanSend()) { dabc::Buffer* buf = Input()->Recv(); Output(fCnt++ % 2)->Send(buf); } }

13. UserModule UserModule UserModule “Pool” “Pool” “Pool” “Output0” “Output0” “Output0” “Input” “Input” “Input” “Output1” “Output1” “Output1” Transport & device – dataflow node1 File Transport Transport Transport Transport Transport DataServer PCIeDevice NetworkDevice • Transport: • manages buffers queue • runs in own thread • decouples user code from actual transport functionality • Device: • represents hardware items • manages several transports node2 Transport Transport NetworkDevice File

14. Many other features • Central dabc::Manager class • Configuration with XML files • Controlling interface and GUI • dabc::Commands • Factories (plugins) architecture • Application and state machine • Event building network (BNET)

15. DABC release 1.0 Download via http://dabc.gsi.de DABC Controls Plugins Applications Core socket IB verbs* bnet mbs bnet-mbs ROC/udp ROC/ABB* core-test net-test bnet-test mbs bnet-mbs ROC Slim (batch) DIM Java GUI Plugins: Implementation of device/transports and experiment specific data handling code (programmers) Applications: Mainly setup or testing programs (users) ROC: CBM Readout Controller board ABB: Active buffer board (PCIe) IB: InfiniBand MBS: Multi Branch System (GSI DAQ) * external packages needed

16. roc::Readout Sorter Combiner Input0 Output0 Input0 Output0 Output1 Input1 Output1 Input2 calibrated data raw data DABC in CBM test beam STS GEM online go4 monitor

17. ROC/udp plugin User access layer DABC ROC/PCIe plugin DABC as access layer to ROC Ethernet Users scripts, GUIs file I/O online monitor optic PCIe

18. Planned CBM beam tests Readout boards Combiner boards Detectors FEE Commodity PCs Copper Optic InfiniBand

19. Explicit config for PCIe board Rnd Rnd Rnd Rnd Rnd Rnd Rnd Rnd Libs for all applications Defaults for all workers Event building network test Event builders Receivers Senders InfiniBand Readouts Worker1 PCIe Worker2 Worker3 Controller Worker4 Rnd Rnd Rnd Rnd

20. Conclusion • DABC is general-purpose DAQ framework • Can be used for different purposes – from small detectors tests to multi-nodes application • Through its plugin architecture can be easy extended to specific needs • Open for improvements and new ideas

21. Thanks!