1 / 18

ROC-based DAQ: latest developments and perspectives

ROC-based DAQ: latest developments and perspectives Jörn Adamczewski-Musch, Hans G. Essel, Sergey Linev GSI, Experiment Electronics: Data Processing group. Optics ROClib version 2 Preparation for beamtime in June 2010 BNet demonstrator Throttling. ROClib - status end 2009.

jared-kinney
Download Presentation

ROC-based DAQ: latest developments and perspectives

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ROC-based DAQ: latest developments and perspectives Jörn Adamczewski-Musch, Hans G. Essel, Sergey LinevGSI, Experiment Electronics: Data Processing group • Optics • ROClib version 2 • Preparation for beamtime in June 2010 • BNet demonstrator • Throttling

  2. ROClib - status end 2009 • Only nXYTER frontend was existing and supported • Connection between ROC and PC via Ethernet • ROC GUI was disabled • rocutil was the only tool for configuration of the ROC • DABC DAQ application for readout of several ROCs • go4monitor for simple online/offline analysis of data, taken during beamtime • Worked reliable during beamtime 2009 S.Linev ROC-based DAQ: status and perspectives

  3. Optic developments • Three kind of traffic in one media: • data • control • DLM • Required extra hardware: • AVNet board • Required extra software (included in ROClib): • PCI driver (root account) • mprace library • abbdaemon application • Lot of tests last several month: • stable work in fixed configuration, bis 250 MB/s datarate • problems with simultaneous usage of both ports • already supported in ROClib, some improvements required S.Linev ROC-based DAQ: status and perspectives

  4. New functionality in ROClib v2.0 • Goal for v 2.0 – optics as mainstream, Ethernet as fallback solution • New functionality in ROC FPGA code: • support of new frontend (FEET) • programmable commands lists • new message format (little endian) • reorganization of control address space (breaks compatibility) • On the PC side: • support of the new FPGA features • device/transport classes for optic • new roc::Message format – support both old and new formats • roc::Iterator class for access data from different data sources • rocGui for ROC/nXYTER/FEET configurations • go4monitor can be connected directly to the ROC (without DAQ app) S.Linev ROC-based DAQ: status and perspectives

  5. New roc::Message and roc::Iterator classes • roc::Message is replacement for old nxyter::Data class • Supports both old (v1.x) and new (v2.x) message formats • roc::Iterator class allows to get data: • from ROC via Ethernet (via roc::Board class) • from ROC via Optic (also via roc::Board class) • from LMD files (replacement for fileapi) roc::Board* brd = roc::Board::Connect(“optic://abb0”); brd->startDaq(); roc::Iterator iter(brd); int cnt = 0; while (iter.next() && (cnt++<1000)) { iter.msg().printData(); } brd->stopDat(); roc::Board::Close(brd); S.Linev ROC-based DAQ: status and perspectives

  6. rocGui • Qt4-based • Only for configurations • DAQ with go4monitor • Can configure: • ROC itself • nXYTER frontend • FEET frontend • Works with optic and Ethernet • Able to produce scriptsfor rocutil • Modular design, easy to extend S.Linev ROC-based DAQ: status and perspectives

  7. Migration from v1.x to 2.0 • Version 2.0 is not backward compatible with 1.x • Old LMD files are supported in v2.0 • Final version 2.0 will be announced together with exact instruction for upgrade • Exact sequence to upgrade firmware/software on the ROC: • checkout new ROClib in separate location; • with old rocupload program update first firmware (not forget jumpers): • rocupload cbmtest01 –fw ~/roclib2/firmware/Release/FEET02000004_ETHERNET02000002.bit • with old rocupload program update PowerPC software: • rocupload cbmtest01 –sw ~/roclib2/sw-host/release/image.bin • reboot ROC • compile new ROClib and try to use it with the ROC • rocGui cbmtest01 • For more info see README.txt and cbm-wiki.gsi.de S.Linev ROC-based DAQ: status and perspectives

  8. FEB1nxGen BEAMAUX... FEB4nx SiCBM01B2 FEB4nx SiCBM01B2 FEB4nx SiCBM01B2 ROC ROC ROC ROC ROC ROC ROC ROC ROC ROC ROC ROC SFP Eth SFP SFP SFP Eth SFP SFP SFP SFP SFP SFP InfiniBand switch A A A A A A A A A A A A FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 FEB4nxSiDem1 CBMSTSDemo 1 SYNC-S SYNC-S SYNC-S SYNC-S SYNC-S SYNC-S SYNC-S SYNC-S B B B B B B B B B B B B AUX AUX AUX AUX AUX AUX AUX AUX AUX AUX AUX AUX Data combiner board Data combiner board Data combiner board SFP SFP SFP SFP SFP SFP CBMSTSDemo 1 SFP SFP SFP SFP SFP SFP MBS SFP SFP SFP SFP SFP SFP Eth DABC node3 DABC node2 DABC node1 DABC node4 Readout Event building Filtering File store Readout Event building Filtering File store Readout Event building Filtering File store Readout Event building Filtering File store IB/ PCIe IB/ PCIe IB/ PCIe IB/ PCIe SFP/ PCIe SFP/ PCIe SFP/ PCIe Eth Beamtime - plans end 2008 S.Linev ROC-based DAQ: status and perspectives

  9. FEB4nx SiCBM01B2 FEB4nx SiCBM01B2 FEB4nx SiCBM01B2 ROC ROC ROC ROC ROC ROC ROC ROC SFP Eth SFP SFP SFP Eth SFP SFP A A A A A A A A SYNC-S SYNC-S SYNC-S SYNC-S SyncS SyncM SyncS SyncS B B B B B B B B AUX AUX AUX AUX AUX AUX AUX AUX DABC node2 DABC node1 DABC node4 DABC node3 Readout Event building Filtering File store Readout Event building Filtering File store Readout Event building Filtering File store Readout Event building Filtering File store IB/ PCIe IB/ PCIe IB/ PCIe IB/ PCIe SFP/ PCIe SFP/ PCIe SFP/ PCIe SFP/ PCIe Beamtime in June 2010 – realistic plan FEB1nxGen BEAMAUX... InfiniBand switch Any other frontend ? S.Linev ROC-based DAQ: status and perspectives

  10. Beamtime in June 2010 • Necessary components: • Up to 8 ROCs, 4 ABBs • SyncM/SyncS cabling • 4x Opteron-IB DAQ cluster with 4 TB storage • DABC with ROC-ABB and ROC-BNet plugin • Several PCs for monitoring • Still to do: • reliable work of ABB with both SFPs • modifications in ROC-ABB plugin • Intensive ROC-BNet tests • can be done in next two months • Fallback solution (if optic fails) • Same electronics, but readout via Ethernet • Reduced datarates • With or without BNet S.Linev ROC-based DAQ: status and perspectives

  11. CBM DAQ perspectives - dataflow FEE Readout Buffering BNet FLES Switch ~1 TB/s of raw data over network Provides combined time slices to the FLES Throttle data in case of buffers/networks overflow SLES, storage S.Linev ROC-based DAQ: status and perspectives

  12. BNet simulations with SystemC (2005) S.Linev ROC-based DAQ: status and perspectives

  13. InfiniBand performance tests (2006-2007) S.Linev ROC-based DAQ: status and perspectives

  14. DABC development (since 2007) • General purpose DAQ framework • Data-flow core • threads, device & transports • user modules, application • Configuration, control, monitoring, GUI • Provides components for custom event building network - BNet • Real data sources – MBS, ROC/UDP, ROC/Optic, generic PCIe board driver • Used as DAQ for CBM 2008/2009 beamtimes • Mid- and short-term plans: • Improved command interface (deadlocks recognition) • Support of other kinds of control system (EPICS) • Connection to cluster/job management system S.Linev ROC-based DAQ: status and perspectives

  15. BNet demonstrator • Since 2007: • better understanding of traffic/datarates • more realistic trigger algorithms were proposed • DABC was established • One can launch activity for new BNet demonstrator: • with more realistic traffic generators • with ~100-200 nodes InfiniBand cluster (~20% of final CBM) • use DABC-based infrastructure • emulation of FLES activity (CPU consumption) • Aim of demonstrator: • where InfiniBand technology now, that is near perspectives? • that kind of traffic shaping we need, if any? • is time-scheduled transport gives us significant benefits? • reliability and errors recovery issues? S.Linev ROC-based DAQ: status and perspectives

  16. CBM DAQ perspectives - throttling FEE Readout Buffering BNet No backpressure until BNet. How to throttle data when data rates higher than output link performance? FLES SLES, storage S.Linev ROC-based DAQ: status and perspectives

  17. E1 E5 E2 E6 E3 E7 E4 E8 Throttling algorithm • While all messages are time-stamped, one can provide very simple throttle algorithm, based on absolute epoch number. For instance: • drop every 8-th epoch, if not enough • drop every 4-th epoch, if not enough • drop every 2-nd epoch, if not enough • drop everything, but first epoch • Any other dropping pattern can be implemented • Algorithm required memory for at least last 8 epoch • no problems in readout buffer or BNet • What to do in FEE, where buffer capacity is limited? S.Linev ROC-based DAQ: status and perspectives

  18. Throttling in FEE • Doing same as in readout buffers and Bnet – requires large buffer space and complex arbitration logic, implemented in hardware => too complicated • As proposed by W. Müller on last DAQ meeting, set busy bit (flip-flop) if FIFO full and reject any new data until next epoch. Efficiency of such algorithm is highly depends from relation between FIFO buffer space, epoch length (E) and output link performance (RATE): • can reasonably work only when FIFO << E * RATE, but potentially high data lost • if FIFO ≈ E * RATE, it is about 50% probability if data of even or odd epoch will be excluded by the FEE chip => completely unusable • if FIFO >> E * RATE, id of dropped epochs will vary from chip to chip, no any real correlation, also unusable • The only way to drop data in the FEE – introduce message counter per epoch. If amount of data in current epoch exceeds limit, all consequent messages in current epoch will be dropped. Guarantees, that at least in the epochs begin all data are preserved. Open question – per-channel or per-chip logic? S.Linev ROC-based DAQ: status and perspectives

More Related