1 / 0

Network Architecture for the LHCb DAQ Upgrade

Upgrade DAQ Miniworkshop. Network Architecture for the LHCb DAQ Upgrade. Guoming Liu CERN, Switzerland. May 27, 2013. Outlines. Introduction to the LHCb DAQ Upgrade: numbers Potential network technologies for the DAQ upgrade DAQ network architecture DAQ schemes Summary.

ceana
Download Presentation

Network Architecture for the LHCb DAQ Upgrade

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Upgrade DAQ Miniworkshop

    Network Architecture for the LHCb DAQ Upgrade

    Guoming Liu CERN, Switzerland May 27, 2013
  2. Outlines Introduction to the LHCb DAQ Upgrade: numbers Potential network technologies for the DAQ upgrade DAQ network architecture DAQ schemes Summary
  3. LHCb DAQ Upgrade Timeframe: installation in the second long shut-down of the LHC in 2018, be ready for data taking in 2019 Trigger: a fully flexible software solution. Low Level Trigger (LLT) : tune the input rate to the computing farm from 1 – 40 MHz when the system is not fully ready for 40 MHz The DAQ system should be capable of reading out the whole detector at the LHC collision rate of 40MHz. Numbers for the DAQ Network Event size: ~100 KB Max. event input rate: 40 MHz Unidirectional Bandwidth: ~38.4 Tbit/s (may scale up)
  4. Network Technologies High-speed interconnection technologies Ethernet (10G/40G/00G) InfiniBand (FDR, coming EDR) Some other similar technologies Ethernet Very popular for desktop/station/server Familiar by users/developers InfiniBand Mainly used in high performance computing and large enterprise data center High speed: 56Gb/s FDR Great performance/price
  5. Ethernet vs InfiniBand
  6. Review: Current LHCb DAQ Detector Readout board (TELL1): custom FPGA board UDP-like transport protocol: MEP (Multi-Event Packet) Push DAQ scheme Deep buffer is required in the routers and the switches VELO ECal HCal Muon RICH ST OT L0 Trigger FEE FEE FEE FEE FEE FEE FEE TFC / Readout Supervisor Readout Board Readout Board Readout Board Readout Board Readout Board Readout Board Readout Board Evt m Frag. Evt m Frag. Evt m Frag. Evtm, Dest n MEP Request Evtm, Dest n Core router 1 Core router2 Evtm, Dest n Event building SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU HLT farm CPU n: DataReq Event data Timing and Fast Control Signals
  7. Network Architecture for DAQ upgrdae Unidirectional solution: Dataflow in the core network is unidirectional Bidirectional mixed solution: Readout Unit (RU) & Builder Unit (BU) connected to the same Top-Of-Rack (TOR) switch, dataflow in the core network is bidirectional Bidirectional uniform solution: RU & BU combined in the same server, dataflow in the core network is bidirectional
  8. Unidirectional solution All the readout units are connected to the core network The builder unit and the filter unit are implemented in the same server. The dataflow in the core network is unidirectional
  9. DAQ: Core Network Monolithic core router vs fabric with fat-tree topology
  10. Ethernet vs InfiniBand Monolithic core-router (current solution in LHCb) pros: “simple” architecture, good performance cons: expensive, not many choices Fabric with fat-tree topology : many small Top-of-Rack (TOR) switches pros: cost-efficiency, scalability, flexibility cons: complexity Fabric is quite popular in data center: Cisco FabricPath, Juniper QFabri, and also other large chassis …
  11. Bidirectional mixed solution (1) The builder unit and the filter unit are implemented in the same server All the readout units are connected to the TOR switches instead of the core network.
  12. Bidirectional mixed solution (2) The dataflow in the core network is bi-directional Requires RUs and BU/FUs are close enough to connect the same TOR switch This can save up to 50% of bandwidth and ports in the core network. The price per port in the core network are usually 3 to 4 times more expensive than in a TOR switch
  13. Bidirectional uniform solution (1) The readout unit and builder unit are implemented in the same server (RU/BU server) The RU/BU server connects both the core network (for event building) and the TOR switch (for event filtering)
  14. Bidirectional uniform solution (2) The dataflow in the core network is bi-directional Saves up to 50% ports in the core network. Possible to choose different network technologies for the core layer (event builder network) and the edge layer (event filter network). e.g. cost-effective InfiniBand FDR for the core, low cost 10 GBase-T for the event filter network Increases the flexibility: deep buffer, easy to implement different DAQ schemes in software Not tied to any technology Reduces the complexity in the FPGA receiver card No deep buffer is needed Simple protocol (e.gPCIe) with PC
  15. Bidirectional uniform solution (3) Key to success of the uniform solution: the RU/BU module RU/BU modules serve five purposes: Receives data fragments from the front-end electronics Sends data fragments to the other modules Builds complete events Performs event filtering on a fraction of the complete events Distributes the remaining events to a sub-farm of filter units 1 2 3 4
  16. Bidirectional uniform solution (4) IO bandwidth requirements of RU/BU modules: Full 24x GBT link  ~ 154 Gb/s input and output or ~ 215Gb/s for wide user mode Preliminary tests on a Sandy-Bridge server Intel E5 2650: 2x16x2.0G 2x Mellanox dual-port InfiniBand FDR cards Connect-IB OS: SLC 6.2 Software: MLNX-OFED 2.0 Connect-IB cards send and receive data simultaneously
  17. Bidirectional uniform solution (5) Preliminary test results: input and output throughput MLNX-OFED 2.0 is a beta version, but needed for the new dual-port cards In MLNX-OFED 1.5.3, the throughput of the single-port card is close to the limit More tunings on OS and software are needed to improve the performance
  18. DAQ schemes Several different DAQ schemes in term of the data flow Push data without traffic shaping Push data with barrel-shift traffic shaping Pull data from the destinations Different schemes fit for different network technologies and topologies More details on Daniel’s talk later
  19. Summary Both Ethernet and InfiniBand, or a mix of both can be the candidate for the DAQ network upgrade Several architectures have been discussed, the uniform solution is the most flexible and cost-effective solution Preliminary tests show the uniform solution can work More studies for the LHCb DAQ network upgrade are needed, stay tuned for the development in industry
  20. Thank You !
More Related