Event Building With Smart NICs

1. Event Building With Smart NICs Jean-Pierre Dufey, Beat Jost, Niko Neufeld & Marianna Zuin DAQ 2000Lyon, October 20, 2000

2. Recap: LHCb DAQ System Niko NEUFELD CERN, EP

3. Event Building Components • Readout units (RU): multiplexing of front-end links, destination assignment • Switching read-out network • Sub-farm controllers (SFC): event building and event dispatching Niko NEUFELD CERN, EP

4. Event Building Properties • Static load balancing among the SFCs • RUs send round robin to destinations destination = f(event_number) f being the same for all RUs • Pure push protocol • congestions handled via flow control and more importantly by throttling • Distributes the event data flow of 6 GB/s from m sources to n destinations, each of which has to handle O(1Kb) fragments at 80 kHz Niko NEUFELD CERN, EP

5. Modern Smart NICs are powerful embedded computers Off-load general purpose CPU Take advantage of cheap CPU power on the NIC Facilitate hardware design of the RU (Yet) limited CPU power compared to commodity PC No guarantee that high-end NIC development will continue in this direction (firmware/CPU vs. ASIC/FPGA) Why Use Smart NICs? Niko NEUFELD CERN, EP

6. Alteon Tigon 2 • Features • Dual R4000-class processor running at 88 MHz • Up to 2 MB memory • GigE MAC+link-level interface • PCI interface • Development environment • GNU C cross compiler with few special features to support the hardware • Source-level remote debugger Niko NEUFELD CERN, EP

7. PC/Linux PC/Linux CPU CPU GbE NIC GbE NIC NIC Mem Mem NIC PCI PCI CERN Network Test Setup Niko NEUFELD CERN, EP

8. NIC 2 NIC Performance Niko NEUFELD CERN, EP

9. Performance of Alteon NIC • Can fill the wire at any given frame size (from 64 to 9000 bytes) • Can send out frames at a frequencies of up to 1.4 MHz • For frames bigger than 512 bytes more than 95% of nominal bandwidth available for data (practically 100% for >8000 Jumbo frames) Niko NEUFELD CERN, EP

10. Event Building Algorithm • Assembles events out of fragments from a known number of sources • Handles an adjustable amount of events concurrently (limited only by buffer space) • Implements “Implicit + Time-out Completion” • Uses “scatter/gather” capabilities of NIC’s DMA engine to concatenate the fragments into the host’s memory Niko NEUFELD CERN, EP

11. Start Start Start Start Procedure Procedure Polling Polling Polling Polling New fragment New fragment New fragment New fragment New New New New NO NO NO NO YES YES YES YES event fragment event fragment event fragment event fragment ? ? ? ? Add new event Add new event Add new event Add new event Event Event Event Event descriptor descriptor descriptor descriptor YES YES YES YES NO NO NO NO still in the still in the still in the still in the ? ? ? ? table table table table Fragment Fragment Fragment Fragment Collect the Collect the Collect the Collect the out of time out of time out of time out of time fragment fragment fragment fragment Decrement Decrement Decrement Decrement sources sources sources sources Check for missing fragments Check for missing fragments Check for missing fragments Check for missing fragments in previous events in previous events in previous events in previous events Algorithm Niko NEUFELD CERN, EP

12. PC Test Implementation 400 MHz PIII VC++ 5.0 Niko NEUFELD CERN, EP

13. Performance NIC 2 NIC Average time per fragment 11.65 us Niko NEUFELD CERN, EP

14. Summary • Event building on a smart NIC at a frequency of incoming fragments of almost 100 KHz has been demonstrated • Event building at Gigabit speed for fragments bigger than ~1100 bytes • Code Optimization ongoing (9 us/frag have already been achieved) Niko NEUFELD CERN, EP

15. Program of Work • Evaluate impact of interrupt coalescence on SFC performance • Study possibility of handling some amount of TCP/IP traffic on the outgoing link of the SFC (events to storage) • “Real world” tests on a Gigabit Ethernet switching network • Use measured parameters in a detailed simulation of the readout network Niko NEUFELD CERN, EP