Network Processors for a 1 MHz Trigger-DAQ System

1. Network Processors for a 1 MHz Trigger-DAQ System RT2003, Montreal Artur Barczyk, Jean-Pierre Dufey, Beat Jost and Niko Neufeld CERN-EP & Université de Lausanne

2. Network Processors • Developed for high-end routers, on the market since 1999 • Dedicated processors optimised for high speed packet processing • Large I/O capabilities (up to 10 Gigabit/s), and up to 10 Mp/s • Large and fast buffer memories • Software programmable  Use them in a network based DAQ system, wherever PCs can’t do it (easily) Niko NEUFELD CERN, EP

3. HW Assist Routing and Bridging Tables Packet BufferMemory Anatomy of a Network Processor On-chip Memory + Interfaces for external memories Control and Monitoring Scheduler GeneralPurpose CPU ProcessorComplex Integrated Network Interfaces Multiple RISC processor cores • Several hardware threads • Coprocessors for many commonnetworking tasks SearchEngine MAC/FRAMEProcessor To and From PHYs Niko NEUFELD CERN, EP

4. All infrastructure to operate one IBM PowerNP NP4GS3 3 x 1000 BaseT ports One port converted into PCI, for development purposes 2 NPs can be connected via special cable Build by S3 corp., Ireland NP module as PCI card Niko NEUFELD CERN, EP

5. Frame Merging Decision Sorting Multi-stages switching Network Event Building Sorter TRM FE FE FE FE FE FE FE FE FE FE FE FE NP NP SFC NP NP NP NP SFC SFC SFC SFC SFC NP Readout Network Switch Switch Switch Network Processors in a 1 MHz DAQ Front-end Electronics 125-239Links 1.1 MHz 8.8-16.9 GB/s 349Links 40 kHz 2.3 GB/s Multiplexing Layer Switch 30 Switches Switch 77-135 NPs 24 NPs 77-135 Links 6.4-13.6 GB/s 24 Links 1.5 GB/s L1-Decision TFCSystem 73-140 Links 7.9-15.1 GB/s Event Builder 37-70 NPs 50-100 Links 5.5-10 GB/s 50-100 SFCs Niko NEUFELD CERN, EP

6. Frame Merging Works up to 4 MHz of incoming packets  A. Barczyk’s presentation Works for at least 2 x 100 MB streams Niko NEUFELD CERN, EP

7. Frame Merging • Helps to optimise link usage • Reduces number of links into readout network • Can do re-formatting of data – e.g. protocol adaptation (raw Ethernet  IP) • Can change Maximum Transmission Unit (MTU) • some Ethernet segments provide payload > 1500 bytes • Reduce packet rate at output - important for receiving PCs (interrupt rate!) Niko NEUFELD CERN, EP

8. Building your own switching network from NP modules • Using NP modules gives you full freedom in doing the switching • Large output buffers • Disadvantage: • Module has only eight ports (otherwise switch chip is needed)  need a large number of modules to build a big network • Solution: • Use optimised connection topologies to reduce number of elementary modules, while keeping the load on interconnecting links acceptable Niko NEUFELD CERN, EP

9. Network Topologies • “Fully Connected” Topology • Banyan Topology 64 x 64 port configuration Basic Structure Sources 63 Sources x 72 Destinations Destinations Niko NEUFELD CERN, EP

10. Decision Sorting • In the LHCb trigger decisions are generated as small Ethernet packets in one of 1400 PCs  1 MHz of un-ordered decisions in • Processing time limited but unknown  decisions are taken and sent in arbitrary order • Front-end electronics requires decisions to be ordered before sent to the trigger distribution system  1 MHz of ordered decisions out • Limited buffer size entails maximum trigger latency • Each event entering is made known to central entity (Decision Sorter)  1 MHz of frames Niko NEUFELD CERN, EP

11. 3 4 Readout Network 1 1 2 2 1 2 3 4 Switch Switch Switch Switch Switch 2 4 CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Decision Sorting 3 4 Front-end Electronics FE FE FE FE FE TRM FE TFCSystem L1-Decision 90-153 Links 5.5-10 GB/s 6 5 90-153 SFCs SFC SFC SFC SFC SFC CPUFarm ~1400 CPUs 1 3 Sorter Niko NEUFELD CERN, EP

12. Test Set-up Measurement Procedure • Dual NP connected via back-plane to form 8-port module • Download code into NP4GS3 via RISC Watch (JTAG) or PCI of PPC control point • Generate traffic either via Gigabit Ethernet NICs (Tigon) or using one NP to feed the other • Can use the internal timers of the NP and/or the NICs 4 x Tigon2 1000 SX 8 x 1000 SX Full Duplex Ports Tigon2 NIC Features • Up to 620 kHz fragment rate • 1 s resolution timer IBM NP4GS3R2.0 Reference Kit PPC 750 RISC Watch = JTAG via Ethernet Niko NEUFELD CERN, EP

13. Packet processing for several millions of packets per second Fast and big buffer memories Hardware assists for many common tasks, like check-summing, re-framing, tree look-ups Software programmable Processing power optimised for header region of packets Memory model optimised for the hardware (no linear addressing) Programs need to be written in proprietary assembly language Network Processors Niko NEUFELD CERN, EP

14. Conclusions • Network Processors are a powerful tool for packet processing • They are especially useful, whenever very high rates of packets need to be coped with • We have found a lot of useful applications, all could be done with the same standard NP module – the software defines the functionality • But what if PCs can do it too…? Niko NEUFELD CERN, EP

15. Backup Slides Niko NEUFELD CERN, EP

16. Data flow in the NP4GS3 DASL DASL Access to frame data Access to frame data Ingress Event Building Egress Event Building Niko NEUFELD CERN, EP