1 / 36

Legnaro Event Builder Prototypes

Legnaro Event Builder Prototypes. Luciano Berti, Gaetano Maron. INFN – Laboratori Nazionali di Legnaro. GE Event Builder. Components: Hardware: switch: FoundryNet FastIron NIC: SysKonnect SK9821 PC: Supermicro PIII (i840) Software: vxWorks based. 15 x 15. Test conditions:

lecea
Download Presentation

Legnaro Event Builder Prototypes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Legnaro Event Builder Prototypes Luciano Berti, Gaetano Maron INFN – Laboratori Nazionali di Legnaro CPT Week CERN, 23 April 2001

  2. GE Event Builder • Components: • Hardware: • switch: FoundryNet FastIron • NIC: SysKonnect SK9821 • PC: Supermicro PIII (i840) • Software: vxWorks based 15 x 15 • Test conditions: • No command or event aggregation (each packet transports a command or data frame relative to a single event) • full data transfer from/to PC memory • recovery from packet loss • fixed fragment sizes are varied 400-4000 bytes CPT Week CERN, 23 April 2001

  3. Event builder layout RUs 1 3 4 5 7 9 12 13 2 6 8 10 11 14 15 EVM RU performance problem found with this configuration Slot 2 Slot 4 Slot 1 Slot 3 BUs 1 3 4 5 7 9 12 13 2 6 8 10 11 14 15 • RUs and BUs distributed in all switch slots: • Part of the traffic localized within the slot • Reduces switch backplane utilization CPT Week CERN, 23 April 2001

  4. Modified Event Builder layout Request data commands RUs 1 3 4 5 7 9 12 13 2 6 8 10 11 14 15 EVM Slot 2 Slot 4 Slot 1 Slot 3 Fast Ethernet Slot Request data commands BUs 1 3 4 5 7 9 12 13 2 6 8 10 11 14 15 • RU fast control message over FE (PCI 32/33) • RU data transfer on GE (PCI 64/66) CPT Week CERN, 23 April 2001

  5. The GE Event Builder CPT Week CERN, 23 April 2001

  6. EB protocol RUs BU 1 2 3 n EVM allocate confirm send cache CPT Week CERN, 23 April 2001

  7. Concurrent building threads in the same BU RUs BU 1 2 3 n EVM BU thread 1 BU thread 2 BU thread 3 CPT Week CERN, 23 April 2001

  8. Sequential vs Random reading Sequential reading Random reading RUs RUs 1 2 3 n BU EVM 4 5 1 2 3 n BU EVM allocate allocate confirm confirm send send cache cache CPT Week CERN, 23 April 2001

  9. “Sliding Window” • multiple send to Rus • reduce the total rebuilding time • less events in the Bus • not yet tested RUs 4 5 1 2 3 n BU EVM allocate confirm send cache CPT Week CERN, 23 April 2001

  10. Sequential - random reading comparison • No difference on performance • But more allocated event needed on BUs, • All the measurements with random reading Random reading Sequential reading CPT Week CERN, 23 April 2001

  11. Recovery from Packets loss BU – EVM communication BU – RU communication timer BU EVM timer BU RU Req. EvtId Req. Data start start EvtId EvtData Timeouts 80 - 160 ms timeout timeout Req. EvtId (retry) Req. Data ( retry ) start start EvtId cancel cancel EvtData CPT Week CERN, 23 April 2001

  12. EVB 15x15 performance - Throughput 15 x 15 • Throughput up to 116 MB/s, ie 93% link speed • no packet loss observed (as expected) CPT Week CERN, 23 April 2001

  13. EVB Scaling CPT Week CERN, 23 April 2001

  14. EVB Performance – Event Rate 15 x 15 Nominal fragment size 2kbytes: event rate = 52 kHz CPT Week CERN, 23 April 2001

  15. Conic Event Builder conic EVB symmetric EVB RU RU Event Event Builder Network Builder Network Manager Manager FU FU FU FU FU FU FU FU FU FU FU FU BU FU • faster ports at Rus • slower ports at BUs FU FU FU CPT Week CERN, 23 April 2001

  16. Conic Event Builder: Layout RUs Request Data Command 1 3 4 2 GE Slot 1 EVM FE Slot 1 FE Slot 2 1 5 7 9 13 17 23 25 33 35 3 11 15 19 21 27 29 31 37 39 FUs 2 6 8 10 14 18 24 26 34 36 4 12 16 20 22 28 30 32 38 40 CPT Week CERN, 23 April 2001

  17. EVB throughput – Conic vs Symmetric 4 x 40 conic EVB: no performance degradation vs symmetric CPT Week CERN, 23 April 2001

  18. EVB Conic – Scaling 4 x 40 1 x 10 2 x 20 CPT Week CERN, 23 April 2001

  19. Conic: RU/FU Throughput ratio 1 x n 4 x n 2 x n CPT Week CERN, 23 April 2001

  20. To be done and test • variable size events • EB performances with the new implemented “ window” mechanism • latency times measurements • Fault generation with the new implemented Random Error Generator to check the error recovery procedure CPT Week CERN, 23 April 2001

  21. Multistage Event Builder • All our results have been obtained with a single switch event builder configuration • We propose to extend our tests to a multistage ethernet switches topology and to study the behavior of this configuration. CPT Week CERN, 23 April 2001

  22. Plain Topology RUs • In the Event Builder application data flows in only one direction • The inter-switch Gigabit Ethernet links are full-duplex • Result : half of the inter-switch bandwidth available is wasted BUs CPT Week CERN, 23 April 2001

  23. Full Mesh Topology RUs BUs • RU and BU distributed in all the switches • Inter-switch links are used in both direction • Same number of ports of the plain topology • Twice of the bandwidth of the plain topology in the inter-switch links CPT Week CERN, 23 April 2001

  24. Plain and mesh topology limits • Each couple of switches is connected by a single link • This is a bottleneck if the traffic is not uniformly random • The network is blocking for certain traffic patterns CPT Week CERN, 23 April 2001

  25. Traffic with patterns • If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic. • This mechanism exist and it is called Universal Routing CPT Week CERN, 23 April 2001

  26. Universal Routing Reference • Discovered by L.G. Valiant in 1980 • See: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available on : http://www.pact.srf.ac.uk/macrame/papers/bluebook.html • Those papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based network • We adapted the same concept to a packet switched network like Gigabit Ethernet CPT Week CERN, 23 April 2001

  27. Universal Routing with GigaEthernet • Based on Clos topology • Multiple path available between each couple of switches • Every packets is sent to a randomly chosen intermediate switch • The intermediate switch send the packet to the final destination • Full bandwidth between each couple of switches and uniform buffer utilization CPT Week CERN, 23 April 2001

  28. Universal Routing RUs RUs BUs BUs • Transformation of the CLOS topology to a folded CLOS • The resulting number of ports is the same of the plain topology Full duplex links Half duplex links CPT Week CERN, 23 April 2001

  29. Large (500x500) multistage GE network (1) 1 20 BUs • 25 switches with 60 x 1Gb ports • 20 switches with 25 x 1 Gb ports 1 20 RUs 2 3 2 4 40 Ports 25 20 20 Ports 25 Ports CPT Week CERN, 23 April 2001

  30. Large (500x500) multistage GE network (2) 20 BUs 1 20 RUs • 25 switches with 40 x 1Gb ports + 2 x 10 Gb uplinks • 2 switches with 25 x 10 Gb ports 1 2 2 25 Ports 10G 25 40 Ports 2 Ports 10G CPT Week CERN, 23 April 2001

  31. Proposal for a multistage event builder demonstrator • Multistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigations • The proposal is to have prototypes for: • Full Mesh Topology • Folded CLOS topology with (and without) Universal Routing mechanism CPT Week CERN, 23 April 2001

  32. Full Mesh 64x64 Event Builder Prototype Missing components - 1 host node 4 Rus / 4 BUs or a mix of them - 32 hosts - 128 FE NICS (56 + 72) - 8 24 FE ports switch 8 RUs 8 BUs 1 2 8 7 3 6 4 5 CPT Week CERN, 23 April 2001

  33. Folded CLOS 64x64 Event Builder Prototype Missing components 64x64 - 1 host node 4 Rus / 4 BUs or a mix of them - 32 hosts - 128 FE NICS (56 + 72) - 4 48 FE ports switch - 2 24 FE ports FastIron module + 1 24 FE ports FastIron module 1 16 BUs 1 16 RUs 2 3 2 4 48x48 - 1 host node 3 Rus / 3 BUs or a mix of them - 32 hosts - 96 FE NICS (56 + 40) - 4 36 FE ports switch - 2 24 FE ports FastIron module 3 4 32 Ports 16 16 Ports 4 Ports FastIron with 3 24 FE ports mods CPT Week CERN, 23 April 2001

  34. Folded CLOS 80x80 Event Builder Prototype Missing components 80x80 - 1 host node 4 Rus / 4 BUs or a mix of them - 40 hosts (32+8) - 160 FE NICS (56 +104) - 4 48 FE ports + 2 GE links switch - 1 8 GE (Base SX) ports FastIron module 20 BUs 1 20 RUs 1 2 3 2 4 40 FE Ports FastIron with 8 GE ports module (1000 BaseT or 1000 BaseSX) 2 GE Ports 4 Ports CPT Week CERN, 23 April 2001

  35. 8x80 Conic Event Builder Prototype Missing components 8x80 RU1 - 1 host node 4 FUs - 20 hosts - 80 FE NICS (56 + 24) - 4 24 (48) FE ports + 2 GE up links switch - 2 8 GE (Base SX) ports FastIron module 20 FUs 1 RU2 RU3 2 RU4 RU5 3 RU6 RU7 4 20 FE Ports RU8 2 GE Ports FastIron with 2 8 GE ports modules (1000 BaseT or 1000 BaseSX) CPT Week CERN, 23 April 2001

  36. Material for the event builder multistage prototypes Mesh 64x64 - 72 FE NICs - 8 24 FE ports • Folded CLOS 80x80 • - 8 PCs • - 104 FE NICS • - 4 48 FE ports with 2 GE uplinks 1000 baseT • if the 1000 baseT uplinks are not available: • 1) Folded CLOS 64x64: • 72 FE NICs • 4 48 FE ports switch • 1 24 FE ports FastIron module • 2) Folded CLOS 48x48: • 40 FE NICs • 4 >36 FE ports switch CPT Week CERN, 23 April 2001

More Related