LOFAR – Wide Area Network

1. LOFAR – Wide Area Network LOFAR – Network Experiences Klaas Stuurwold Roel Gloudemans Peter Maat

2. LOFAR Network Overview LOFAR – Arms • ~ 45 Stations • 2.5Gb/s per station in 4 approx. 600 Mb/s data streams • Max distance Station to core is 80km LOFAR – Core • ~ 32 Stations • 2 - 12Gb/s per station in max 12 1Gb/s Mb/s data streams • Max distance Station to core is 80km Data characteristics • Raw Ethernet or IP/UDP • Unidirectional • Jumbo frames • Per station a few large data streams • Point to point connections • Monitoring and control ~ 100Mb/s per station • LOFAR Partner data ~ 100Mb/s per station • Max latency: 10ms • Availability min. 95% • Upgradeable: • Longer arms • Higher data rates

3. Initial approach: I • Use COTS equipment • Experience readily available • Cheap • Short order to deployment time • Flexible • Use 1Gb/s technology combined with CWDM • Cheapest option • Use switches and no media converters • Optimal flexibility, no replugging needed on RFP board or port • Cheap, one switch with multiple ports costs about as much as one media converter • However, networks for different goals not isolated by default • Use VLAN’s to separate data from management and partners • Separate IP address spaces possible • Separation of network traffic • Bandwith control or Quality of Service per VLAN possible on most switches • Use multi-link trunking to combine several 1Gb/s ports into one logical port • No need for Spanning tree protocol which generates network overhead • One single logical interface to configure

4. Initial Approach: II • See Poster

5. Unexpected Challenges: I • Configuration errors might make a remote station unaccessible. • In the beginning it was thought that not much configuration would be needed • In practice, changes where happening every week if not day • Result: Regular occurrence of “terminal” miss-configuration. • Solution: Add “out-of-band” management equipment for switches and servers • For network equipment a separate data-path is needed • RS-232 to Ethernet boxes are readily available. Often with extended features which prove to be extremely useful. e.g. the possibility to connect door switches and temperature sensors • The Jumbo frame “standard” • Jumbo frames can be defined according to the Ethernet II standard, or the IEEE802.3 standard • The difference: In the IEEE standard the VLAN tag is included in the 9000 bytes, thus the data frame must be smaller. Result: No data transmission if part of the network is using the Ethernet II standard with max data frame size. • Solution: Check specs carefully before buying.

6. Unexpected Challenges: II • Real Unidirectional traffic • The data receiving stations do no transmit any ethernet traffic. This prevents the switches from learning their MAC addresses. • Thus all data is broadcasted on all ports • So when one station starts transmitting the station becomes unreachable (if we didn’t have VLAN’s the whole network would become unreachable) • Solutions: • Create a static MAC address table on the switch. Surprisingly few switch brands support this. It is also error prone. • Schedule a network ping on the data stations. Works very well, but is not possible with a Blue Gene • The tree is not fair • We want our switches to be able to operate on full speed on all ports • Nowadays this is possible on most 1Gb/s switches if you plug right (Cisco is the most famous exception) • On the switch mainboard there is often one ASIC per couple of ports. These ports can communicate to each other on full speed all the time. However these ASIC’s are grouped under another ASIC, this puts a limitation on ASIC to ASIC traffic. • Some of these switches are sold as Full Speed Non-Blocking! • Solution: Read switch specs very careful and agree on a return policy. Don’t trust cheap switches!

7. Unexpected Challenges: III • Multi-Link trunking, is the pain worth the gain? • One sender-receiver data stream cannot be balanced over the links in a trunk • Thus, on a trunk consisting of 2 links, it is only possible to have 2 600Mb/s data streams and not 3. • Data streams are not balanced over the links according to the current load of the link. An algorithm with sender-receiver MAC/IP addresses or protocol numbers is used. • Solutions: • Use 10Gb/s networking • Choose source and destination addresses wisely. Obtain the balancing algorithm from the vendor. (Very often this is classified information!) • Buy cheap, pay later • In our experience the firmware of cheap switches often contain blocking bugs • The performance of cheap switches is not always what you’d expect from the documentation. Fair tree and/or CPU issues • The better (more expensive) switches often have features that help you overcome unexpected challenges • Price and MTBF are related

8. The 10G story: I • The results are preliminary. We had some problems with our special “Glow-in-the-dark” ™ fibre. This limited the connection speeds to max 8Gb/s. • 10G networking, a solution for: • Trunking; not needed at station level for LOFAR. One 10G lightpath is an ideal mechanism to connect stations abroad. • The amount of fibres/muxes needed per station • Built in overhead for future growth • Less maintenance/less complicated infrastructure • At the same price (or less!). Short range (10km) optical at 2000 euro per switch port • Modern premium brand switches are able do 10Gb/s full speed non blocking. • The addition of 2 10Gb/s switches between 2 computer systems which where connected directly before only added 0.3ms extra delay • Modern PCI-X based systems are able to almost saturate a 10Gb/s link with TCP/UDP traffic if protocol offloading cards are used. (e.g. Myrinet cards; use Jumbo frames!).

9. The 10G story: II • Connections between networks often require filtering or some form of address translation • We were not able to to find any differences in network performance when filter rules where added to the switch. • When filtering was done at the PC level, with <10 rules no difference, but with 255 rules, the performance dropped to 7.8Gb/s • Fibre quality is important. A fibre which performs well on 1Gb/s might not perform with 10Gb/s. No clear problems could be detected except for a few bad packets and inconsistent test results.