MB – NG SuperJANET4 Development Network

1. WorldCom MCC Manc SJ4 Dev C-PoP Warrington 12416 Leeds UCL ULCC SuperJANET4 Production Network Gigabit Ethernet 2.5 Gbit POS 2.5 Gbit POS core SJ4 Dev C-PoP London 12416 SJ4 Dev C-PoP Reading 12416 WorldCom Dark Fibre (SSE) RAL / UKERNA RAL / UKERNA MB – NG SuperJANET4 Development Network MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

2. Some Edge (Bbone) Router requirements • Connect to the test systems in the IP domain. • Accept marked packets and be able to re-mark them • Mark the packets on input according to at least: • Source – dest IP address IP port • Layer3 and Layer4 headers TCP/UDP • Support Diffserve code points • Perform policing and admission control • Provide a range of Queue type options – • Several queues with different priorities • Real-time queue • Provide flexible Queue Scheduling • Weighted round robin • Weighted fair queuing • Provide Congestion control such as • RED • ECN • Be able to assign MPLS labels to the packets in a flexible manner. MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

3. Gigabit Ethernet 2.5 Gbit POS 2.5 Gbit POS core Generate MPLS frames QoS – WRED MPLS Classify packets QoS Difserv MPLS Multi-loop Load PC Test PC Test PC 4 * 1 Gigabit Ethernet blade MPLS 4 * 1 Gigabit Ethernet blade QoS Marking Test PC 10 Gbit POS Load PC Test PC Load PC Test PC 3 * 1 Gigabit Ethernet blade 2.5 Gbit POS MPLS 2.5 Gbit POS MPLS 10 Gbit POS Load PC Supervisor Supervisor Load PC IP or MPLS SJDN Core MPLS Backbone Router 12416 Difserve router OR MPLS Edge Router MPLS Backbone Router OSR Equipment for the External / Edge DomainsFunctionality MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

4. Monitor PC 3508 Ethernet Switch 3512 Ethernet Switch 3512 Local site Ethernet Local site Ethernet Equipment for the External / Edge DomainsConnectivity Gigabit Ethernet 2.5 Gbit POS 2.5 GbitPOS core Spy Load PC Test PC Test PC 4 * 1 Gigabit Ethernet blade MPLS 4 * 1 Gigabit Ethernet blade QoS Marking Test PC Test PC Load PC MPLS to core 2.5 Gbit POS MPLS Load PC Supervisor Supervisor Load PC Difserve router OR MPLS Edge Router MPLS Backbone Router OSR MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

5. IP Access to MB-NG SystemsA suggestion only – we all must decide ! • End/Test systems routing: • The default route from the PC is set to the Gigabit link i.e. into the development network. • Static routes for researchers only are set for the 100 Mbit interface, e.g. to Manchester HEP, MCC, RAL, UCL, ULCC, UKERNA. • The link between the Cisco 3512 and the Campus is set to 10 Mbit – this would minimise any unintentional traffic. • IP access to core routers for researchers only • Similar approach? via the test PCs at the CPoP MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

6. Some Questions ??? • Can one Gigabit Ethernet blade perform the IP and MPLS functions we need or do we require two cards – one for IP input functions, the other for MPLS output? •  What is the performance of the 4-port Gigabit Ethernet interfaces? • Confirm that an interface can support both IP and MPLS frames at same time. • Can the GSRs use MPLS to loop the frames round n times? • Can all our “requirements be” met? MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

7. Focus on the routers - Hardware • Cisco OSR based on 6500 Are the details correct ? More memory MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

8. Focus on the routers - Summary MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

9. High Bandwidth High Throughput Protocol Investigations (mainly) EthernetSwitch 3508 Test PC 4 * 1 Gigabit Ethernet blade Monitor port Monitor PC NIC 2.5 Gbit POS Test PC 4 * 1 Gigabit Ethernet blade Monitor PC NIC NIC 2.5 Gbit POS Hardware & Software Tools • Optical splitter or opto-electrical tap • opto-electrical tap constructed for logic analyser • Software use of : • tcpdump tcptrace • Web100 MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

10. Latency Tests with 64-bit PCI • Alteon Gigabit NIC + RAW • SuperMicro 370DLE • ServerWorks III LE Chipset • 64 bit PCI @ 33 MHz • Request – Response • Transfer rates: observe expect us/B 32+32 bit 0.0265 0.0232 -> 108Mbyte/s 32+64 bit 0.0209 0.0211 64+64 bit 0.0149 0.0156 64 byte Request 1400 byte Response MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

11. RAW Stream Not using IP stack Vary time between packets 1.6 factor improvement over 32-32 bit transfers Throughput Tests with 64-bit PCI MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

12. UDP Stream Vary time between packets 660-690 Mbit/s @ 17us No interrupt coalescing 15 μs between packets Throughput Tests UDP SysKonnect 1400 byte sent 1400 byte received Pci-pci latency 28.3 μs MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

13. UDP request - response Latency and transfer rates very similar to raw frames observe us/B offset us 64+64 bit 0.0225 46.88 64+64 bit 0.0149 54. (raw) Latency Tests UDP SysKonnect 1400 byte sent 10.66 μs pci time 1400 byte received 10.96 μs pci time Pci-pci latency 28.3 μs MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

14. UDP Stream Vary time between packets 880-930 Mbit/s @ 11us interrupt coalescing 15 μs between packets Throughput Tests UDP Alteon MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

15. UDP request – response Due to interrupt coalescing Latency Tests UDP Alteon MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester