1 / 30

TCP/IP Overview & Performance

MB - NG. TCP/IP Overview & Performance. Richard Hughes-Jones The University of Manchester. UKLIGHT. TCP (Reno) – What’s the problem?. TCP has 2 phases: Slowstart Probe the network to estimate the Available BW Exponential growth

tekla
Download Presentation

TCP/IP Overview & Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MB - NG TCP/IP Overview & Performance Richard Hughes-Jones The University of Manchester UKLIGHT e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  2. TCP (Reno) – What’s the problem? • TCP has 2 phases: • Slowstart Probe the network to estimate the Available BWExponential growth • Congestion AvoidanceMain data transfer phase – transfer rate glows “slowly” • AIMD and High Bandwidth – Long Distance networks Poor performance of TCP in high bandwidth wide area networks is due in part to the TCP congestion control algorithm. • For each ack in a RTT without loss: cwnd -> cwnd + a / cwnd - Additive Increase, a=1 • For each window experiencing loss: cwnd -> cwnd – b (cwnd) - Multiplicative Decrease, b= ½ • Packet loss is a killer !! e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  3. TCP (Reno) – Details • Time for TCP to recover its throughput from 1 lost packet given by: • for rtt of ~200 ms: 2 min e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  4. Investigation of new TCP Stacks • The AIMD Algorithm – Standard TCP (Reno) • For each ack in a RTT without loss: cwnd -> cwnd + a / cwnd - Additive Increase, a=1 • For each window experiencing loss: cwnd -> cwnd – b (cwnd) - Multiplicative Decrease, b= ½ • High Speed TCP a and b vary depending on current cwnd using a table • a increases more rapidly with larger cwnd – returns to the ‘optimal’ cwnd size sooner for the network path • b decreases less aggressively and, as a consequence, so does the cwnd. The effect is that there is not such a decrease in throughput. • Scalable TCP a and b are fixed adjustments for the increase and decrease of cwnd • a = 1/100 – the increase is greater than TCP Reno • b = 1/8 – the decrease on loss is less than TCP Reno • Scalable over any link speed. • Fast TCP Uses round trip time as well as packet loss to indicate congestion with rapid convergence to fair equilibrium for throughput. • HSTCP-LP, H-TCP, BiC-TCP e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  5. MB-NG Managed Bandwidth Packet Loss and new TCP Stacks • TCP Response Function • Throughput vs Loss Rate – further to right: faster recovery • Drop packets in kernel MB-NG rtt 6ms DataTAG rtt 120 ms e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  6. Packet Loss and new TCP Stacks UKLIGHT • TCP Response Function • UKLight London-Chicago-London rtt 180 ms • 2.6.6 Kernel • Agreement withtheory good e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  7. Send data with TCP Drop Packets Monitor TCP with Web100 man03 lon01 High Throughput Demonstrations London (Chicago) Manchester (Geneva) Dual Zeon 2.2 GHz Dual Zeon 2.2 GHz Cisco GSR Cisco GSR Cisco 7609 Cisco 7609 1 GEth 1 GEth 2.5 Gbit SDH MB-NG Core e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  8. MB-NG Managed Bandwidth High Performance TCP – MB-NG • Drop 1 in 25,000 • rtt 6.2 ms • Recover in 1.6 s Standard HighSpeed Scalable e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  9. High Performance TCP – DataTAG • Different TCP stacks tested on the DataTAG Network • rtt 128 ms • Drop 1 in 106 • High-Speed • Rapid recovery • Scalable • Very fast recovery • Standard • Recovery would take ~ 20 mins e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  10. On the way to Higher Bandwidth e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  11. End Hosts & NICs SuperMicro P4DP6 • Use UDP packets from udpmon to characterise Host & NIC • SuperMicro P4DP6 motherboard • Dual Xenon 2.2GHz CPU • 400 MHz System bus • 66 MHz 64 bit PCI bus Throughput Latency Bus Activity e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  12. Network switch limits behaviour • End2end UDP packets from udpmon • Only 700 Mbit/s throughput • Lots of packet loss • Packet loss distributionshows throughput limited e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  13. TCP Window Scale factor not set correctly • SC2004 London-Chicago-London tests • Server quality hosts – 2.8 GHz Dual Xeon; 133 MHz PCI-X bus • TCP window scale factor should allow the pipe to be filled • Delay*BW 22 Mbytes • Web100 output shows: • Cwnd dows not open • Data set at line speedbut as 1 burst/rtt • Data stops at Cwnd • Average throughput: 100 Mbit/s • Limited by sender • Kernel configuration problem e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  14. Network & Disk Interactions • Hosts: • Supermicro X5DPE-G2 motherboards • dual 2.8 GHz Zeon CPUs with 512 k byte cache and 1 M byte memory • 3Ware 8506-8 controller on 133 MHz PCI-X bus configured as RAID0 • six 74.3 GByte Western Digital Raptor WD740 SATA disks 64k byte stripe size • Measure memory to RAID0 transfer rates with & without UDP traffic CPU load Disk write 1735 Mbit/s Disk write + 1500 MTU UDP 1218 Mbit/s Drop 30% Disk write + 9000 MTU UDP 1400 Mbit/s e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  15. BaBar on Production network • Standard TCP • Average: 425 Mbit/s • DupACKs 350-400 – re-transmits iperf Throughput + Web100 • SuperMicro on MB-NG network • HighSpeed TCP • Average: Linespeed 940 Mbit/s • DupACK ? <10 (expect ~400) e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  16. Disk-Disk bbftp • bbftp file transfer program uses TCP/IP • UKLight: Path:- London-Chicago-London; PCs:- Supermicro +3Ware RAID0 • MTU 1500 bytes; Socket size 22 Mbytes; rtt 177ms; SACK off • Move a 2 Gbyte file • Web100 plots: • Standard TCP • Average 825 Mbit/s • Scalable TCP • Average 875 Mbit/s e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  17. Parameters to Consider – Only some of them! • Server quality hosts • Check that UDP packets use BW expected • Poor (old), or wrongly configured routers / switches • Overloaded access links – campus / country • Hunt down packet loss at your desired sending rate • Fill the pipe with packets in flight • set socket buffer to2* Delay*BW • Kernel configuration settings: • Allow large socket buffer (TCP window) settings • Set length of transmit queue large (~2000) • TCP window scale factor should allow the pipe to be filled • Disallow “Moderation” in TCP stack • Consider tuning off SACKs in 2.4.x and maybe up to 2.6.6 • Large MTUs – reduces CPU load • Enable Interrupt coalescence – reduces CPU load e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  18. Real Time TCP in e-VLBI e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  19. Does TCP delay the data ? • Work in progress !! • Send blocks of data (10kbytes) at regular intervals • Drop every 10,000 packet • Measure the arrival time of the data e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  20. More Information Some URLs • MB-NG project web site:http://www.mb-ng.net/ • DataTAG project web site: http://www.datatag.org/ • UDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net • Motherboard and NIC Tests: www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 • TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html& http://www.psc.edu/networking/perf_tune.html • TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  21. Backup Slides e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  22. UKLight in the UK UKLIGHT e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  23. MB-NG Managed Bandwidth Amsterdam SC2004 UKLIGHT Overview Manchester SLAC Booth SC2004 Cisco 6509 MB-NG 7600 OSR Caltech Booth UltraLight IP UCL network UCL HEP NLR Lambda NLR-PITT-STAR-10GE-16 ULCC UKlight K2 K2 Ci UKlight 10G Four 1GE channels Ci Caltech 7600 UKlight 10G Surfnet/ EuroLink 10G Two 1GE channels UKLIGHT Chicago Starlight K2 e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  24. Manchester Domain UKERNA DevelopmentNetwork UCL Domain Boundary Router Cisco 7609 Boundary Router Cisco 7609 lon02 lon01 lon03 HW RAID HW RAID Edge Router Cisco 7609 MB-NG ral02 ral02 RAL Domain Managed Bandwidth Key Gigabit Ethernet 2.5 Gbit POS Access MPLS Admin. Domains ral01 Boundary Router Cisco 7609 man02 man03 man01 Topology of the MB – NG Network e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  25. The Bandwidth Challenge at SC2003 • Peak bandwidth 23.21Gbits/s • 6.6 TBytes in 48 minutes Phoenix - Amsterdam 4.35 Gbit HighSpeed TCP rtt 175 ms , window 200 MB e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  26. Average Transfer Rates Mbit/s e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  27. bbftp: Host & Network Effects • 2 Gbyte file RAID5 Disks: • 1200 Mbit/s read • 600 Mbit/s write • Scalable TCP • BaBar + SuperJANET • Instantaneous 220 - 625 Mbit/s • SuperMicro + SuperJANET • Instantaneous 400 - 665 Mbit/s for 6 sec • Then 0 - 480 Mbit/s • SuperMicro + MB-NG • Instantaneous 880 - 950 Mbit/s for 1.3 sec • Then 215 - 625 Mbit/s e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  28. Applications: Throughput Mbit/s • HighSpeed TCP • 2 GByte file RAID5 • SuperMicro + SuperJANET • bbcp • bbftp • Apachie • Gridftp • Previous work used RAID0(not disk limited) e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  29. Host, PCI & RAID Controller Performance • RAID0 (striped) & RAID5 (stripped with redundancy) • 3Ware 7506 Parallel 66 MHz 3Ware 7505 Parallel 33 MHz • 3Ware 8506 Serial ATA 66 MHz ICP Serial ATA 33/66 MHz • Tested on Dual 2.2 GHz Xeon Supermicro P4DP8-G2 motherboard • Disk: Maxtor 160GB 7200rpm 8MB Cache • Read ahead kernel tuning: /proc/sys/vm/max-readahead e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

  30. RAID Controller Performance Write Speed Read Speed RAID 0 RAID 5 e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester

More Related