1 / 36

VLBI Data Transfer Tests Recent and Current Work.

VLBI Data Transfer Tests Recent and Current Work. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. Outline. Throughput Tests on Mark5s TCP Memory-2-memory tests CPU Load tests Data delay on a TCP link – How suitable is TCP?

baina
Download Presentation

VLBI Data Transfer Tests Recent and Current Work.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VLBI Data Transfer TestsRecent and Current Work. Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  2. Outline • Throughput Tests on Mark5s • TCP Memory-2-memory tests • CPU Load tests • Data delay on a TCP link – How suitable is TCP? • 4th Year MPhys Project • Stephen Kershaw & James Keenan • The effect of distance • Throughput on the 630Mbit JB-JIVE UKLight Link • TCP Performance ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  3. Jodrell’s VLBI Mark5 Problem • Why were Jodrell to JIVE VLBI data transfers not able to do 512 Mbit even on UKLight ? • Why can Onsala Mk5 achieve 512 Mbps to JIVE Mk5 ?Onsala can even high rates transatlantic – iGrid2005 SC|05? • Identical Mk5 hardware to JBO • Same kernel and drivers • Longer links • Hint given as the general Network load increased: • Normally Onsala – JIVE iperf TCP ~900-950 Mbit/s VLBI OK at 512 Mbit • Sometimes Onsala – JIVE iperf TCP ~750 Mbit/s VLBI not OK at 512 Mbit Is it the network ? ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  4. VLBI Network Topology ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  5. VLBI Network Topology ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  6. TCP Tests Jodrell’s Mark5 • Standard Mark5 PCs 1.2GHz PIII • End host iperf TCP flow memory-to-memory only • 960 Mbit/s with rtt 1 msJBO - Manchester • Falls to 770 Mbit/s when rtt 15 ms JBO - JIVE • JBO - Manchester • 94.7% kernel mode idle 1.5 % • JBO - JIVE • 96.3% kernel mode idle 0.05 % • No Loss No Timeouts • 200* more TCPPureACK seen for JBO-Manchester • TCPHPACKs about the same • Help with meanings please ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  7. JBO – Manchester Asus NCCH-DL2.8 GHz Xeon • TCP Throughput constant as priority increases • % Kernel and %nice constant • No Loss No Timeouts TCP Throughput & CPU Load • JBO – Manchester 1.2 GHz PIII • TCP Throughput falls as priorityincreases • % Kernel mode drops and %nice increases ad priority increases • CPU mode shares with %nice • No Loss No Timeouts • Measure iperf TCP throughput and CPU load • Run CPU intensive task with different priority (nice High number = low priority) Onsala has a Faster Clock ! ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  8. TCP Throughput while reading SuperStor • Reading SuperStor from disk to memory only 1.48 Gbit/s • Reading SuperStor with iperf 1.15 Gbit/s • Iperf TCP rate 420 Mbit/s • 15 ms SS read spacing~1Gbit/s to memory • Corresponding CPU load • Measure iperf TCP throughput while reading data from disk to memory ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  9. TCP Delay and VLBI Transfers Manchester 4th Year MPhys Project by Stephen Kershaw & James Keenan ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  10. Timestamp3 Timestamp4 Packet loss Timestamp5 Receiver Sender Data3 Data4 RTT ACK Segment time on wire = bits in segment/BW Time • Remember Bandwidth*Delay Product BDP = RTT*BW VLBI Application Protocol TCP & Network Sender Receiver Timestamp1 Timestamp2 Data1 Data2 ●●● • VLBI data is Constant Bit Rate • tcpdelay • instrumented TCP program emulates sending CBR Data. • Records relative 1-way delay Time ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  11. Check the Send Time Send time – 10000 packets • 10,000 Messages • Message size: 1448 B • Wait time: 0 • TCP buffer 64k • Slope 0.44 ms/message • Expect 42 messages/rtt~0.6ms/message Send time sec 1 sec Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  12. Send Time Detail 26 messages About 25 us One rtt Message 76 Send time sec Message 102 100 ms Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  13. 1-Way Delay 1 way delay – 10000 packets • 10,000 Messages • Message size: 1448 B • Wait time: 0 • TCP buffer 64k 1 way delay 100 ms Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  14. 1-Way Delay Detail • 10,000 Messages • Message size: 1448 B • Wait time: 0 • TCP buffer 64k = 1 x RTT 26 ms 1 way delay 100 ms = 1.5 x RTT • Why not 1 rtt? • Why does it vary? • Effect of “send time delay”TCP slow start? ≠ 0.5 x RTT Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  15. Comparison of Send Time & 1-way delay 26 messages Message 102 Message 76 Send time sec 100 ms Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  16. 1 way delay – 10000 packets Packet 1214 1 way delay μs ~ 5.5 x RTT 1575 packets Packet number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  17. 1-Way Delay 724 byte msg • 10,000 Messages • Message size: 724 Bytes • Wait times: 20, 25, 30, 35, 40, 45 μs • TCP buffer 64k 1 way delay 100 ms Message number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  18. 1-Way Delay 724 bytes Detail • 10,000 Messages • Message size: 724 Bytes • Wait times: 20, 25, 30, 35, 40, 45 μs • TCP buffer 64k • Regular cycle of ~125 packets 1 way delay 100 ms Packet number ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  19. 1-Way Delay 1448 byte msg • Route:Man-ukl-ams-prod-man • Rtt 27ms • 10,000 Messages • Message size: 1448 Bytes • Wait times: 0 μs • DBP = 3.4MByte • TCP buffer 10MByte 50 ms Message number • Web100 plot • Starts after 5.6 Secdue to Clock Sync. • ~400 pkts/10ms • Rate similar to iperf ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  20. 5 ms 1-Way Delay with packet drop • Route:LAN gig8-gig1 • Ping 188us • 10,000 Messages • Message size: 1448 Bytes • Wait times: 0 μs • Drop 1 in 1000 Message number 28 ms 800 us ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  21. TCP on the 630 Mbit Link Jodrell – UKLight – JIVE ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  22. TCP Throughput on 630 Mbit UKLight • Manchester gig7 – JBO 606 • 4 Mbyte TCP buffer • test 0 • Dup ACKs seen • Other Reductions • test 1 • test 2 ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  23. Any Questions? ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  24. More Information Some URLs 1 • UKLight web site: http://www.uklight.ac.uk • MB-NG project web site:http://www.mb-ng.net/ • DataTAG project web site: http://www.datatag.org/ • UDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net • Motherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ • TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html& http://www.psc.edu/networking/perf_tune.html • TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 • PFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ • Dante PERT http://www.geant2.net/server/show/nav.00d00h002 ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  25. More Information Some URLs 2 • Lectures, tutorials etc. on TCP/IP: • www.nv.cc.va.us/home/joney/tcp_ip.htm • www.cs.pdx.edu/~jrb/tcpip.lectures.html • www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS • www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm • www.cis.ohio-state.edu/htbin/rfc/rfc1180.html • www.jbmelectronics.com/tcp.htm • Encylopaedia • http://www.freesoft.org/CIE/index.htm • TCP/IP Resources • www.private.org.il/tcpip_rl.html • Understanding IP addresses • http://www.3com.com/solutions/en_US/ncs/501302.html • Configuring TCP (RFC 1122) • ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt • Assigned protocols, ports etc (RFC 1010) • http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  26. Backup Slides ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  27. Latency Measurements • UDP/IP packets sent between back-to-back systems • Processed in a similar manner to TCP/IP • Not subject to flow control & congestion avoidance algorithms • Used UDPmon test program • Latency • Round trip times measured using Request-Response UDP frames • Latency as a function of frame size • Slope is given by: • Mem-mem copy(s) + pci + Gig Ethernet + pci + mem-mem copy(s) • Intercept indicates: processing times + HW latencies • Histograms of ‘singleton’ measurements • Tells us about: • Behavior of the IP stack • The way the HW operates • Interrupt coalescence ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  28. Sender Receiver Zero stats OK done Send data frames at regular intervals Inter-packet time (Histogram) ●●● ●●● Time to receive Time to send Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Signal end of test OK done Time Number of packets n bytes  time Wait time Throughput Measurements • UDP Throughput • Send a controlled stream of UDP frames spaced at regular intervals ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  29. Gigabit Ethernet Probe CPU CPU NIC NIC PCI bus PCI bus chipset chipset mem mem Logic Analyser Display Possible Bottlenecks PCI Bus & Gigabit Ethernet Activity • PCI Activity • Logic Analyzer with • PCI Probe cards in sending PC • Gigabit Ethernet Fiber Probe Card • PCI Probe cards in receiving PC ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  30. “Server Quality” Motherboards • SuperMicro P4DP8-2G (P4DP6) • Dual Xeon • 400/522 MHz Front side bus • 6 PCI PCI-X slots • 4 independent PCI buses • 64 bit 66 MHz PCI • 100 MHz PCI-X • 133 MHz PCI-X • Dual Gigabit Ethernet • Adaptec AIC-7899W dual channel SCSI • UDMA/100 bus master/EIDE channels • data transfer rates of 100 MB/sec burst ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  31. “Server Quality” Motherboards • Boston/Supermicro H8DAR • Two Dual Core Opterons • 200 MHz DDR Memory • Theory BW: 6.4Gbit • HyperTransport • 2 independent PCI buses • 133 MHz PCI-X • 2 Gigabit Ethernet • SATA • ( PCI-e ) ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  32. Network switch limits behaviour • End2end UDP packets from udpmon • Only 700 Mbit/s throughput • Lots of packet loss • Packet loss distributionshows throughput limited ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  33. 10 Gigabit Ethernet: UDP Throughput • 1500 byte MTU gives ~ 2 Gbit/s • Used 16144 byte MTU max user length 16080 • DataTAG Supermicro PCs • Dual 2.2 GHz Xenon CPU FSB 400 MHz • PCI-X mmrbc 512 bytes • wire rate throughput of 2.9 Gbit/s • CERN OpenLab HP Itanium PCs • Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz • PCI-X mmrbc 4096 bytes • wire rate of 5.7 Gbit/s • SLAC Dell PCs giving a • Dual 3.0 GHz Xenon CPU FSB 533 MHz • PCI-X mmrbc 4096 bytes • wire rate of 5.4 Gbit/s ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  34. mmrbc 512 bytes mmrbc 1024 bytes mmrbc 2048 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update mmrbc 4096 bytes 5.7Gbit/s 10 Gigabit Ethernet: Tuning PCI-X • 16080 byte packets every 200 µs • Intel PRO/10GbE LR Adapter • PCI-X bus occupancy vs mmrbc • Measured times • Times based on PCI-X times from the logic analyser • Expected throughput ~7 Gbit/s • Measured 5.7 Gbit/s ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  35. Tests on the UKLight switched light-path Manchester : Dwingeloo • Throughput as a function of inter-packet spacing (2.4 GHz dual Xeon machines) • Packet loss for small packet size • Maximum size packets can reach full line rates with no loss, and there was no re-ordering (plot not shown). ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

  36. UKLight using Mk5 recording terminals ESLEA VLBI Bits&Bytes Workshop , 4-5 May 2006, R. Hughes-Jones Manchester

More Related