1 / 32

The Network Transport layer and the Application or TCP/IP and VLBI Data

The Network Transport layer and the Application or TCP/IP and VLBI Data. Richard Hughes-Jones & Stephen Kershaw The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” and look for Haystack. Outline. Throughput Tests on Mark5s TCP Memory-2-memory tests CPU Load tests

duaa
Download Presentation

The Network Transport layer and the Application or TCP/IP and VLBI Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Network Transport layer and the ApplicationorTCP/IP and VLBI Data Richard Hughes-Jones & Stephen Kershaw The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” and look for Haystack 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  2. Outline • Throughput Tests on Mark5s • TCP Memory-2-memory tests • CPU Load tests • Data delay on a TCP link – How suitable is TCP? • 4th Year MPhys Project • Stephen Kershaw & James Keenan • The effect of distance • Throughput on the 630Mbit JB-JIVE UKLight Link • TCP Performance 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  3. VLBI Application Protocol • VLBI signal wave front • Want to examine the data wave front 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  4. TCP: Data moving - Filling the Pipe • TCP has to hold a copy of data in flight in case of re-transmission • Optimal (TCP buffer) window size depends on: • Bandwidth end to end, i.e. min(BWlinks) i.e. the bottleneck bandwidth • Round Trip Time (RTT) • The number of bytes in flight to fill the entire path: • Bandwidth*Delay Product BDP = RTT*BW • Can increase bandwidth by orders of magnitude Receiver Sender RTT ACK Segment time on wire = bits in segment/BW Time 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  5. Timestamp3 Timestamp4 Packet loss Timestamp5 Data3 Data4 VLBI Application Protocol • VLBI data is produced at Constant Bit Rate TCP & Network Sender Receiver Timestamp1 Timestamp2 Data1 Data2 • Want to examine how TCP moves Constant Bit Rate Data • tcpdelay a test program: • instrumented TCP program emulates sending CBR Data. • Records relative 1-way delay • Record TCP Stack activity with web100 ●●● Time 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  6. Send time sec 1 sec Message number Check the Message Send Times Manchester 4th Year MPhys Project Stephen Kershaw & James Keenan • 10,000 Messages • Message size: 1448 Bytes • Wait time: 0 • TCP buffer 64k • Route:Man-UKL-JIVE-prod-Man • RTT ~26 ms • Slope 0.44 ms/message • From TCP buffer size & RTT Expect: ~42 messages/RTT~0.6ms/message 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  7. 26 messages About 25 us One rtt Message 76 Send time sec Message 102 100 ms Message number Send Time Detail • TCP Send Buffer limited • After SlowStart theTCP Buffer is full • packets sent out in bursts each RTT • Program blocked on sendto() 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  8. 1 way delay 100 ms 100 ms Message number 1-Way Delay • 10,000 Messages • Message size: 1448 Bytes • Wait time: 0 • TCP buffer 64k • Route:Man-UKL-JIVE-prod-Man • RTT ~26 ms 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  9. = 1 x RTT 26 ms 1 way delay 10 ms = 1.5 x RTT 10 ms ≠ 0.5 x RTT Message number 1-Way Delay Detail • Why not just 1 RTT? • After SlowStart TCP Buffer Full • Messages at front of TCP Send Buffer have to wait for next ACKs burst – 1 RTT later • Messages further back in the TCP Send Buffer wait for 2 RTT 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  10. Problem #1 Packet Loss Is it important ? 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  11. 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  12. 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  13. t Packet loss Is TCP is suitable for use in transferring real-time eVLBI data, on a lossy network? 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  14. Packet loss 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  15. Packet loss 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  16. Problem #1 Packet Loss Is it important ? 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  17. 5 ms 1-Way Delay with packet drop • Route:B2B on LAN gig8-gig1 • Ping 188 μs • 10,000 Messages • Message size: 1448 Bytes • Wait times: 0 μs • Drop 1 in 1000 • Manc-JIVE tests showtimes increasing with a “saw-tooth” around 10 s 1 way delay 10 ms Message number 28 ms 800 us 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  18. Summary & Conclusions • Standard TCP not optimum for high throughput long distance links • Packet loss is a killer for TCP • Check on campus links & equipment, and access links to backbones • Users need to collaborate with the Campus Network Teams • Dante Pert • New stacks are stable and give better response & performance • Still need to set the TCP buffer sizes ! • Check other kernel settings e.g. window-scale maximum • Watch for“TCP Stack implementation Enhancements” • TCP tries to be fair • Large MTU has an advantage • Short distances, small RTT, have an advantage • TCP does not share bandwidth well with other streams • The End Hosts themselves • Plenty of CPU power is required for the TCP/IP stack as well and the application • Packets can be lost in the IP stack due to lack of processing power • Interaction between HW, protocol processing, and disk sub-system complex • Application architecture & implementation are also important • The TCP protocol dynamics strongly influence the behaviour of the Application. • Users arenow able to perform sustained 1 Gbit/s transfers 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  19. More Information Some URLs 1 • UKLight web site: http://www.uklight.ac.uk • MB-NG project web site:http://www.mb-ng.net/ • DataTAG project web site: http://www.datatag.org/ • UDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net • Motherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ • TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html& http://www.psc.edu/networking/perf_tune.html • TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 • PFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ • Dante PERT http://www.geant2.net/server/show/nav.00d00h002 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  20. More Information Some URLs 2 • Lectures, tutorials etc. on TCP/IP: • www.nv.cc.va.us/home/joney/tcp_ip.htm • www.cs.pdx.edu/~jrb/tcpip.lectures.html • www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS • www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm • www.cis.ohio-state.edu/htbin/rfc/rfc1180.html • www.jbmelectronics.com/tcp.htm • Encylopaedia • http://www.freesoft.org/CIE/index.htm • TCP/IP Resources • www.private.org.il/tcpip_rl.html • Understanding IP addresses • http://www.3com.com/solutions/en_US/ncs/501302.html • Configuring TCP (RFC 1122) • ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt • Assigned protocols, ports etc (RFC 1010) • http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  21. Any Questions? 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  22. Backup Slides 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  23. Receiver Sender Segment n Sequence 1024 Length 1024 RTT ACK of Segment n Ack 2048 Segment n+1 Sequence 2048 Length 1024 RTT ACK of Segment n +1 Ack 3072 Time TCP – providing reliability • Positive acknowledgement (ACK) of each received segment • Sender keeps record of each segment sent • Sender awaits an ACK – “I am ready to receive byte 2048 and beyond” • Sender starts timer when it sends segment – so can re-transmit • Inefficient – sender has to wait 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  24. Comparison of Send Time & 1-way delay 26 messages Message 102 Message 76 Send time sec 100 ms Message number 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  25. 1 way delay – 10000 packets Packet 1214 1 way delay μs ~ 5.5 x RTT 1575 packets Packet number 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  26. 1-Way Delay 724 byte msg • 10,000 Messages • Message size: 724 Bytes • Wait times: 20, 25, 30, 35, 40, 45 μs • TCP buffer 64k 1 way delay 100 ms Message number 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  27. 1-Way Delay 724 bytes Detail • 10,000 Messages • Message size: 724 Bytes • Wait times: 20, 25, 30, 35, 40, 45 μs • TCP buffer 64k • Regular cycle of ~125 packets 1 way delay 100 ms Packet number 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  28. Sender Receiver Zero stats OK done Send data frames at regular intervals Inter-packet time (Histogram) ●●● ●●● Time to receive Time to send Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Signal end of test OK done Time Number of packets n bytes  time Wait time Throughput Measurements • UDP Throughput • Send a controlled stream of UDP frames spaced at regular intervals 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  29. tcpdump / tcptrace • tcpdump: dump all TCP header information for a specified source/destination • ftp://ftp.ee.lbl.gov/ • tcptrace: format tcpdump output for analysis using xplot • http://www.tcptrace.org/ • NLANR TCP Testrig : Nice wrapper for tcpdump and tcptrace tools • http://www.ncne.nlanr.net/TCP/testrig/ • Sample use: tcpdump -s 100 -w /tmp/tcpdump.out host hostname tcptrace -Sl /tmp/tcpdump.out xplot /tmp/a2b_tsg.xpl 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  30. tcptrace and xplot • X axis is time • Y axis is sequence number • the slope of this curve gives the throughput over time. • xplot tool make it easy to zoom in 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  31. Zoomed In View • Green Line: ACK values received from the receiver • Yellow Line tracks the receive window advertised from the receiver • Green Ticks track the duplicate ACKs received. • Yellow Ticks track the window advertisements that were the same as the last advertisement. • White Arrows represent segments sent. • Red Arrows (R) represent retransmitted segments 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

  32. TCP Slow Start 5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester

More Related