1 / 16

vlbi_udp Throughput Performance and Stability.

vlbi_udp Throughput Performance and Stability. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. Ring buffer. vlbi_udp Flow Diagram. Vlbi_control. vlbi_send. vlbi_recv. Control thread. Control thread. TCP Control. Disk. Memory. Disk. Memory.

brasen
Download Presentation

vlbi_udp Throughput Performance and Stability.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. vlbi_udpThroughput Performance and Stability. Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  2. Ring buffer vlbi_udp Flow Diagram Vlbi_control vlbi_send vlbi_recv Control thread Control thread TCP Control Disk Memory Disk Memory Input thread Output thread Ring buffer UDP Data Send thread Receive thread From Simon Casey ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  3. History or the story so far: • iGrid2002 monolithic code • Convert to use pthreads • control • Data input • Data output • Code branch for Simon’s file transfer tests • Work on vlbi_recv: • Output thread polled for data in the ring buffer – burned CPU • Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput • Output thread uses sched_yield() when no work to do – CPU used • Network performance – in progress ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  4. Instability: 2 streams into JIVE, March 2006 From Simon Casey ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  5. Early B2B UDP Tests • Kernel 2.6.6 (May06) • vlbi_recv semaphore • Wait 15 us; signal 200 events • Unstable throughput • ~815 Mbit/s • Packet loss ~ 0.05% • Lost at top of IP stack • Inter-packet time • Processing time • mean 0.1166 sigma 0.3493 • CPU load: L1 ; 0; 0; 0; 100; 0; 0; 0;; L2 ; 0; 0; 0; 99; 0; 0; 0;; L3 ; 0; 0; 0; 99; 0; 0; 0;; L4 ; 2; 0; 43; 53; 0; 1; 18 ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  6. Confirm with B2B UDP Tests • Kernel 2.6.6 (Aug06) • vlbi_recv semaphore • Wait 15 us; signal 100 events • Throughput a bit unstable • ~815 Mbit/s • Packet loss ~ 0.05% • Lost at top of IP stack • CPU number & % packet loss • Have loss when using same CPU as that servicing interrupts • Confirmed by many udpmon tests ! ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  7. B2B UDP Tests – different kernel • Kernel 2.6.9 • vlbi_recv semaphore • Wait 12 us; signal 200 events • Stable throughput • 999 Mbit/s variation less than 1 Mbit/s • No packet loss • Inter-packet time • Processing time • mean 0.1206 sigma 0.3274 • CPU load: Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.3% hi, 0.0% si Cpu1 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Cpu3 : 6.0% us, 17.0% sy, 0.0% ni, 40.3% id, 0.0% wa, 1.0% hi, 35.7% si ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  8. B2B UDP Tests • Kernel 2.6.9 • vlbi_recv sched_yield() • Wait 20 us • Stable throughput • 612.5 Mbit/s variation less than 1 Mbit/s • No packet loss • Inter-packet time • Processing time • mean 0.1005 sigma 0.1438 • CPU load: • Cpu0 : 3.7% us, 8.0% sy, 0.0% ni, 88.4% id, 0.0% wa, 0.0% hi, 0.0% si • Cpu1 : 1.0% us, 4.0% sy, 0.0% ni, 95.0% id, 0.0% wa, 0.0% hi, 0.0% si • Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si • Cpu3 : 11.6% us, 22.9% sy, 0.0% ni, 41.9% id, 0.0% wa, 1.0% hi, 22.6% si ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  9. B2B UDP Tests • Kernel 2.6.9 • vlbi_recv sched_yield() • Wait 12 us • Stable throughput • 999 Mbit/s variation less than 1 Mbit/s • No packet loss • Inter-packet time • Processing time • mean 0.1005 sigma 0.1438 • CPU load: • Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si • Cpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si • Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si • Cpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  10. Next Steps • Fix the timing precision – again! • Test with latest 2.6.16 and 2.6.17 kernels • Why does shed_yield() take more CPU time? Is it a problem or not? • Test on long-distance Networks • Production (packet switched) • UKLight • Test at 4-8 Gbit ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  11. Any Questions? ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  12. More Information Some URLs 1 • UKLight web site: http://www.uklight.ac.uk • MB-NG project web site:http://www.mb-ng.net/ • DataTAG project web site: http://www.datatag.org/ • UDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net • Motherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ • TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html& http://www.psc.edu/networking/perf_tune.html • TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 • PFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ • Dante PERT http://www.geant2.net/server/show/nav.00d00h002 ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  13. More Information Some URLs 2 • Lectures, tutorials etc. on TCP/IP: • www.nv.cc.va.us/home/joney/tcp_ip.htm • www.cs.pdx.edu/~jrb/tcpip.lectures.html • www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS • www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm • www.cis.ohio-state.edu/htbin/rfc/rfc1180.html • www.jbmelectronics.com/tcp.htm • Encylopaedia • http://www.freesoft.org/CIE/index.htm • TCP/IP Resources • www.private.org.il/tcpip_rl.html • Understanding IP addresses • http://www.3com.com/solutions/en_US/ncs/501302.html • Configuring TCP (RFC 1122) • ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt • Assigned protocols, ports etc (RFC 1010) • http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  14. Backup Slides ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  15. VLBI Network Topology ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

  16. Geant2 Launch event, June 2005 ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester

More Related