1 / 17

Protocols Working with 10 Gigabit Ethernet

Learn about UDP latency measurements, throughput testing, server configurations, and more with 10 GigE protocols. Explore the tools and techniques used for UDP measurements on high-performance network systems. Discover the insights into hardware operations, latency behavior, and available throughput in LAN/MAN/WAN networks.

ewest
Download Presentation

Protocols Working with 10 Gigabit Ethernet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ProtocolsWorking with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” CALICE, Mar 2007, R. Hughes-Jones Manchester

  2. Introduction to Measurements • 10 GigE on SuperMicro X7DBE • 10 GigE on SuperMicro X5DPE-G2 • 10 GigE and TCP – Monitor with web100 disk writes • 10 GigE and Constant Bit Rate program • UDP + memory access CALICE, Mar 2007, R. Hughes-Jones Manchester

  3. Udpmon: Latency & Throughput Measurements • Tells us about: • Behavior of the IP stack • The way the HW operates • Interrupt coalescence • UDP/IP packets sent between back-to-back systems • Similar processing to TCP/IP but no flow control & congestion avoidance algorithms • Latency • Round trip times using Request-Response UDP frames • Latency as a function of frame size • Slope s given by: • Mem-mem copy(s) + pci + Gig Ethernet + pci + mem-mem copy(s) • Intercept indicates processing times + HW latencies • Histograms of ‘singleton’ measurements • UDP Throughput • Send a controlled stream of UDP frames spaced at regular intervals • Vary the frame size and the frame transmit spacing & measure: • The time of first and last frames received • The number packets received, lost, & out of order • Histogram inter-packet spacing received packets • Packet loss pattern • 1-way delay • CPU load • Number of interrupts • Tells us about: • Behavior of the IP stack • The way the HW operates • Capacity & Available throughput of the LAN / MAN / WAN CALICE, Mar 2007, R. Hughes-Jones Manchester

  4. Sender Receiver Zero stats OK done Send data frames at regular intervals Inter-packet time (Histogram) ●●● ●●● Time to receive Time to send Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Signal end of test OK done Time Number of packets n bytes  time Wait time Throughput Measurements • UDP Throughput with udpmon • Send a controlled stream of UDP frames spaced at regular intervals CALICE, Mar 2007, R. Hughes-Jones Manchester

  5. High-end Server PCs • Boston/Supermicro X7DBE • Two Dual Core Intel Xeon Woodcrest 5130 • 2 GHz • Independent 1.33GHz FSBuses • 530 MHz FD Memory (serial) • Parallel access to 4 banks • Chipsets: Intel 5000P MCH – PCIe & MemoryESB2 – PCI-X GE etc. • PCI • 3 8 lane PCIe buses • 3* 133 MHz PCI-X • 2 Gigabit Ethernet • SATA CALICE, Mar 2007, R. Hughes-Jones Manchester

  6. Histogram FWHM ~1-2 us 10 GigE Back2Back: UDP Latency • Motherboard: Supermicro X7DBE • Chipset: Intel 5000P MCH • CPU: 2 Dual Intel Xeon 5130 2 GHz with 4096k L2 cache • Mem bus: 2 independent 1.33 GHz • PCI-e 8 lane • Linux Kernel 2.6.20-web100_pktd-plus • Myricom NIC10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • rx-usecs=0 Coalescence OFF • MSI=1 • Checksums ON • tx_boundary=4096 • MTU 9000 bytes • Latency 22 µs & very well behaved • Latency Slope 0.0028 µs/byte • B2B Expect: 0.00268 µs/byte • Mem 0.0004 • PCI-e 0.00054 • 10GigE 0.0008 • PCI-e 0.00054 • Mem 0.0004 CALICE, Mar 2007, R. Hughes-Jones Manchester

  7. 10 GigE Back2Back: UDP Throughput • Kernel 2.6.20-web100_pktd-plus • Myricom 10G-PCIE-8A-R Fibre • rx-usecs=25 Coalescence ON • MTU 9000 bytes • Max throughput 9.4 Gbit/s • Notice rate for 8972 byte packet • ~0.002% packet loss in 10M packetsin receiving host • Sending host, 3 CPUs idle • For <8 µs packets, 1 CPU is >90% in kernel modeinc ~10% soft int • Receiving host3 CPUs idle • For <8 µs packets, 1 CPU is 70-80% in kernel modeinc ~15% soft int CALICE, Mar 2007, R. Hughes-Jones Manchester

  8. 10 GigE Cisco 7600: UDP Latency • Motherboard: Supermicro X7DBE • PCI-e 8 lane • Linux Kernel 2.6.20 SMP • Myricom NIC10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • Rx-usecs=0 Coalescence OFF • MSI=1 Checksums ON • MTU 9000 bytes • Latency 36.6 µs & very well behaved • Switch Latency 14.66 µs • Switch internal: 0.0011 µs/byte • PCI-e 0.00054 • 10GigE 0.0008 CALICE, Mar 2007, R. Hughes-Jones Manchester

  9. The “SC05” Server PCs • Not ALL PCs work that well !! • Boston/Supermicro X7DBE • Two Intel Xeon Nocona • 3.2 GHz • Cache 2048k • Shared 800 MHz FSBus • DDR2-400 Memory • Chipsets: Intel 7520 Lindenhurst • PCI • 2 8 lane PCIe buses • 1 4 lane PCIe buse • 3* 133 MHz PCI-X • 2 Gigabit Ethernet CALICE, Mar 2007, R. Hughes-Jones Manchester

  10. 10 GigE X7DBEX6DHE: UDP Throughput • Kernel 2.6.20-web100_pktd-plus • Myricom 10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • rx-usecs=25 Coalescence ON • MTU 9000 bytes • Max throughput 6.3 Gbit/s • Packet loss ~ 40-60 % in receiving host • Sending host, 3 CPUs idle • 1 CPU is >90% in kernel mode • Receiving host3 CPUs idle • For <8 µs packets, 1 CPU is 70-80% in kernel modeinc ~15% soft int CALICE, Mar 2007, R. Hughes-Jones Manchester

  11. So now we can run at 9.4 Gbit/s Can we do any work ? CALICE, Mar 2007, R. Hughes-Jones Manchester

  12. 10 GigE X7DBEX7DBE: TCP iperf Web100 plots of TCP parameters • No packet loss • MTU 9000 • TCP buffer 256k BDP=~330k • Cwnd • SlowStart then slow growth • Limited by sender ! • Duplicate ACKs • One event of 3 DupACKs • Packets Re-Transmitted • Throughput Mbit/s • Iperf throughput 7.77 Gbit/s • Not bad ! CALICE, Mar 2007, R. Hughes-Jones Manchester

  13. 10 GigE X7DBEX7DBE: TCP iperf Web100 plots of TCP parameters • Packet loss 1: 50,000 -recv-kernel patch • MTU 9000 • TCP buffer 256k BDP=~330k • Cwnd • SlowStart then slow growth • Limited by sender ! • Duplicate ACKs • ~10 DupACKs every lost packet • Packets Re-Transmitted • One per lost packet • Throughput Mbit/s • Iperf throughput 7.84 Gbit/s • Even Better !!! CALICE, Mar 2007, R. Hughes-Jones Manchester

  14. 10 GigE X7DBEX7DBE: CBR/TCP Web100 plots of TCP parameters • Packet loss 1: 50,000 -recv-kernel patch • tcpdelay message 8120bytes • Wait 7 µs • RTT 36 µs • TCP buffer 256k BDP=~330k • Cwnd • Dips as expected • Duplicate ACKs • ~15 DupACKs every lost packet • Packets Re-Transmitted • One per lost packet • Throughput Mbit/s • tcpdelay throughput 7.33 Gbit/s CALICE, Mar 2007, R. Hughes-Jones Manchester

  15. B2B UDP with memory access • Send UDP traffic B2B with 10GE • On receiver run independent memory write task • L2 Cache 4096 k Byte • Write 8000k Byte blocks in loop • 100% user mode • Achievable UDP Throughput • mean 9.39 Gb/s sigma 106 • mean 9.21 Gb/s sigma 37 • mean 9.2 sigma 30 • Packet loss • mean 0.04% • mean 1.4 % • mean 1.8 % • CPU load: Cpu0 : 6.0% us, 74.7% sy, 0.0% ni, 0.3% id, 0.0% wa, 1.3% hi, 17.7% si, 0.0% st Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st Cpu3 : 100.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st CALICE, Mar 2007, R. Hughes-Jones Manchester

  16. Backup Slides CALICE, Mar 2007, R. Hughes-Jones Manchester

  17. 10 Gigabit Ethernet: Neterion NIC Results • X5DPE-G2 Supermicro PCs B2B • Dual 2.2 GHz Xeon CPU • FSB 533 MHz • XFrame II NIC • PCI-X mmrbc 4096 bytes • Low UDP rates ~2.5Gbit/s • Large packet loss • TCP • One iperf TCP data stream 4 Gbit/s • Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s CALICE, Mar 2007, R. Hughes-Jones Manchester

More Related