170 likes | 298 Views
Protocols Working with 10 Gigabit Ethernet. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. Introduction to Measurements 10 GigE on SuperMicro X7DBE 10 GigE on SuperMicro X5DPE-G2 10 GigE and TCP – Monitor with web100 disk writes
E N D
ProtocolsWorking with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” CALICE, Mar 2007, R. Hughes-Jones Manchester
Introduction to Measurements • 10 GigE on SuperMicro X7DBE • 10 GigE on SuperMicro X5DPE-G2 • 10 GigE and TCP – Monitor with web100 disk writes • 10 GigE and Constant Bit Rate program • UDP + memory access CALICE, Mar 2007, R. Hughes-Jones Manchester
Udpmon: Latency & Throughput Measurements • Tells us about: • Behavior of the IP stack • The way the HW operates • Interrupt coalescence • UDP/IP packets sent between back-to-back systems • Similar processing to TCP/IP but no flow control & congestion avoidance algorithms • Latency • Round trip times using Request-Response UDP frames • Latency as a function of frame size • Slope s given by: • Mem-mem copy(s) + pci + Gig Ethernet + pci + mem-mem copy(s) • Intercept indicates processing times + HW latencies • Histograms of ‘singleton’ measurements • UDP Throughput • Send a controlled stream of UDP frames spaced at regular intervals • Vary the frame size and the frame transmit spacing & measure: • The time of first and last frames received • The number packets received, lost, & out of order • Histogram inter-packet spacing received packets • Packet loss pattern • 1-way delay • CPU load • Number of interrupts • Tells us about: • Behavior of the IP stack • The way the HW operates • Capacity & Available throughput of the LAN / MAN / WAN CALICE, Mar 2007, R. Hughes-Jones Manchester
Sender Receiver Zero stats OK done Send data frames at regular intervals Inter-packet time (Histogram) ●●● ●●● Time to receive Time to send Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Signal end of test OK done Time Number of packets n bytes time Wait time Throughput Measurements • UDP Throughput with udpmon • Send a controlled stream of UDP frames spaced at regular intervals CALICE, Mar 2007, R. Hughes-Jones Manchester
High-end Server PCs • Boston/Supermicro X7DBE • Two Dual Core Intel Xeon Woodcrest 5130 • 2 GHz • Independent 1.33GHz FSBuses • 530 MHz FD Memory (serial) • Parallel access to 4 banks • Chipsets: Intel 5000P MCH – PCIe & MemoryESB2 – PCI-X GE etc. • PCI • 3 8 lane PCIe buses • 3* 133 MHz PCI-X • 2 Gigabit Ethernet • SATA CALICE, Mar 2007, R. Hughes-Jones Manchester
Histogram FWHM ~1-2 us 10 GigE Back2Back: UDP Latency • Motherboard: Supermicro X7DBE • Chipset: Intel 5000P MCH • CPU: 2 Dual Intel Xeon 5130 2 GHz with 4096k L2 cache • Mem bus: 2 independent 1.33 GHz • PCI-e 8 lane • Linux Kernel 2.6.20-web100_pktd-plus • Myricom NIC10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • rx-usecs=0 Coalescence OFF • MSI=1 • Checksums ON • tx_boundary=4096 • MTU 9000 bytes • Latency 22 µs & very well behaved • Latency Slope 0.0028 µs/byte • B2B Expect: 0.00268 µs/byte • Mem 0.0004 • PCI-e 0.00054 • 10GigE 0.0008 • PCI-e 0.00054 • Mem 0.0004 CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE Back2Back: UDP Throughput • Kernel 2.6.20-web100_pktd-plus • Myricom 10G-PCIE-8A-R Fibre • rx-usecs=25 Coalescence ON • MTU 9000 bytes • Max throughput 9.4 Gbit/s • Notice rate for 8972 byte packet • ~0.002% packet loss in 10M packetsin receiving host • Sending host, 3 CPUs idle • For <8 µs packets, 1 CPU is >90% in kernel modeinc ~10% soft int • Receiving host3 CPUs idle • For <8 µs packets, 1 CPU is 70-80% in kernel modeinc ~15% soft int CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE Cisco 7600: UDP Latency • Motherboard: Supermicro X7DBE • PCI-e 8 lane • Linux Kernel 2.6.20 SMP • Myricom NIC10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • Rx-usecs=0 Coalescence OFF • MSI=1 Checksums ON • MTU 9000 bytes • Latency 36.6 µs & very well behaved • Switch Latency 14.66 µs • Switch internal: 0.0011 µs/byte • PCI-e 0.00054 • 10GigE 0.0008 CALICE, Mar 2007, R. Hughes-Jones Manchester
The “SC05” Server PCs • Not ALL PCs work that well !! • Boston/Supermicro X7DBE • Two Intel Xeon Nocona • 3.2 GHz • Cache 2048k • Shared 800 MHz FSBus • DDR2-400 Memory • Chipsets: Intel 7520 Lindenhurst • PCI • 2 8 lane PCIe buses • 1 4 lane PCIe buse • 3* 133 MHz PCI-X • 2 Gigabit Ethernet CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE X7DBEX6DHE: UDP Throughput • Kernel 2.6.20-web100_pktd-plus • Myricom 10G-PCIE-8A-R Fibre • myri10ge v1.2.0 + firmware v1.4.10 • rx-usecs=25 Coalescence ON • MTU 9000 bytes • Max throughput 6.3 Gbit/s • Packet loss ~ 40-60 % in receiving host • Sending host, 3 CPUs idle • 1 CPU is >90% in kernel mode • Receiving host3 CPUs idle • For <8 µs packets, 1 CPU is 70-80% in kernel modeinc ~15% soft int CALICE, Mar 2007, R. Hughes-Jones Manchester
So now we can run at 9.4 Gbit/s Can we do any work ? CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE X7DBEX7DBE: TCP iperf Web100 plots of TCP parameters • No packet loss • MTU 9000 • TCP buffer 256k BDP=~330k • Cwnd • SlowStart then slow growth • Limited by sender ! • Duplicate ACKs • One event of 3 DupACKs • Packets Re-Transmitted • Throughput Mbit/s • Iperf throughput 7.77 Gbit/s • Not bad ! CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE X7DBEX7DBE: TCP iperf Web100 plots of TCP parameters • Packet loss 1: 50,000 -recv-kernel patch • MTU 9000 • TCP buffer 256k BDP=~330k • Cwnd • SlowStart then slow growth • Limited by sender ! • Duplicate ACKs • ~10 DupACKs every lost packet • Packets Re-Transmitted • One per lost packet • Throughput Mbit/s • Iperf throughput 7.84 Gbit/s • Even Better !!! CALICE, Mar 2007, R. Hughes-Jones Manchester
10 GigE X7DBEX7DBE: CBR/TCP Web100 plots of TCP parameters • Packet loss 1: 50,000 -recv-kernel patch • tcpdelay message 8120bytes • Wait 7 µs • RTT 36 µs • TCP buffer 256k BDP=~330k • Cwnd • Dips as expected • Duplicate ACKs • ~15 DupACKs every lost packet • Packets Re-Transmitted • One per lost packet • Throughput Mbit/s • tcpdelay throughput 7.33 Gbit/s CALICE, Mar 2007, R. Hughes-Jones Manchester
B2B UDP with memory access • Send UDP traffic B2B with 10GE • On receiver run independent memory write task • L2 Cache 4096 k Byte • Write 8000k Byte blocks in loop • 100% user mode • Achievable UDP Throughput • mean 9.39 Gb/s sigma 106 • mean 9.21 Gb/s sigma 37 • mean 9.2 sigma 30 • Packet loss • mean 0.04% • mean 1.4 % • mean 1.8 % • CPU load: Cpu0 : 6.0% us, 74.7% sy, 0.0% ni, 0.3% id, 0.0% wa, 1.3% hi, 17.7% si, 0.0% st Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st Cpu3 : 100.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st CALICE, Mar 2007, R. Hughes-Jones Manchester
Backup Slides CALICE, Mar 2007, R. Hughes-Jones Manchester
10 Gigabit Ethernet: Neterion NIC Results • X5DPE-G2 Supermicro PCs B2B • Dual 2.2 GHz Xeon CPU • FSB 533 MHz • XFrame II NIC • PCI-X mmrbc 4096 bytes • Low UDP rates ~2.5Gbit/s • Large packet loss • TCP • One iperf TCP data stream 4 Gbit/s • Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s CALICE, Mar 2007, R. Hughes-Jones Manchester