190 likes | 197 Views
Learn how to configure and use WUGS-20 GigE line cards for experimentation in a network environment. Instructions, utilities, and examples provided.
E N D
Using the WUGS-20GigE Line Card John DeHartApplied Research LaboratoryComputer Science and Engineering Departmenthttp://www.arl.wustl.edu/arl/projects/techX
Introduction • Really understanding how the WUGS-20 GigE line cards work requires understanding how ethernet, ARP, Routers and ethernet switches all work. • This set of instructions can’t possibly teach all of that. If you require all the details you are going to have to do a lot of work/study on your own. • These instructions will hopefully get you to the point where you can get the GigE line cards working for some useful experimentation. • Questions are always welcome: • Please keep in mind that we are continuing to support Kits users without support. The Kits program is officially over. • The number of Kits users who have GigE cards is very small (2). • If you have questions about the GigE cards, you should probably send them to John DeHart directly (jdd@arl.wustl.edu) and CC the other users: • Ron Srodawa (srodawa@oakland.edu) • Min Song (msong@odu.edu) • Our standard configuration has migrated to use • FPXs and SPCs on each port of an NSP (formerly called MSR) • Linux for all end hosts used in our router testbeds • NetBSD used ONLY for building kernels and plugins • We will still try to help you with other configurations but it is becoming increasingly difficult to replicate other configurations
Configuring a GigE Line Card • SUNI Config • There is a SUNI chip on the GigE card which needs some configuration. • We won’t go into the details, just don’t change the SUNI configuration commands in the configuration script. Don’t even change the order. • Ethernet Config • MAC Address • Mask • IP Address • Next Hop IP Address
Configuring a GigE Line Card • Utilities • readWriteRegs: low level utility for configuring the GigE card(s) • Src Directory: • wu_arl/wugs/gige/src/utilities/ReadWriteRegisters • Binary • Linux/readWriteRegs • configGigE.sh: script for configuring the GigE cards • uses readWriteRegs • Src Directory: • wu_arl/wugs/gige/src/utilities/configScripts • requires that WUARL environment variable be set to point to the root of your wu_arl tree. • for me, I set: > setenv WUARL /d/jdd/wu_arl • iperf: IP traffic generator • useful for experimentation • version 1.6.1 exists in CVS tree • wu_arl/utilities/IP/iperf • website: http://dast.nlanr.net/Projects/Iperf/ • tcpdump: Useful for debugging • run as root • run it on the gigabit ethernet interface of your host, something like: • tcpdump –i eth1 • you’ll see things like the ARP messages and ping requests and replies. • Just be careful of running it when you are sending LOTS of data
configGigE.sh utility Usage Message • Running configGigE.sh with no arguments will give the Usage message: > ./configGigE.sh ./configGigE.sh -vci <VCI> -mac <MAC_H16> <MAC_L32> [-{0,1,2} <MYIP#> <MASK#> <NHIP#>] -vci VCI : VCI on which to send GigE control packets from the CP -mac MAC_H16 MAC_L32: hex MAC address to assign to GigE card, high 16 bits first then low 32 bits -0 MYIP0 MASK0 NHIP0: hex values for MYIP0 MASK0 NHIP0 -1 MYIP1 MASK1 NHIP1: hex values for MYIP1 MASK1 NHIP1 -2 MYIP2 MASK2 NHIP2: hex values for MYIP2 MASK2 NHIP2 Example> ./configGigE.sh -vci 100 -mac 0x4400 0x5E040001 -0 0xC0A82801 0xFFFFFF00 0xC0A82805 -1 0 0 0 -2 0 0 0 this would assign: MAC address: 44 00 5e 04 00 01 MYIP0 : 192.168.50.1 MASK0 : 255.255.255.0 (24 bits) NHIP0 : 192.168.50.5 MYIP1 : 0 MASK1 : 0 NHIP1 : 0 MYIP2 : 0 MASK2 : 0 NHIP2 : 0
Control Connection for GigE Configuration CP VCI=N VCI=N VCI=30 P0:GLink P1:GigE APIC VCI=N+1 VCI=N+1 Commands to read/write registers in GigE card VCI=30 P2:GigE • Each GigE card needs a separate bidirectional control connection between it and the CP. • At the GigE end it must use VCI=30 • In the example above we have two GigE cards and we are using VCI N and VCI N+1 at the CP end. • There are two connections on P0 mapping • VCI N P1, VCI 30 • VCI N+1 P2 VCI 30 • And one connection on each of P1 and P2: • P1 VCI 30 P0 VCI N • P2 VCI 30 P0 VCI N+1 • Remember: You still need at least one ATM interface on your switch so you can connect a CP that can send ATM control cells!
A Simple Configuration Host1 VCI=128 GigE-1 GE Eth • MYIP0: 192.168.40.1 • MYMAC: 00:00:5E:04:00:01 • MASK0: * (Don’t care) • NHIP0: * (Don’t care) • IP: 192.168.40.2 • 24 bit netmask • MAC: <as defined by hw> • route: 192.168.50.0/24 gw: 192.168.40.1 Host2 VCI=128 GigE-2 • IP: 192.168.50.2 • 24 bit netmask • MAC: <as defined by hw> • route: 192.168.40.0/24 gw: 192.168.50.1 GE Eth • MYIP0: 192.168.50.1 • MYMAC: 00:00:5E:04:00:02 • MASK0: * (Don’t care) • NHIP0: * (Don’t care)
Simple Configuration: Ping • When Host1 tries to ‘ping’ Host2 for the first time: • Host1 has a route that tells it to use 192.168.40.1 as the next hop to get to Host 2 (192.168.50.2) • Host1 has no ARP entry for 192.168.40.1 • Host1 generates an ARP request asking for MAC address of 192.168.40.1 and sends it out its GE Ethernet interface • GigE-1 receives the ARP request which matches its MYIP0 address and responds with its MAC address • Host1 receives the ARP reply • Host1 then sends out the ping packet. • GigE-1 receives the ping packet and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128. • WUGS-20 is configured to send VPI/VCI 0/128 to port of GigE-2 • GigE-2 receives first ping packet destined for 192.168.50.2 • GigE-2 has no ARP entry for 192.168.50.2 • GigE-2 generates an ARP request asking for the MAC address of 192.168.50.2 and sends it out its ethernet interface. • Host2 receives the ARP request which matches its IP address and responds with its MAC address • GigE-2 receives the ARP reply and puts an entry in its ARP table.
Simple Configuration: Ping • When Host1 tries to ‘ping’ Host2 for the second time: • Host1 has a route that tells it to use 192.168.40.1 as the next hop to get to Host 2 (192.168.50.2) • Host1 has an ARP entry for 192.168.40.1 • Host1 sends out an ethernet frame for the ping packet with a next hop MAC address of GigE-1. • GigE-1 receives the ethernet frame and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128 • WUGS-20 is configured to send VPI/VCI 0/128 to the port of GigE-2 • GigE-2 receives first ping packet destined for 192.168.50.2 • GigE-2 has an ARP entry for 192.168.50.2 • GigE-2 sends the ping packet out its ethernet interface • ETC… (similar stuff happens for the ping reply except some ARP information was gleaned on the forward path…)
A Configuration with a Switch IP: 192.163.204.10 MAC: 08:00:20:7C:E3:25 Host1 IP: 192.163.204.2 MAC: 08:00:20:7C:E3:25 IP: 192.163.204.3 MAC: 08:00:20:7C:F2:45 Host2 Host3 P3 Ethernet Switch MSR P1 Port 1: MAC: 00:00:5E:04:00:01 MYIP0: 192.163.204.1 MASK0: * (Don’t care) NHIP0: * MYIP1: * MASK1: NHIP1: * MYIP2: * MASK2: * NHIP2: * Host4 IP: 192.163.204.4 MAC: 00:01:03:7C:23:03
A Configuration with a Switch: Ping • When Host1 tries to ‘ping’ Host2: • The MSR Port 3 is configured to route packets destined for Host2 to Port 1 on the base VPI/VCI of 0/128. • Port 1 has a GigE interface. • Port 1 initially does not have an ARP entry for Host2. • Port 1 sends an ARP request looking for Host2 and drops this first ping packet. • The Ethernet switch sends this packet to Host2 which sends an ARP reply. • The Ethernet switch sends the ARP reply back to the MSR Port 3. • The next ping from Host1 to Host2 will go through. • Similar for when Host1 pings Host3 or Host4. • This configuration still does not need to use the Next Hop information in the GigE card.
A Configuration with Routers Host 1 192.168.100.10 192.168.202.0/24 Hosts 192.168.210.0/24 Hosts 192.168.201.0/24 Hosts P3 Ethernet Switch2 P0 Router2 P1 Ethernet Switch1 MSR P0 192.168.200.0/24 Hosts Ethernet Switch3 P0 Router3 P1 192.168.220.0/24 Hosts
A Configuration with Routers • In this configuration we have Ethernet Switch1 supporting three subnets: • 192.168.200.0/24 • 192.168.201.0/24 • 192.168.202.0/24 • Each of the Routers (Router1, Router2, MSR) has a presence in each of the subnets (200, 201, 202) on their P0 interfaces • MSR P0 (limited to three): • 192.168.200.1 • 192.168.201.1 • 192.168.202.1 • Router2 P0: • 192.168.200.2 • 192.168.201.2 • 192.168.202.2 • Router3 P0: • 192.168.200.3 • 192.168.201.3 • 192.168.202.3 • MSR also supports hosts on the 192.168.100.0/24 subnet • such as Host1 192.168.100.10 • Router2 also supports the 192.168.210.0/24 subnet • Router3 also supports the 192.168.220.0/24 subnet
A Configuration with Routers (con’t) • MSR P0 GigE Card Configuration • MAC: 44 00 5E 04 00 01 • MYIP0: 192.168.200.1 • MASK0: 0xffffff00 • NHIP0: 192.168.200.2 • used for Egress VCI 129 • MYIP1: 192.168.201.1 • MASK1: 0xffffff00 • NHIP1:192.168.201.3 • used for Egress VCI 130 • MYIP2: 192.168.202.1 • MASK2: 0 • NHIP2:0 • used for Egress VCI 131 (unused in this example)
Example Commands to Configure P0 GigE • Assume that VCI 101 is set up in the MSR switch to go from the CP port to P0 VCI 30. • cd $WUARL/wugs/gige/src/configScripts • ./configGigE.sh -vci 101 -mac 0x4400 0x5E040001 -0 0xc0A8C801 0xffffff00 0xc0A8C802 -1 0xc0A8C901 0xffffff00 0xc0A8C903 -2 0xc0A8CA01 0 0
A Configuration with Routers: Ping • When Host1 tries to ‘ping’ any host in the (200, 201, 202) subnets • Just like our earlier example where there were no routers.
A Configuration with Routers: Ping • When Host1 tries to ‘ping’ a host in the 210 subnet • The MSR P3 is configured to route packets destined for the 210 subnet to P0 on the VPI/VCI of 0/129. This will cause P0 to use MYIP0, MASK0 and NHIP0. • NHIP0 is the IP address of Router2 • A function applied to (MYIP0, MASK0, NHIP0) is used to generate an index into the ARP table in the GE card to store the MAC address of Router2 • details are not important at this point. • P0 has a GigE interface. • P0 GigE initially does not have an ARP entry for Router2. • P0 GigE sends an ARP request looking for Router2 and drops this first ping packet. • Router2 sends an ARP reply with the MAC address from its P0. • The next ping from Host1 to the 210 subnet will go through.
A Configuration with Routers: Ping • When Host1 tries to ‘ping’ a host in the 220 subnet • The MSR P3 is configured to route packets destined for the 220 subnet to P0 on the VPI/VCI of 0/130. This will cause P0 to use MYIP1, MASK1 and NHIP1. • NHIP1 is the IP address of Router3 • A function applied to (MYIP1, MASK1, NHIP1) is used to generate an index into the ARP table in the GE card to store the MAC address of Router3 • details are not important at this point. • P0 has a GigE interface. • P0 GigE initially does not have an ARP entry for Router3. • P0 GigE sends an ARP request looking for Router3 and drops this first ping packet. • Router3 sends an ARP reply with the MAC address from its P0. • The next ping from Host1 to the 220 subnet will go through.
Performance • In our tests we have seen the following: • Using two 2.4 GHz Athlon machines • Using iperf • UDP: • packet size 512 bytes • sending rate: ~720 Mb/s • receiving rate: ~720 Mb/s with about 0.2% drops at receiver • receiving host couldn’t keep up • TCP: 700 Mb/s - 770 Mb/s • using the iperf defaults • 8K buffer size • MSS set by TCP based on MTU