Operating Wide-Area Ethernet Networks

1. Operating Wide-Area Ethernet Networks • Matt Davy • Global NOC

2. Outline • Overview of Networks • Configuration • Troubleshooting/Monitoring

3. Overview of Networks • NLR FrameNet • nationwide ethernet over dwdm • 18 Cisco 6509 switches • 10GbE backbone • p2p and multipoint vlans • dedicated and best effort

5. Overview of Networks • I-Light • Indiana’s statewide higher ed network • statewide ethernet over dwdm • 19 Cisco 6509 switches (layer2 & layer3) • 10GbE backbone with p2p vlans

7. Overview of Networks • MANLAN • ethernet exchange in new york city • Cisco 6513 switch • 1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn • local and wide-area connections

8. Overview of Networks • Indiana University campus network • large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses) • Cisco 6500’s and HP Procurve • very interesting stp design • ~1,500 total switches

9. Configuration Issues • configuration of vlans • very manual and time intensive (manual = error prone) • need to automate this process • various control plane projects are one option, but could use something more lightweight • could use vtp ?

10. Configuration Issues • VLAN ID Assignment • big problem when interconnecting multiple layer2 domains • does Q-in-Q solve this ? • does vlan id translation solve this ?

11. Configuration Issues • Q-in-Q • sounds good, but not flexible enough • want to map some .1q tags to outside vlan and want other .1q tags to get switched normally • customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in-Q. • also not implemented in all switches

12. Configuration Issues • VLAN ID Translation • could help, but limitations in currently implementation • each port needs it’s own translation table • on 6500, translation table is shared across multiple ports • greatly confuses cross-domain troubleshooting

13. Configuration Issues • loops and spanning tree fun • spanning-tree is often not well understood • some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology • often does not help anyway when multiple layer2 domains are interconnected

14. Configuration Issues • things that might help some: • enable spanning-tree within your domain • filter bpdus at the edge of your domain • limit total broadcast traffic on every port • make sure config has enough granularity for port speed (1% of 10G is still too much)

15. Loops Outside of Your Domain #1 #2

16. Configuration Issues • why will a loop outside your domain hose your switch ? • not 100% clear • one possibility is mac address learning overload • switch flooded with packets for which it has to learn source mac addresses • mac addresses quickly flip-flop between ports

17. Configuration Issues • how could this be avoided ? • turn off mac address learning • for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway • could also have out-of-band mechanism to statically configure mac forwarding tables • will this entirely protect you ? don’t know

18. Troubleshooting/Monitoring • how can you tell when a vlan is down ? • hint: think break in the middle of the topology • on vlan trunks, can’t see how much traffic is associated with each vlan • CoS hack on the 6500’s for this • lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking

19. Troubleshooting/Monitoring • tools to trace current vlan path across the network • IU has developed a spanning-tree mapping tool that helps with this • “turn-around interfaces” useful for debugging performance problems

20. Thank You