UNIDAD 6 Configuracion N5K y FEX

1. UNIDAD 6Configuracion N5K y FEX 2012 Alejandro Brown Systems Engineer

2. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

3. Cisco Nexus 2000 Series Fabric Extender Cisco Nexus Parent Switch Virtual Modular System + Nexus 2000 Fabric Extender =

4. Cisco FEXlink Architecture FEXlink: Cisco’s Fabric Extender Architecture Unified access for any server (100M1GE10GE FCoE): Scalable Ethernet, HPC, unified fabric or virtualization deployment Reduced TCO Less cabling, less management points, low power Simplified management and provisioning Business Agility with VM aware network services & quick addition of capacity Available on Nexus 5000, 7000 Virtual Modular System Parent Switch + Fabric Extender Single Management Domain

5. Fabric Extender Terminology • Fabric interface - A 10-Gigabit Ethernet uplink port designated for connection from the Fabric Extender to its parent switch. A fabric interface cannot be used for any other purpose. It must be directly connected to the parent switch. • Port channel fabric interface - A port channel uplink connection from the Fabric Extender to its parent switch. This connection consists of fabric interfaces bundled into a single logical channel. • Host interface - An Ethernet host interface for connection to a server or host system. • Do not connect a bridge or switch to a host interface. These interfaces are designed to provide end host or server connectivity. • Port channel host interface - A port channel host interface for connection to a server or host system

6. Host Interfaces • The Fabric Extender provides connectivity for computer hosts and other edge devices in the network fabric. • The following guidelines should be followed when connecting devices to Fabric Extender host interfaces: • All Fabric Extender host interfaces run as spanning tree edge ports with BPDU Guard enabled and cannot be configured as Spanning Tree network ports. • Servers utilizing active/standby teaming, 802.3ad port channels, or other host-based link redundancy mechanisms can be connected to Fabric Extender host interfaces. • Any device running spanning tree connected to a Fabric Extender host interface will result in that host interface being placed in an error-disabled state when a BPDU is received. • Any edge switch that leverages a link redundancy mechanism not dependent on Spanning Tree such as Cisco Flexlink or vPC (with BPDUFilter enabled) may be connected to a Fabric Extender host interface.Since spanning tree is not utilized to eliminate loops, care must be taken to ensure a loop-free topology below the Fabric Extender host interfaces. • You can enable host interfaces to accept Cisco Discovery Protocol (CDP) packets. This protocol only works when it is enabled for both ends of a link.

7. FEX Terminology

8. FEX Fabric Connectivity Models • Two models (mutually exclusive per FEX) • Static Pinning • Port Channel • Static Pinning: • fixed relationship between a host port and an uplink port • when the uplink goes down, corresponding pinned server ports are brought down and remain down till the uplink is brought up again. • Port Channel: • all the server ports pinned to the port channel, and if there is any link failure within the port channel, traffic automatically use the remaining up link ports

9. Static Pinning If the uplink goes down the corresponding server ports will go down SFP+ phy SFP+ phy SFP+ phy SFP+ phy FabricExtender Redwood 6 Redwood 0 Redwood 2 Redwood 3 Redwood 4 Redwood 5 Redwood 1 octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T Color indicates the corresponding uplink pinned interface

10. Static Pinning • Sticky, based on configuration, not operational state of uplinks • A configuration parameter ‘pinning max-links’ determines how many uplink port are to be used for pinning. The server ports will be divided by the number of ‘max-links’ and distributed accordingly: • 4 uplinks: every contiguous 12 ports pinned to one uplink • 3 uplinks: every contiguous 16 ports pinned to one uplink • 2 uplinks: every contiguous 24 ports pinned to one uplink • 1 uplinks: all 48 ports pinned to the uplink • Default max-links is 1, max is 4 • Note: changing max-links configuration is disruptive, it causes all FEX server ports to go down and come up. Note that the disruption applies to all server ports even if they ended up re-pinned to the same fabric link

11. Static Pinning (Cont.) • Pinning order is dependent on the order of fabric port configuration, irrespective of when they come up or whether they come up. Example 1: fex 101 with fabric ports configured in the order ethernet1/5, ethernet1/2, ethernet1/3, ethernet1/6 • Ethernet 101/1/1-12 are pinned to ethernet1/5 • ethernet 101/1/13-24 are pinned to ethernet1/2 • ethernet 101/1/25-36 are pinned to ethernet 1/3 • ethernet 101/1/37-48 are pinned to ethernet 1/6 • On N5K bootup, pinning order is determined by fabric port id, sorted in ascending order. Example 2, with the above information, the pinning order after reboot is • ethernet 101/1/1-12 pinned to ethernet 1/ 2 • ethernet 101/1/13-24 pinned to ethernet 1/3 • ethernet 101/1/25-36 pinned to ethernet 1/5 • ethernet 101/1/37-48 pinned to ethernet 1/6

12. Static Pinning (Cont.) • To guarantee deterministic and sticky pinning order across reboot, the ‘redistribute’ command should be used before reboot • Example 3: based on Example 1, after the redistribute command, pinning order is as follows, therefore it will be the same order after reboot, as in Example 2 • ethernet 101/1/1-12 pinned to ethernet1/2, • ethernet 101/1/13-24 pinned to ethernet 1/3, • ethernet 101/1/25-36 pinned to ethernet 1/5, • ethernet 101/1/37-48 pinned to ethernet 1/6

13. Port Channel Port Channel If an uplink goes down, server ports are not affected as long as the Port Channel is up SFP+ phy SFP+ phy SFP+ phy SFP+ phy FabricExtender Redwood 6 Redwood 0 Redwood 1 Redwood 2 Redwood 3 Redwood 4 Redwood 5 octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T octal 1G base-T All server ports are pinned to the Port Channel

14. Port Channel • All server ports pinned to fabric port channel • One fabric port channel per FEX. ‘pinning max-links’ must be 1 • Redwood uses the same hashing algorithm as in N5K • The hashing could be based on a combination of source/destination L2 and source/destination L3

15. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

16. Fabric Extender forwarding overview • Fabric Extender does not support local switching. • Forwarding is based on VNTag • VNTag is a Network Interface Virtualization (NIV) technology • VNTag allows the Fabric Extender to act as a data path of Nexus 5000 for all policy and forwarding • VNTag is added to the packet between Fabric Extender and Nexus 5000 • VNTag is stripped before the packet is sent to hosts

17. Redwood Forwarding unicast/multicast (Host to Network) • For Host to Network direction, a VNTag is added to the packet. The source VIF in VNTag represents the host port • The source VIF is unique within the scope of the pinned uplink • The packet is sent upstream based on the pinned interface

18. Redwood Forwarding unicast (Network to Host) • N5K adds VNTag to packet in Network-to-Host direction • The Dest VIF in the VNTag is looked up to identify the Host interface • Source VIF and Loop bit in VNTag allows for loop detection • VNTag is removed before sending out of Host port

19. Redwood Forwarding multicast (Network to Host) • Redwood supports multicast replication from Network to Host direction. • The multicast packet from network to host will carry a VNTag that corresponds to a multicast group • VNTag is used as in index into the multicast forwarding table to find the set of output interfaces to replicate the packet • Loop check is done as in unicast case • VNTag is removed before sending out of Host interface

20. FEX forwarding host to network (Unicast / multicast)

21. Forwarding VNTag packet within N5K (Unicast)

22. FEX forwarding network to host (Unicast)

23. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

24. FabricExtender Fabric Extender Management model • Fabric Extender is NOT an independent manageable entity • From a management perspective, FEX is modeled as a Nexus 5000 line-card • Nexus 5000 and FEX communicate via in-band connectivity • Out-of-band method for debugging and disaster recovery only to be used by Cisco field personnel through dongle connector • No configuration is stored on the FEX N5000 Single Management Domain Single Management Domain 10GE 10GE Fabric Extender 10GE or 1GE Server Racks

25. FEX Bootstrap • Fabric Extender is discovered by N5K using an L2 Satellite Discovery Protocol (SDP) • SDP runs on every uplink port of Fabric Extender • FEX and N5K advertises device information and negotiates control channel VNTag • FEX also includes uplink SFP information • Once discovered, Fabric Extender registers with N5K through Satellite Registration Protocol (SRP) • One SRP session per FEX • FEX sends image version, SRG, and mac address to N5K • N5K checks software image compatibility, assigns an internal IP address and upgrades the Fabric Extender software image if necessary • N5K also sends switch id, switch ip address, switch MTS address and FEX MTS address and FEX ID to FEX

26. Management Model • FEX can be connected to one Nexus 5000 or Nexus 7000 (straight-through topology) • FEX can be connected to 2 Nexus 5000 (active-active topology). This will become available with vPC on Nexus 5000 • With Active-active mode, FEX will be managed by both Nexus 5000s. • Note that in active-active mode there is no master. The configuration of each Nexus 5000 has to be performed separately and to be consistent otherwise the vPC will not come up. • When FEX connect via vPC to 2 Nexus 5000s the forwarding is synchronized through common Vntag • If the connectivity to one of Nexus 5000 is lost, the convergence is sub-second

27. Management Model (cont) • FEX may not go Online under certain circumstances such as faulty uplinks, or a system failure which prevents FEX to boot successfully. In those cases, current state of the FEX can be read from the system LED.

28. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

29. Software Images • Bootloader One copy of bootloader image stored in the write protected boot flash. Not field upgradeable. • System Image • Two copies of system image. • Primary System Image • Upgradeable by switch. • Stored in the Image flash. • Backup (a.k.a Golden) System Image • Stored in the write protected boot flash. • Serves as a backup when primary image is corrupted. • Not field upgradeable.

30. Boot Sequence • Bootloader verifies image checksum. • When primary image checksum is invalid, backup image is booted. • Primary image checksum is valid but fails to boot successfully: • Bootloader maintains a boot failure count which is reset by the system image after booting successfully. When the count exceeds a threshold (3), backup image is booted. • Hardware watchdog kicks in when system hangs. • When both images are invalid (unlikely event) user is notified via system LED (blinking Amber). • Boot variables are stored along with bootloader. User can override the default boot sequence via debug dongle.

31. FEX Software Installation • When a fabric extender is connected to a Cisco Nexus 5000 Series Switch, a handshake verifies the image version on the fabric extender, and the switch determines whether the fabric extender needs to be updated. • If an upgrade is necessary, the Cisco Nexus 5000 Series Switch loads the code onto the fabric extender and reboots the fabric extender with the new image. The update takes approximately 8 minutes.

32. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

33. Fabric Extender configuration steps • Static Pinning • Define the number of uplink for FEX (valid range: 1-4) • Associate FEX to an Ethernet interface on N5K • switch# configure terminal • switch(config)# fex 100 • switch(config-fex)# pinning max-links 4 • switch# configure terminal • switch(config)# interface ethernet 1/1 • switch(config-if)# switchport mode fex-fabric • switch(config-if)# fex associate 100

34. Fabric Extender configuration

35. Verify FEX connectivity • N5k# show fex • FEX FEX FEX FEX • Number Description State Model Serial • ------------------------------------------------------------------------ • 100 FEX0100 Online N2K-C2148T-1GE FOX1242GJTA

36. Verify FEX connectivity (cont) • N5k# sh fex 100 detail • FEX: 100 Description: FEX0100 state: Online • FEX version: 4.0(1a)N2(0.103) [Switch version: 4.0(1a)N2(0.103)] • Extender Model: N2K-C2148T-1GE, Extender Serial: JAF1244ATEQ • Part No: 73-12009-02 • Card Id: 70, Mac Addr: 00:0d:ec:b1:3e:82, Num Macs: 64 • Module Sw Gen: 17 [Switch Sw Gen: 17] • pinning-mode: static Max-links: 4 • Fabric port for control traffic: Eth1/3 • Fabric interface state: • Eth1/1 - Interface Up. State: Active • Eth1/2 - Interface Up. State: Active • Eth1/3 - Interface Up. State: Active • Eth1/4 - Interface Up. State: Active Fabric Ports

37. Verify FEX connectivity (cont) • Fex Port State Fabric Port Primary Fabric • Eth100/1/1 Up Eth1/1 Eth1/1 • Eth100/1/2 Down Eth1/1 Eth1/1 • Eth100/1/3 Down Eth1/1 Eth1/1 • Eth100/1/4 Down Eth1/1 Eth1/1 • Eth100/1/5 Down Eth1/1 Eth1/1 • Eth100/1/6 Down Eth1/1 Eth1/1 • Eth100/1/7 Down Eth1/1 Eth1/1 • Eth100/1/8 Down Eth1/1 Eth1/1 • Eth100/1/9 Down Eth1/1 Eth1/1 • Eth100/1/10 Down Eth1/1 Eth1/1 • Eth100/1/11 Down Eth1/1 Eth1/1 • Eth100/1/12 Down Eth1/1 Eth1/1 Pinned fabric Port

38. FEX fabric ports This command shows which port on N5K is connected to Fabric Extender, useful if the user does not know which ports are attached to FEX before the configuration. • N5k# show interface fex-fabric • Fabric Fabric Fex FEX • Fex Port Port State Uplink Model Serial • --------------------------------------------------------------- • 114 Eth1/1 Active 1 N5K-C5110T-BF-1GE JAF1238ANHB • 119 Eth1/8 Active 4 N5K-C5110T-BF-1GE JAF1241BLFQ • 120 Eth1/9 Discovered 1 N5K-C5110T-BF-1GE JAF1241BLGF • 120 Eth1/10 Configured 2 N5K-C5110T-BF-1GE JAF1241BLGF • 116 Eth1/25 Identity-Mismatch 1 N5K-C5110T-BF-1GE JAF1236ARCL Fex Id Fabric port Status Switch port FEX Uplink number Serial Number

39. Fabric Extender configuration steps • Port Channel • Define the number of uplink for FEX (valid range: 1) • Associate FEX to an Port-channel interface on N5K • Associate FEX to an Ethernet interface on N5K • switch# configure terminal • switch(config)# fex 100 • switch(config-fex)# pinning max-links 1 • switch# configure terminal • switch(config)# interface port-channel 100 • switch(config-if)# switchport mode fex-fabric • switch(config-if)# fex associate 100 • switch# configure terminal • switch(config)# int e1/17,e1/19 • switch(config-if)# switchport mode fex-fabric • switch(config-if)# fex associate 100 • switch(config-if)# channel-group 100 mode on

40. FEX Port-Channel Verification

41. vPC with FEX connectivity options vPC option 2 FEX Active-Active vPC option 1 FEX Straight Through N5K N5K N5K N5K FEX FEX FEX FEX

42. vPC peer link failure vPC option 1 Primary vPC Secondary vPC Primary vPC Secondary vPC vPC option 2 N5K N5K N5K N5K FEX FEX FEX FEX Secondary vPC peer brings down all its vPC members

43. Keep alive link failure vPC option 1 vPC option 2 Primary vPC Secondary vPC Primary vPC Secondary vPC N5K N5K N5K N5K FEX FEX FEX FEX No Change to the connectivity

44. Both peer-link and Keep alive-link failure Primary vPC Primary vPC vPC option 2 vPC option 1 Primary vPC Primary vPC N5K N5K N5K N5K FEX FEX FEX FEX If both MCT link and keep alive link fails at the same time or keep alive link fails first then MCT fails, both switch becomes primary vPC keeping all vPC member link up.

45. vPC OPTION 2 – FEX Active-Active

46. Agenda • Pinning and FEX Terminology • Forwarding • Management • Platform Software Overview • Installation and Configuration • FEX Supported topologies with N5K • FEX Supported topologies with N7K

47. Single attached FEX, single attached Host

48. Single attached FEX, multi attached Host

49. Single attached FEX, multi attached Host (continue)

50. Dual attached FEX, single attached Host