1 / 42

Quick Start Guide Virtual Port Channel (vPC)

Quick Start Guide Virtual Port Channel (vPC) . Architecture & Solutions Group US Public Sector Advanced Services Mark Stinnette, CCIE Data Center #39151. Date 1 August 2013 Version 1.7.2.

katina
Download Presentation

Quick Start Guide Virtual Port Channel (vPC)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quick Start GuideVirtual Port Channel (vPC) Architecture & Solutions Group US Public Sector Advanced Services Mark Stinnette, CCIE Data Center #39151 Date 1 August 2013 Version 1.7.2

  2. This Quick Start Guide (QSG) is a Cookbookstyle guide to Deploying Data Center technologies with end-to-end configurations for several commonly deployed architectures. This presentation will provide end-to-end configurations mapped directly to commonly deployed data center architecture topologies. In this cookbook style; quick start guide; configurations are broken down in an animatedstep by step process to a complete end-to-end good clean configuration based on Cisco best practices and strong recommendations. Each QSG will contain set the stage content, technology component definitions, recommended best practices, and more importantly different scenario data center topologies mapped directly to complete end-to-end configurations. This QSG is geared for network engineers, network operators, and data center architects to allow them to quickly and effectively deploy these technologies in their data center infrastructure based on proven commonly deployed designs.

  3. Commonly Deployed Designs – Single-Sided vPC vPC Configuration

  4. Commonly Deployed Designs – Double-Sided vPC (Dual-Layer vPC) vPC Configuration Double-Sided vPC

  5. Commonly Deployed Designs – Hierarchical vPC vPC Configuration

  6. Commonly Deployed Designs – Enhanced vPC vPC Configuration

  7. vPC Terminology & Components vPC Configuration vPC Peer-Keepalive Link vPC Peer-Link Dedicated Layer 3 Infrastructure vPC Peer Device vPC Member Port vPC vPC Domain Orphan Port vPC vPC VLAN :: VLAN(s) carried over the vPC peer-link and used to communicate via a vPC; As soon as a VLAN is defined on vPC peer-link it becomes a vPC VLAN non-vPC VLAN :: VLAN(s) that is not part of any vPC and not present on the vPC Peer-Link

  8. vPC Benefits Overview vPC Configuration vPC is a virtualization technology that presents paired or two Nexus devices as a unique Layer 2 logical node to the access layer devices or endpoints. vPC belongs to MultichassisEtherChannel [MCEC] family of technology. A virtual port channel (vPC) allows links that are physically connected to two different Cisco Nexus 7000 or 5000 Series devices to appear as a single port channel to a third device. The third device can be a switch, server, firewall, load balancer or any other networking device that supports link aggregation technology. vPC provides the following technical benefits: • Eliminates Spanning Tree Protocol (STP) blocked ports • Uses all available uplink bandwidth; Layer 2 hashing algorithm • Allows dual-homed servers to operate in active-active mode • Provides fast convergence upon link or device failure • Offers dual active/active FHRP (default gateways) for servers • Each peer device in the vPC domain runs its own control plane, and both devices work independently Using vPC; you gain immediate operational and architectural advantages: • Simplifies network design • Build highly resilient and robust Layer 2 network • Enables seamless virtual machine mobility and server high-availability clusters • Scales available Layer 2 bandwidth, increasing bisectional bandwidth • Grows the size of the Layer 2 network • vPC feature is included in the base NX-OS software license vPC also leverages native split horizon/loop management provided by port-channeling technology; meaning a packet entering a port-channel cannot immediately exit that same port-channel.

  9. vPC Spanning-Tree Recommendations vPC Configuration

  10. vPC Feature Configuration vPC Configuration

  11. Building a vPC Peer-Keepalive Link – Nexus 7000 Best Practices vPC Configuration Option 1 Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKALVRF) Use separate line cards & 1Gig ports are enough, else you burn 10Gig interfaces Option 2 Use Mgmt0 interfaces off Supervisors to dedicated routable OOB network + use management VRF Peer-Keepalive traffic along with management traffic Option 3 As a last resort, route the peer-keepalive traffic over the Layer 3 infrastructure + use default VRF

  12. Building a vPC Peer-Keepalive Link – Nexus 5000/6000 Best Practices vPC Configuration Option 1 Use Mgmt0 interfaces to dedicated routable OOB network + use management VRF Peer-Keepalive traffic along with management traffic Option 2 (Nexus 5000 with L3 module) Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKALVRF) Use separate interfaces & will burn 10Gig interfaces Option 2 (Nexus 5000 without L3 module) Dedicated link(s) in a separate Layer 2 Port-Channel , have the peer-keepalive peer across to the SVI’s, manually prune those VLANs off the peer-link making those VLANs (non-vPC VLANs), only trunk the peer-keepalive VLAN across this Layer 2 Port-Channel Due to ISSU checks via the show spanning issue-impact, ISSU will fail, workaround is to disable STP on this dedicated Layer 2 Port-Channel via the spanning-tree port type edge trunk command, assuming you have the global command spanning-tree port type edge bpdufilter default enabled Will burn 10Gig interfaces

  13. (Let’s get the Peer-Keekalive & Peer-Link Infrastructure Done First) vPC Configuration feature lacp vlan 1 – 200 vrf context PKAL interface port-channel 1 vrf member PKAL ip address [….]/30 interface e1/1 , e2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport switchport mode trunk interface e3/1 , e4/1 channel-group 2 force mode active feature lacp vlan 1 – 200 vrf context PKAL interface port-channel 1 vrf member PKAL ip address [….]/30 interface e1/1 , e2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport switchport mode trunk interface e3/1 , e4/1 channel-group 2 force mode active 7K-1 7K-2 Po1 1/1 1/1 2/1 2/1 3/1 3/1 4/1 4/1 Po2 Animation 5K-1 5K-2 Create a dedicated VRF for the vPC peer-keepalive link (best practice) Step 1 :: turn on LACP feature Step 2 :: define your vlans Step 3 :: build peer-keepalive Step 4 :: build L2 port channel for peer-link Building a vPC peer-link, follow these guidelines :: (1) Must have Peer-keepalive link up first; ensure the peer-link member ports are 10 Gig interfaces (2) Use a minimum of two 10 Gig ports (M1 up to 8 member ports & F1/F2 up to 16 member ports) (3) Use at least two different line cards to increase high availability of peer-link

  14. (Let’s get the STP & VPC Domain Configuration Built) vPC Configuration (Optional Config) – when using vPC peer-switch in a ‘hybrid’ environment use the spanning-tree pseudo-information to load balance VLANs across the 2 peer devices feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default no spanning-tree loopguard default spanning-tree vlan 1-200 priority 0 spanning-tree pseudo-information vlan 1-200 root priority 4096 vlan 1-100 designated priority 16384 vlan 101-200 designated priority 8192 vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination [….] source [….] vrf PKAL peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilterdefault no spanning-tree loopguarddefault spanning-tree vlan 1-200 priority 0 spanning-tree pseudo-information vlan 1-200 root priority 4096 vlan 1-100 designated priority 8192 vlan 101-200 designated priority 16384 vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination [….] source [….] vrf PKAL peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize 7K-1 7K-2 Animation 5K-1 5K-2 Step 1 :: turn on vpc feature Step 2 :: configure spanning-tree defaults Step 3 :: configure spanning-tree vlan root priorities Step 4 :: configure vpc domain (per best practices) Enable peer-switch; when activated both vPC peer devices must have the same STP priority set for all vPC VLANs – making them appear as a unique STP root bridge Hard set the Nexus 7K on the left vPC role primary and Nexus 7K on the right vPC role secondary (deterministic) Make the Nexus 7Kscontrol LACP establishment for all port-channels; (lowest) vpc domain id + system priority Enable peer-gateway, auto-recovery, delay restore, and iparp synchronize (per best practice) … see Strong Recommendations & Key Notes sections Setup the peer-keepalive; use the correct VRF accordingly

  15. (Let’s complete the Peer-Link Configuration) vPC Configuration feature lacp feature vpc vlan 1 – 200 vrf context PKAL interface port-channel 1 vrf member PKAL ip address [….]/30 interface e1/1 , e2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type network vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode active feature lacp feature vpc vlan 1 – 200 vrf context PKAL interface port-channel 1 vrf member PKAL ip address [….]/30 interface e1/1 , e2/1 channel-group 1 mode active ------------------------------------------------------ interface port-channel 2 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type network vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode active 7K-1 7K-2 peer-link Animation 5K-1 5K-2 Step 1 :: enable vPC peer-link on the L2 port channel Bridge Assurance is enabled by default when configuring vPC peer-link (spanning-tree port type network); Do NOT disable it on the vPC peer-link Always perform VLAN pruning on vPC peer-link with the allowed list of vPC VLANs; vPC VLANsmust also be pruned on the vPC member port s as well

  16. (Let’s Build the Access Layer Peer-Keekalive & Peer-Link Infrastructure) vPC Configuration 7K-1 7K-2 peer-keepalive link Use Mgmt0 interfaces to dedicated routable OOB network + use management VRF (configured during initial device setup); includes Peer-Keepalivetraffic along with management traffic Animation feature lacp vlan 1 – 200 vrf context management ip route 0.0.0.0/0 [….] interface mgmt0 ip address [….]/24 interface port-channel 1 switchport switchport mode trunk interface e1/1 - 2 channel-group 2 force mode active feature lacp vlan 1 – 200 vrf context management ip route 0.0.0.0/0 [….] interface mgmt0 ip address [….]/24 interface port-channel 1 switchport switchport mode trunk interface e1/1 - 2 channel-group 2 force mode active 1/1 1/1 1/2 1/2 mgmt0 mgmt0 5K-1 5K-2 OOB Step 1 :: turn on LACP feature Step 2 :: define your vlans Step 3 :: build L2 port channel for peer-link

  17. (Let’s Build the Access Layer vPC Configuration) Always use a different domain ID in a double-sided vPC topology; once configured, both peer devices use the vPC domain ID to automatically assign a unique vPC system MAC address; which is used as part of the LACP protocol vPC Configuration vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize vpcdomain 1 role priority 2 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize 7K-1 7K-2 Animation feature lacp feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilterdefault no spanning-tree loopguarddefault vpc domain 10 role priority 1 system-priority 8096 peer-keepalive destination [….] source [….] vrf management auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize feature lacp feature vpc vlan 1 – 200 spanning-tree pathcost method long spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default no spanning-tree loopguard default vpcdomain 10 role priority 2 system-priority 8096 peer-keepalive destination [….] source [….] vrf management auto-recovery auto-recovery reload-delay delay restore 30 iparp synchronize 5K-1 5K-2 OOB Manually set vPC system priority to ensure vPC peer devices are the primary devices on LACP at the aggregation layer or not the primary devices on LACP at the access layer Step 1 :: turn on vpc feature Step 2 :: configure spanning-tree defaults Step 3 :: configure vpc domain (per best practices)

  18. (Let’s complete the Peer-Link Configuration) vPC Configuration 7K-1 7K-2 Animation feature lacp feature vpc vlan 1 – 200 interface port-channel 1 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type network vpc peer-link interface e1/1 - 2 channel-group 2 force mode active feature lacp feature vpc vlan 1 – 200 interface port-channel 1 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type network vpc peer-link interface e1/1 - 2 channel-group 2 force mode active peer-link 5K-1 5K-2 OOB Step 1 :: enable vPC peer-link on the L2 port channel Bridge Assurance is enabled by default when configuring vPC peer-link (spanning-tree port type network); Do NOT disable it on the vPC peer-link Always perform VLAN pruning on vPC peer-link with the allowed list of vPC VLANs; vPC VLANsmust also be pruned on the vPC member port s as well

  19. (Let’s Build the Double-Sided / Dual-Layer vPC) vPC Configuration Configure vPC member port as spanning-tree port type normal Keep Spanning Tree protocol root function on the aggregation layer of the network; For each vPC peer device, configure root guard on ports connected to access devices interface port-channel 10 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal spanning-tree guard root vpc 10 interface e1/13 , e2/13 channel-group 10 force mode active port-channel load-balance src-dstip-l4port-vlan interface port-channel 10 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal spanning-tree guard root vpc 10 interface e1/13 , e2/13 channel-group 10 force mode active port-channel load-balance src-dstip-l4port-vlan 7K-1 7K-2 Animation 2/13 1/13 1/13 2/13 vPC 10 interface port-channel 10 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal vpc 10 interface e1/9 , e1/10 channel-group 10 force mode active port-channel load-balance src-dstip-l4port-vlan interface port-channel 10 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal vpc 10 interface e1/9 , e1/10 channel-group 10 force mode active port-channel load-balance src-dstip-l4port-vlan 1/10 1/9 1/9 1/10 5K-1 5K-2 Step 1 :: enable vPC on the member ports Step 2 :: enable spanning-tree port configurations Step 3 :: change port channel load balancing method The configuration of the vPC member port must match on both vPC peer devices. If there is a inconsistency, a VLAN or the entire port channel may suspend (depending on type-1 or type-2 consistency check for the vPC member port). Use the same vPC ID as the port channel ID for ease of configuration, monitoring, and troubleshooting Use source-destination, IP, L4 port and VLAN as fields for the port channel load balancing hashing algorithm; this improves fair usage of all member ports forming in the port channel

  20. (Let’s Build a vPC to a Standalone Switch & LACP Enabled Server) vPC Configuration interface port-channel 20 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal spanning-tree port guard root vpc 20 interface e3/13 channel-group 20 force mode active interface port-channel 30 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type edge trunk vpc 30 interface e3/14 channel-group 30 force mode active interface port-channel 20 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type normal spanning-tree port guard root vpc 20 interface e3/13 channel-group 20 force mode active interface port-channel 30 switchport switchport mode trunk switchport trunk allowed vlan 1-200 spanning-tree port type edge trunk vpc 30 interface e3/14 channel-group 30 force mode active 7K-1 7K-2 Animation 3/14 3/13 3/13 3/14 vPC 20 vPC 30 1/26 1/25 interface port-channel 20 switchport switchport mode trunk switchport trunk allowed vlan 1-200 interface e1/25 , e1/26 channel-group 20 force mode active Step 1 :: enable vPC on the member ports + enable spanning-tree port configurations accordingly

  21. (Let’s Add Enhanced vPC between Nexus 5000 and Nexus 2000 FEX) vPC Configuration Notice in the 5k/2kEvPC topology you DON’T need the vPC command under the port channel towards the server feature lacp feature fex fex 100 pinning max-links 1 fex 199 pinning max-links 1 interface port-channel 100 switchport mode fex-fabric vpc 100 fex associate 100 interface port-channel 199 switchport mode fex-fabric vpc 199 fex associate 199 interface e1/28 channel-group 100 interface e1/29 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk interface e100/1/1 , e199/1/1 channel-group 1000 force mode active feature lacp feature fex fex 100 pinning max-links 1 fex199 pinning max-links 1 interface port-channel 100 switchport mode fex-fabric vpc 100 fexassociate 100 interface port-channel 199 switchport mode fex-fabric vpc199 fexassociate 199 interface e1/28 channel-group 100 interface e1/29 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk interface e100/1/1 , e199/1/1 channel-group 1000 force mode active 7K-1 7K-2 Animation vPC 10 5K-1 5K-2 1/29 1/28 1/28 1/29 vPC 199 vPC 100 FEX 100 FEX 199 100/1/1 199/1/1 Po 1000

  22. (Let’s Add a vPC between a Host and a Nexus 7000 / 2000 Topology) vPC Configuration Straight-Through Topology (only supported topology between 7k& 2k FEX) install feature-set fex feature lacp feature-set fex fex 199 pinning max-links 1 interface port-channel 199 switchport mode fex-fabric fex associate 199 interface e5/28, e6/28 switchport mode fex-fabric fex associate 199 channel-group 199 interface port-channel 1000 switchportmode trunk switchport trunk allowed vlan10, 20 spanning-tree port type edge trunk vpc 1000 interface e199/1/1 channel-group 1000 force mode active install feature-set fex feature lacp feature-set fex fex199 pinning max-links 1 interface port-channel 199 switchportmode fex-fabric fexassociate 199 interface e5/28, e6/28 switchportmode fex-fabric fex associate 199 channel-group 199 interface port-channel 1000 switchport mode trunk switchport trunk allowed vlan 10, 20 spanning-tree port type edge trunk vpc 1000 interface e199/1/1 channel-group 1000 force mode active Default VDC Only Default VDC Only 7K-1 7K-2 Animation 5/28 6/28 5/28 6/28 Po 199 Po 199 FEX 199 FEX 199 199/1/1 199/1/1 vPC 1000 Notice in the 7k/2k Straight-through topology you need the vPC command under the port channel towards the server FET is an optical transceiver that provides a highly cost-effective solution for connecting FEX to its parent switch (7k, 5k, 6k). Note that FET can only be used to connect Fabric links between the Fabric Extender and the parent switch; FET-10Gmust be connected to another FET-10G)

  23. (Let’s Build a vPC to an ASA Firewall Appliance) vPC Configuration feature interface-vlan feature hsrp interface port-channel 80 switchport mode trunk switchport trunk allowed vlan 100, 200 spanning-tree port type edge trunk vpc 80 interface e6/13 channel-group 80 force mode active interface vlan200 ip address 20.20.20.6/24 no ip redirect hsrp 200 preempt ip20.20.20.254 iproute 10.10.10.0/24 20.20.20.1 feature interface-vlan feature hsrp interface port-channel 80 switchportmode trunk switchport trunk allowed vlan 100, 200 spanning-tree port type edge trunk vpc 80 interface e6/13 channel-group 80 force mode active interface vlan 200 ip address 20.20.20.5/24 no ip redirect hsrp 200 preempt priority 110 ip 20.20.20.254 ip route 10.10.10.0/24 20.20.20.1 7K-1 7K-2 Animation 6/13 6/13 vPC 80 0/1 0/0 ASA-5585-X interface port-channel 80.100 vlan 100 nameif inside secruity-level 99 ip address 10.10.10.1 255.255.255.0 standby 10.10.10.2 interface port-channel 80.200 vlan200 nameifoutside secruity-level 1 ip address 20.20.20.1 255.255.255.0 standby 20.20.20.2 route outside 0.0.0.0 0.0.0.0 20.20.20.254 interface GigabitEthernet0/0, Ge0/1 channel-group 80 mode active no nameif no secruity-level no ip address interface port-channel 80 port-channel load-balance vlan-src-dst-ip no nameif no secruity-level no ip address Subnet 10.10.10.0 /24 is serviced by the ASA in this example See VMDC Architecture for more virtual firewall configuration use cases and best practices

  24. vPC Layer 2 & Layer 3 Separation Design (Best Practice & Recommendations) vPC Configuration Animation Separate Layer 3 (routed traffic) and Layer 2 (bridged traffic) infrastructure. Use dedicated Layer 3 point-to-point link between the vPC peer devices for backup path to core CAN’T Dynamically route over a vPC – road mapped in version 7.x • Use a dedicated Layer 2 port-channel trunk for non-vPC VLAN and create dedicated VLAN/SVI to established a Layer 3 relationship (note those VLANS are not on the peer-link – manually pruned off) Firewalls attached in a vPC; use static routing ASAstatic route to HSRP on Nexus Nexus static route to ASA VIP Firewalls attached in a VRF sandwich; separate vPC attachment

  25. (Let’s Configure Layer 3 Routing and FHRP – Separate L2/L3 per Best Practices) vPC Configuration featuelacp feature ospf feature interface-vlan feature hsrp vlan1 – 200 interface loopback0 ip address [….]/32 router ospf 1 router-id [….] log-adjacency-changes detail auto-cost reference-bandwidth 100Gbps interface port-channel 5 ip address [….]/30 ip router ospf 1 area 0.0.0.0 ipospf network point-to-point interface e1/32, e2/32 channel-group 5 force mode active interface e3/32 ip address [….]/30 ip router ospf 1 area 0.0.0.0 ipospf network point-to-point interface vlan100 ipaddress [10.10.10.2]/24 no ipredirects iprouter ospf 1 area 0.0.0.10 ipospfpassive-interface hsrp100 preempt priority 110 ip[10.10.10.1] featuelacp feature ospf feature interface-vlan feature hsrp vlan 1 – 200 interface loopback0 ip address [….]/32 router ospf 1 router-id [….] log-adjacency-changes detail auto-cost reference-bandwidth 100Gbps interface port-channel 5 ip address [….]/30 ip router ospf 1 area 0.0.0.0 ipospf network point-to-point interface e1/32, e2/32 channel-group 5 force mode active interface e3/32 ip address [….]/30 ip router ospf 1 area 0.0.0.0 ipospf network point-to-point interface vlan 100 ip address [10.10.10.3]/24 no ip redirects ip router ospf 1 area 0.0.0.10 ipospf passive-interface hsrp 100 preempt ip[10.10.10.1] 7K-1 7K-2 1/32 1/32 3/32 3/32 2/32 2/32 Animation vPC 10 5K-1 5K-2 Use dedicated Layer 3 point-to-point link between the vPC peer devices for backup path to core Define the SVI associated with HSRPas passive routing interface in order to avoid forming routing adjacency over vPC peer-link Define vPC primary peer device as the active HSRP instance and vPC secondary peer device as the standby HSRP(from control plane standpoint) for ease of operations Disable ip redirect (no ip redirect) on the interface VLAN where HSRPis configured

  26. vPC Failure Scenarios vPC Configuration Failure 1 :: Peer-Keepalive fails Failure 2 :: Peer-Link fails on Aggregation Failure 3 :: Peer-Link fails on Access Layer Failure 4 :: Peer-Keepalive fails + Peer-Link fails (Split Brain) nothing happens – no traffic loss Role Secondary Role Primary Role Operational Primary Animation vPC member ports are shut down and all the vPC VLAN interfaces (SVIs) are shut down; meaning no more L3 advertisements – all this happens on the secondary vPC peer device When PKL link fails and PL fails ( in this order ) , you have a dual active situation , while both links are down , the primary vPC peer device remains primary and your secondary vPC device becomes operational primary In a vPC environment only operational primary switch behaves as STP root and processes BPDU and your secondary switch do not process BPDUs ( this is regardless of whichever switch is configured as STP root ) Existing flows continue to be forwarded as before the failure; but new flows learning are impaired and uncertain forwarding (or broken state) for new flows will be observed. So when the links comes back up, the originally primary switch will see that, there is an existing operational primary switch (originally secondary) which is behaving like an STP root and processing BPDUs If the originally primary switch tries to reclaim the primary role at this point, that would mean more convergence time while operational root role is being switched, hence we do not try to reclaim the vpc primary (and acting STP root ) role back to avoid more convergence times Role Secondary Role Primary Black hole traffic to single attached devices connected to vPC Peer device with secondary role

  27. vPC Failure Scenarios (5k & 2k FEX) vPC Configuration Failure 1 :: Peer-Link fails on FEX Parent Switch at Access Layer Failure 2 :: Single FEX Fails or Power Loss Role Secondary Role Primary No traffic loss – Only the vPC members are shut down northbound facing the Aggregation Layer and the NIF interfaces are lost on the FEX facing the secondary vPC peer device – all traffic will be forward from both FEXs to the primary vPC peer device Single Attached hosts connected to the FEX are unaffected Animation Role Secondary Role Primary 5K Parent switches have lost communication to the failed FEX; resulting all host traffic will forward out the secondary FEX. Minimal to no traffic loss when hosts are dual attached in LACP; Active / Standby NIC teaming will failover over to the secondary FEX Black hole traffic to devices connected to single FEX Active / Standby NICteaming will failover over to the secondary FEX

  28. vPC Bridge Assurance vPC Configuration Without Bridge Assurance Bridge Assurance prevents a spanning-tree domain from failing in an “open” state. When a port configured for Bridge Assurance stops receiving BPDU’s, the port transitions into a “blocking” state as opposed to remaining in a “forwarding” state. This “closed” state reduces the likelihood for mis-configured devices from creating STP loops. ‘spanning-tree bridge assurance’ is enabled by defaultfor all ‘network’ port types Specifies bi-directional transmission of BPDUs on all ports of type “network”. Protects against unidirectional links and peer switch software issues Provides IGP like hello-dead timer behavior for Spanning Tree In all versions of NX-OS, available in IOS on the Catalyst 6500 beginning 12.2(33) SXI Recommended in STP topologies Not recommend in vPC topologies; only on the peer-link (default) With Bridge Assurance

  29. Checking vPC Configuration Consistency (Type-1 & Type-2 consistency checks) vPC Configuration There are two types of consistency checks : Type-1 ::Puts peer device or interface into a suspended state to prevent invalid packet forwarding behavior. With vPC Graceful Consistency check, suspension occurs only on the secondary peer device. Type-2 ::Peer device or Interface still forward traffic; however they are subject to undesired packet forwarding behavior. Type 1 and Type 2 consistency check apply both for global configuration and for vPC interface configuration.

  30. Checking vPC Configuration Consistency (Type-1 & Type-2 consistency checks) vPC Configuration

  31. Checking vPC Configuration Consistency (Type-1 & Type-2 consistency checks) vPC Configuration If any of the vPC Type-2 parameters listed in the table below are not configured identically on both vPC peer devices, the inconsistent configuration can cause undesirable behavior in the traffic flow

  32. Strong Recommendations and Key Notes vPC Configuration • Always use different domain ID in double-sided vPC topology • Operations perspective, define vPC primary on the left Nexus and vPC secondary on the right Nexus (role priority) • When configuring large number of VLANs in a vPC environment, use the range command (vlan x-z) vs. individually configuring one at a time • Create a dedicated VRF for the vPC peer-keepalive link (ie. vrf context PKAL) • When building a vPC peer-link, follow these guidelines • Must have Peer-keepalive link up first; ensure the peer-link member ports are 10 Gig interfaces • Use a minimum of two 10 Gig ports (M1 up to 8 member ports & F1/F2 up to 16 member ports) • Use at least two different line cards to increase high availability of peer-link • Use dedicated mode 10 Gig ports with M1 32 line card vs. shared mode ports • Split vPC and non-vPC VLANs on different interswitch port channels • Don’t insert any device between vPC peers; a peer-link is a point-to-point link • Any vPC VLAN allowed on the vPC member port MUST be allowed on the vPC peer-link • Always perform VLAN pruning on vPC peer-link with allowed list of vPC VLAN; vPC VLAN must have been pruned on the vPC member port previously • If the M1 32 is used for both the vPC peer-link and L3 uplinks to L3 Core, use vPC object tracking feature • When building a vPC member port, follow these guidelines • The configuration of the vPC member port must match on both vPC peer devices • If there is a inconsistency, a VLAN or the entire port channel may suspend (depending on type-1 or type-2 consistency check for the vPC member port) • Use the same vPC ID as port channel ID for ease of configuration, monitoring, and troubleshooting • With the M1 Series line card :: there can be up to 8 active ports bundled – resulting a 16-way port channel to be built for the whole vPC • With the F1/F2Series line card :: there can be up to 16 active ports bundled – resulting a 32-way port channel to be built for the whole vPC • Do not mix different port types (M1, F1, F2) in the same vPC member port; this is not allowed by the software • Both sides of the vPC member ports must be of the same port type

  33. Strong Recommendations and Key Notes vPC Configuration • The vPC peer-keepalive link carries periodic heartbeat (UDP 3200) between vPC peer devices. It is used at the boot up of the vPC systems to guarantee both peer devices are up before forming vPC domain and also when vPC peer-link fails to down state; in the latter case, vPC peer-keepalive link is leveraged to detect split brain scenario (both vPC peer devices are active–active) [when vPC peer-link is down, there is no more real time synchronization between the 2 peer devices so vPC systems must react to this active-active situation; this is done by shutting down vPC member ports on secondary peer device]. • The vPC peer-keepalive link is a pre-requisite for the vPC domain to form initially (ie. prior to the vPC peer-link configuration + if peer-link is initial up before peer-keepaliveis up) • vPC has 3 timers; hold-timeout (default 3 sec), timeout (default 5s), hello interval (default 1s). The hold-timeout starts once the vPC peer link goes to a down state; during this time period the secondary vPC peer will ignore any peer-keepalive hello messages. During the timeout period, the secondary vPC peer device will look for vPC peer-keepalive hello messages from the primary vPC peer device. If a single hello is received, the secondary vPC peer concludes that there must be a dual active scenario and therefore will disable all its vPC member ports (that is, all port-channels that carry the keyword vpc). Command line configuration to modify vPC timers is (under vPC domain configuration context): peer-keepalive destination ipaddress [source ipaddress | hold-timeout secs | interval msecs {timeout secs}] The default values are ok in most situations. vPC peer link is down ! Keepalive Hold Timeout Keepalive Timeout

  34. Strong Recommendations and Key Notes vPC Configuration • Always enable vPC peer-gateway in the vPC domain (on both peer devices), even if there is no end device using this feature (devices that don’t perform standard ARP request for their default IP gateway), there is no side effect enabling it • (Corner Case) always use vPC peer-gateway exclude-vlan when a transit VLAN (over vPC peer-link) is used in the vPC domain, this is applicable only for mixed chassis mode (M1/F1) with peer-link on F1ints; note only static routing supported • Always enable vPC ARP sync on both vPC peers; performs a bulk ARP sync, improves convergence time for L3 flows • Always enable vPC delay restore on both vPC peer devices and tune the timer according based on the network profile • Always enable vPC graceful type-1 check on both vPC peer devices; graceful consistency-check; (enabled by default) • Always enable vPC auto-recovery on both vPC peer devices • Always enable vPC auto-recovery reload-delay on both vPC peer devices (note the vPC auto-recovery reload-delay deprecates the previous feature called vPC reload restore) • Use vPC orphan port suspend when single-attached devices connected to a vPC domain need to be disconnected from the network when vPC peer-link fails • Always use a different domain ID in a double-sided vPC topology; once configured, both peer devices use the vPC domain ID to automatically assign a unique vPC system MAC address; which is used as part of the LACP protocol • vPC role is non-preemptive so vPC operational role is the most relevant of the information per table below • With NX-OS 6.1 and prior releases, always use identical line cards on either side of the vPC Peer Link and vPC member ports (legs to downstream device) • Starting in NX-OS 6.2, always use identical line cards on either side of the vPC Peer Link and vPC member ports (legs to downstream device) when M1/M2 & F2E • Starting in NX-OS 6.2, VDC type must match between the 2 vPC peer devices when F2 & F2E are used in same VDC; meaning its ok to have F2 on vPC peer device 1 and F2E on vPC peer device 2 for the vPC Peer Link or vPC member ports. Note: in a F2 & F2E type of design; only features related to F2 apply (lowest common denominator)

  35. Strong Recommendations and Key Notes vPC Configuration • Use LACP protocol when connecting access devices to vPC domain (channel-group [x] mode active • Use LACP when available for graceful failover and misconfiguration protection • LACP mode active on both sides of the port channel • If access device does not support LACP, use manual bundling (channel-group [x] mode on) • If the downstream access switch is a Cisco Nexus device, enable LACP graceful-convergence (its on by default) • If the downstream access switch is NOT is a Cisco Nexus device, disableLACP graceful-convergence • Use source-destination, IP, L4 port and VLAN as fields for the port channel load balancing hashing algorithm; this improves fair usage of all member ports forming in the port channel • When possible, always dual-attach access devices to a vPC domain using a port channel • When connecting a single-attached access device to a vPC domain using a vPC VLAN, always connect it to the vPC primary peer device; reason is when if the vPC peer-link fails down any single attached device connected to the secondary peer device (and using vPC VLAN) will become completely isolated with the rest of the network • Single Attached Recommendations (descending order of priority): • Connect access device to an intermediate switch which is dual-attached to a vPC domain • Connect single-attached device to a vPC domain using non-vPC VLAN (must also create an inter-switch link between the 2 peer devices to transport non-vPC VLAN • Connect single-attached device to a vPC domain using vPC VLAN and leveraging vPC peer-link • In a double-sided vPC topology, all interconnect links between the 2 vPC domains MUST belong to the same vPC ID; all links form a unique vPC (on both sides of the 2 vPC domains) • LACP port suspend :: By default, LACP sets a port to the suspended state if it does not receive an LACPPDU from the peer (ie a server or host). In some cases, although this feature helps in preventing loops created due to misconfigurations, it can cause servers to fail to boot up because they require LACP to logically bring up the port. You can put a port into an individual state by using the lacp suspend-individual command. • On the Nexus 5000 this feature is disabled (no lacp suspend-individual) for servers connecting via LACP; on the Nexus 7000 this feature is enabled by default (lacp suspend-individual)

  36. Strong Recommendations and Key Notes vPC Configuration • Recommended Spanning Tree Protocol Configuration with vPC • Spanning Tree protocol must remain enabled for all VLANs (even if all access devices are vPC attached to the vPC domain); Do NOT disable spanning-tree protocol • Use MST with vPC if you need to build a large L2 domain; Plan ahead to avoid future configuration changes that can trigger vPC type-1 consistency failure • Implement consistent STP mode in the same L2 domain, ensure that all switch in your L2 domain are running with Rapid-PVST+ (default) or MST to avoid slow Spanning Tree convergence (30 seconds or more) • Perform VLAN pruning on vPC member ports to reduce internal resource consumption • Keep Spanning Tree protocol root function on the aggregation layer of the network (aggregation vPC domain) • For each vPC peer device, configure root guard on ports connected to access devices • Bridge Assurance is enabled by default when configuring vPC peer-link (spanning-tree port type network); Do NOT disable it on the vPC peer-link • It is not necessary to enable Bridge Assurance on the vPC (members ports in the vPC) – configure vPC member port as spanning-tree port type normal • Configure port fast (edge or edge trunk port type) on the host facing interfaces to avoid slow Spanning Tree protocol convergence (30 seconds or more) when port transitions to an up state • Configure BPDU guard on host facing interfaces to block any BPDU sent from the host (access switch port receiving the BPDU will be put in errdisable mode) – enable BPDU guard globally • Always define the vPC domain as the STP root for all VLANs in that domain (configure aggregation vPC peer devices as the STP root primary and STP root secondary) – enforce this rule with root guard on vPC peer device ports connected to another L2 switch • IF the vPC peer-switch is activated, both vPC peer devices MUST have the SAME spanning tree configuration (same priority for all vPC VLANs) – recommendation to activate vPC peer-switch in the environment • Do not enable Loop guard on vPC (disabled by default) • When using vPC peer-switch in a hybrid environment use the spanning-tree pseudo-information to load balance VLANs across the 2 peer devices • Enable UDLD in normal mode on vPC peer-link and vPC member ports

  37. Strong Recommendations and Key Notes vPC Configuration • Layer 3 and vPC Guidelines and Recommendations • Use separate Layer 3 link(s) to connect to L3 devices (like a router or firewall in routed mode) to a vPC domain; use individual Layer 3 links for routed traffic and a separate Layer 2 port-channel for bridged traffic if both routed and bridged traffic are required • Always build a Layer 3 backup routed path for the vPC domain in order to increase network resilience and availability; use an OSPF point-to-point adjacency (or equivalent L3 protocol) between the 2 vPC peer devices to establish a L3 backup path to the core in case of uplink failure • Do NOT use a Layer 2 vPC to attach Layer 3 devices to a vPC domain unless the Layer 3 device can statically route to the HSRP address configured on the vPC peer devices • You can’t dynamically route over a vPC • Layer 3 backup routing path options (descending order of preference) • Use dedicated Layer 3 point-to-point link between the vPC peer devices for backup path to core • Use a dedicated Layer 2 port-channel trunk for non-vPC VLAN and create dedicated VLAN/SVI to established a Layer 3 relationship (note those VLANS are not on the peer-link) • HSRP / VRRP Guidelines and Recommendations • When running HSRP/VRRP in active-active mode (data plane standpoint), aggressive timers can be relaxed; use the default HSRP/VRRP timers • Define the SVI associated with HSRP/VRRP as passive routing interface in order to avoid forming routing adjacency over vPC peer-link • Define vPC primary peer device as the active HSRP/VRRP instance and vPC secondary peer device as the standby HSRP/VRRP (from control plane standpoint) for ease of operations • Disable ip redirect (no ip redirect) on the interface VLAN where HSRP/VRRP is configured. • Do NOT use HSRP/VRRP object tracking in a vPC domain

  38. Strong Recommendations and Key Notes vPC Configuration • Recommendations for Multilayer vPC for DCI Solution • Use different vPC domain-id for each vPC domain (DC1: vPC domain for aggregation, vPC for DCI. DC2: vPC domain for aggregation, vPC for DCI) • For each data center, interconnect the aggregation vPC domain to the DCI vPC domain using a vPC (double-sided topology) • Interconnect the 2 data centers using a vPC (vPC between DCI vPC domain in site 1 and site 2) • Enable BPDU filter on the vPC used for DCI (under the port-channel configuration, activate the following command: spanning-tree bpdufilter enable) to avoid BPDU propagation • Configure the vPC used for DCI as spanning-tree port type edge (i.e port fast) to fasten port state forwarding mode when port is operationally up • Remember by default vPC peer-link runs in spanning-tree port type network i.e bridge assurance is activated on the link • Configure root guard on aggregation vPC domain (more exactly on vPC between this vPC domain and DCI vPC domain). STP root must remain on aggregation vPC domain on each side of the data center • No loop must exist outside the vPC domains. • Do not use Layer 3 peering between data centers (in other words, there is no Layer 3 over vPC). • Do not use bridge assurance for interconnect vPC (DCI vPC) – use spanning-tree port type edge trunk • Use M1 ports for DCI vPC if flows between the 2 data centers need to be encrypted using 802.1aeMACsec

  39. Strong Recommendations and Key Notes vPC Configuration • Best Practices for Network Services / Appliances and vPC • Configure vPC to the inside and outside interfaces for ASA firewalls – use spanning-tree port type edge trunk • If needed, use multiple VRF instances for the inside interfaces – intra data center nets (see VMDC architecture) • Be aware of the following Layer 3 over vPC design caveat • Use dedicated Layer 2 port-channel for the service appliances state and keepaliveVLANs (recommend don’t use the vPC peer-link) • Recommended the ASA port channel hashing algorithm and the Nexus vPC hashing algorithm are the same • Connect ASA in routed mode to a vPC – must use static routing • ASA static route to HSRP on Nexus • Nexus static route to ASA VIP • If Connected ASA in routed mode and use dynamic routing • Single attach ASA to vPC domain • Create separate non-vPC interswitch link • Peer with non-vPC VLAN/SVIs SLB attached via Po with orphan port suspend SLB attached via vPC Firewall attached via vPC & Static Routes Firewall attached via non-vPC Po & Dynamic Routing Bandwidth maintained during certain failure scenarios Bandwidth reduced during certain failure scenarios

  40. Mixed Chassis Mode Supported Topologies vPC Configuration Interop F2 & F2E VDC • With NX-OS 6.1 and Prior Releases :: • Alwaysuse identical line cards on either side of the vPC Peer Link and vPC member ports (legs to downstream device) • The F1-series line cards can mix with M-series line cards • The F2-series line cards have to be in their own VDC; VDC type [F2] meaning they can’t mix with F1 or the M-series in the same VDC

  41. Mixed Chassis Mode Supported Topologies vPC Configuration • Starting in NX-OS 6.2 and Later Releases :: • VDC type [F2, F2E, F2 F2E] must match between the 2 vPC peer devices when F2 & F2E are used in same VDC; meaning its ok to have F2 on vPC peer device 1 and F2E on vPC peer device 2 for the vPC Peer Link or vPC member ports • Note: in a F2 & F2E type of design; only features related to F2 apply (lowest common denominator) • Alwaysuse identical line cards on either side of the vPC Peer Link and vPC member ports when M1, M1-XL, M2 & F2E in same VDC [M-F2E] or system • When F2E is placed in a chassis with M-series it will operate in Layer 2 mode only leveraging the M for Layer 3 (proxy L3 forwarding)

  42. vPC Resources & Additional Reading vPC Configuration External (public) Nexus vPC best practices design guide http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf Nexus 7000/6000/5000 Configuration Guides http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html Nexus 5000 Enhanced vPC Configuration Guide http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_enhanced_vpc.html Great External Resource

More Related