Chapter 5: Implementing High Availability and Redundancy in a Campus Network

1. Chapter 5: Implementing High Availability and Redundancy in a Campus Network CCNP SWITCH: Implementing IP Switching

2. Chapter 5 Objectives • Understand high availability. • Implement high availability. • Describe high availability monitoring options. • Describe switch supervisor redundancy. • Describe gateway redundancy protocols. • Configure and verity Cisco IOS server load balancing.

3. Understanding High Availability

4. Components of High Availability • Redundancy • Technology (including hardware and software features) • People • Processes • Tools

5. Redundancy • Geographic diversity and path diversity are often included. • Dual devices and links are common. • Dual WAN providers are common. • Dual data centers are sometimes used, especially for large companies and large e-commerce sites. • Dual collocation facilities, dual phone central office facilities, and dual power substations can be implemented.

6. Technology • Cisco Nonstop Forwarding (NSF) • Stateful Switchover (SSO) • Graceful Restart • Cisco IOS IP Service Level Agreements (SLA) • Object Tracking • Firewall Stateful Failover

7. People • Prepare, Plan, Design, Implement, Operate, and Optimize (PPDIOO) is a guide. • Work habits and attention to detail important. • Skills are acquired via ongoing technical training. • Good communication and documentation critical. • Use lab testing to simulate failover scenarios. • Take time to design. • Identify roles. • Identify responsibilities. • Align teams with services. • Ensure time to do job.

8. Processes • Organizations should build repeatable processes. • Organizations should use labs appropriately. • Organizations need meaningful change controls. • Management of operational changes is important.

9. Tools • Network diagrams. • Documentation of network design evolution. • Key addresses, VLANs, and servers documented. • Documentation tying services to applications and physical servers.

10. Resiliency for High Availability • Network-Level Resiliency • High Availability and Failover Times

11. Network-Level Resiliency • Built with device and link redundancy. • Employs fast convergence. • Relies on monitoring with NTP, SNMP, Syslog, and IP SLA.

12. High Availability and Failover Times • Tuned routing protocols failover in less than 1 second. • RSTP converges in about 1 second. • EtherChannel can failover in approximately 1 second. • HSRP timers are 3 seconds for hello and 10 seconds for hold time. • Stateful service modules typically failover within 3-5 seconds. • TCP/IP stacks have up to a 9-second tolerance.

13. Optimal Redundancy • Provide alternate paths. • Avoid too much redundancy. • Avoid single point of failure. • Use Cisco NSF with SSO, if applicable. • Use Cisco NSF with routing protocols.

14. Provide Alternate Paths • Use redundant distribution-to-core links in case a core switch fails. • Link distribution switches to support summarization of routing information from the distribution to the core.

15. Avoid Too Much Redundancy • Where should the root switch be placed? With this design, it is not easy to determine where the root switch is located. • What links should be in a blocking state? It is hard to determine how many ports will be in a blocking state. • What are the implications of STP and RSTP convergence? The network convergence is definitely not deterministic. • When something goes wrong, how do you find the source of the problem? The design is much harder to troubleshoot.

16. Avoid Single Point of Failure • Key element of high availability. • Easy to implement at core and distribution. • Access layer switch is single point of failure. Reduce outages to 1 to 3 seconds in the access layer with: • SSO in L2 environment • Cisco NSF with SSO in L3 environment.

17. Cisco NSF with SSO • Supervisor redundancy mechanism in Cisco IOS enabling supervisor switchover at L2-L3-L4. • SSO enables standby RP to take control after fault on active RP. • Cisco NSF is L3 function that works with SSO to minimize time network unavailable following switchover, continuing to forward IP packets following RP switchover.

18. Routing Protocols and NSF • NSF enables continued forwarding of packets along known routes while routing protocol information is being restored during switchover. • Switchover must complete before NSF dead and hold timers expire or routing peers will reset adjacencies and reroute traffic.

19. Implementing High Availability

20. Distributed VLANs on Access Switches • Use Rapid STP (RSTP) as the version of STP. • Provide a Layer 2 trunk between the two distribution switches to avoid unexpected traffic paths and multiple convergence events. • Place the Hot Standby Router Protocol (HSRP) primary and the STP primary root on the same distribution layer switch if you choose to load balance VLANs across uplinks. • The HSRP and RSTP root should be colocated on the same distribution switches to avoid using the interdistribution link for transit.

21. Local VLANs on Access Switches • No VLANs span between access layer switches across distribution switches. • Here a single voice VLAN and a single data VLAN are restricted to a single access switch. • Root for each VLAN aligned with active HSRP instance. • Distribution-to-distribution L3 link required for route summarization in this design.

22. Layer 3 Access to the Distribution Interconnection • L3 or routed links connect distribution and access layer switches in this design – in the future this will be the standard (even the links to the end stations will be L3 in the future as prices of RPs continue to decline). • Recommended practice is to map the L2 VLAN number to the L3 subnet for ease of use and management.

23. Daisy Chaining Access Layer Switches (1) • No links block from an STP perspective. • Both uplinks are available to send and receive traffic. • If a link or node in the middle of the chain or stack fails, standby HSRP peer (Dist-B) can go active as it loses connectivity to its primary peer (Dist-A).

24. Daisy Chaining Access Layer Switches (2) • Here the core switch sees both distribution switches advertise the VLAN 2 subnet, doing equal cost load balancing for traffic destined to VLAN 2 between Dist-A and Dist-B. • 50% chance that return traffic arrives on distribution switch that does not have connectivity to half of stack where traffic destined. Solution: add A-a to A-c connection.

25. StackWise Access Switches • Supports recommended practice of using L3 connection between distribution switches without having to use loopback cable or perform extra configuration. • Uses Cisco Catalyst 3750 switches in the access layer. • Much less complex than chains or stacks of other models. • Appears as one node from network topology perspective.

26. Too Little Redundancy (1) • VLANs span multiple access layer switches. • No L2 link between distribution switches. • Design is looped in figure-8 topology. • Once access layer uplink is blocking • HSRP hellos exchanged by transiting access switches.

27. Too Little Redundancy (2) • When uplink from Access A to Distribution A fails, there are 3 convergence events.

28. Implementing Network Monitoring

29. Network Management Overview • Ability to verify the network is working well and behaving in the planned manner • Ability to characterize the performance of the network • Ability to understand how much traffic is flowing and where it is flowing in the network • Ability to troubleshoot the network

30. Network Management Tools • Syslog • SNMP • IP SLA

31. Syslog • System Message Logging • Enables device to report error and notification messages. • Uses UDP port 514. • Every message contains a severity level and a facility. • Routers, switches, application servers, firewalls, and other network appliances support syslog.

32. Syslog Severity Levels Smaller numerical levels are the more critical syslog alarms.

33. Syslog Facilities • Service identifiers. • Identify and categorize system state data for error and event message reporting. • Cisco IOS has more than 500 facilities. • Most common syslog facilities: • IP • OSPF • SYS operating system • IP Security (IPsec) • Route Switch Processor (RSP) • Interface (IF)

34. Syslog Message Format • System messages begin with a percent sign (%) • Facility: A code consisting of two or more uppercase letters that indicates the hardware device, protocol, or a module of the system software. • Severity: A single-digit code from 0 to 7 that reflects the severity of the condition. The lower the number, the more serious the situation. • Mnemonic: A code that uniquely identifies the error message. • Message-text: A text string describing the condition. This portion of the message sometimes contains detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space.

35. Sample Syslog Messages 08:01:13: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/5, changed state to up 08:01:23: %DUAL-5-NBRCHANGE: EIGRP-IPv4:(1) 1: Neighbor 10.1.1.1 (Vlan1) is up: new adjacency 08:02:31: %LINK-3-UPDOWN: Interface FastEthernet0/8, changed state to up 08:18:20: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/5, changed state to down 08:18:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/5, changed state to up 08:18:24: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down 08:18:24: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa0/2: PD removed 08:18:26: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to down 08:19:49: %ILPOWER-7-DETECT: Interface Fa0/2: Power Device detected: Cisco PD 08:19:53: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up 08:19:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up

36. Configuring Syslog (1) • To configure a syslog server, use the logging ip_addrglobal configuration command. • To which severity levels of messages are sent to the syslog server, use the global configuration command logging trap level. Switch(config)# logging trap ? <0-7> Logging severity level alerts Immediate action needed (severity=1) critical Critical conditions (severity=2) debugging Debugging messages (severity=7) emergencies System is unusable (severity=0) errors Error conditions (severity=3) informational Informational messages (severity=6) notifications Normal but significant conditions (severity=5) warnings Warning conditions (severity=4)

37. Configuring Syslog (2) • To configure logging to the buffer of the local switch, use the command logging buffered. Switch(config)# logging buffered ? <0-7> Logging severity level <4096-2147483647> Logging buffer size alerts Immediate action needed (severity=1) critical Critical conditions (severity=2) debugging Debugging messages (severity=7) discriminator Establish MD-Buffer association emergencies System is unusable (severity=0) errors Error conditions (severity=3) informational Informational messages (severity=6) notifications Normal but significant conditions (severity=5) warnings Warning conditions (severity=4) xml Enable logging in XML to XML logging buffer

38. Verifying Syslog Configuration • Use the show logging command to display the content of the local log files. • Use the pipe argument (|) in combination with keywords such as includeorbeginto filter the output. Switch# show logging | include LINK-3 2d20h: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up 2d20h: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up 2d20h: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up Switch# show logging | begin %DUAL 2d22h: %DUAL-5-NBRCHANGE: EIGRP-IPv4:(10) 10: Neighbor 10.1.253.13 (FastEthernet0/11) is down: interface down 2d22h: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to down 2d22h: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/11, changed state to down

39. SNMP • SNMP has three elements: • Network Management Application (SNMP Manager) • SNMP Agents (running inside a managed device) • MIB Database object that describes the information in a predetermined format that the agent can use to populate the data. • SNMP defines how management information is exchanged between network management applications and management agents.

40. SNMP Version 1 • SNMP version 1 (SNMPv1), defined in RFC 1157. Five basic SNMP messages the network manager uses to transfer data from agents that reside on managed devices: • Get Request: Used to request the value of a specific MIB variable from the agent. • Get Next Request: Used after the initial Get Request to retrieve the next object instance from a table or a list. • Set Request: Used to set a MIB variable on an agent. • Get Response: Used by an agent to respond to a Get Request or Get Next Request from a manager. • Trap: Used by an agent to transmit an unsolicited alarm to the manager. An agent sends a Trap message when a certain condition occurs, such as a change in the state of a device, a device or component failure, or an agent initialization or restart.

41. SNMP Version 2 • SNMPv2 introduced with RFC 1441, but members of IETF subcommittee could not agree on the security and administrative sections of SNMPv2 specification. There were several attempts to achieve acceptance of SNMPv2 through release of experimental modified versions. • Community-based SNMPv2 (SNMPv2C), defined in RFC 1901, is most common implementation. SNMPv2C deploys administrative framework defined in SNMPv1, which uses read/write community strings for administrative access. • SNMPv2 introduces two new message types: • Get Bulk Request: Reduces repetitive requests and replies and improves performance when retrieving large amounts of data (e.g., tables). • Inform Request: Alert an SNMP manager of specific conditions. Unlike unconfirmed SNMP Trap messages, NMS acknowledges Inform Request by sending an Inform Response message back to requesting device.

42. SNMP Version 3 • SNMPv3 is described in RFCs 3410 through 3415. It adds methods to ensure the secure transmission of critical data between managed devices. • SNMPv3 introduces three levels of security: • noAuthNoPriv: No authentication is required, and no privacy (encryption) is provided. • authNoPriv: Authentication is based on Hash-based Message Authentication Code with Message Digest 5 (HMAC-MD5) or Hash-based Message Authentication Code with Secure Hash Algorithm (HMAC-SHA). No encryption is provided. • authPriv: In addition to authentication, Cipher Block Chaining-Data Encryption Standard (CBC-DES) encryption is used as the privacy protocol. • Security levels implemented for each security model determine which SNMP objects a user can access for reading, writing, or creating and list of notifications that its users can receive.

43. SNMP Recommendations • SNMPv1 and SNMPv2 use community strings in clear text and so should be carefully chosen to ensure they are not trivial. • Community strings should be changed at regular intervals and in accordance with network security policies. For example, the strings should be changed when a network administrator changes roles or leaves the company. • If SNMP is used only to monitor devices, use read-only communities. • Ensure that SNMP messages do not spread beyond the management consoles. Use access-lists to prevent SNMP messages from going beyond the required devices and on the monitored devices to limit access for management systems only. • SNMPv3 is recommended because it provides authentication and encryption.

44. Configuring SNMP • Step 1. Configure SNMP access lists. • Step 2. Configure SNMP community strings. • Step 3. Configure SNMP trap receiver. • Step 4. Configure SNMPv3 user. Switch(config)# access-list 100 permit ip 10.1.1.0 0.0.0.255 any Switch(config)# snmp-server community cisco RO 100 Switch(config)# snmp-server community xyz123 RW 100 Switch(config)# snmp-server trap 10.1.1.50

45. IP Service Level Agreement • Contract between service provider and customers. • Specifies connectivity and performance agreements. • Includes guaranteed level of network availability, network performance in terms of round-trip time, and network response in terms of latency, jitter, and packet loss.

46. IP SLA Measurements • In Cisco IOS, IP SLA measurement enables configuration of router to send synthetic traffic to host or router configured to respond. • One-way travel times and packet loss are gathered.

47. IP SLA Operations • Network engineer configures a target device, protocol, and UDP or TCP port number on the IP SLA source for each operation. Source uses IP SLA control protocol to communicate with responder before sending test packets. • To increase security on IP SLA measurements control messages, responder can utilize MD5 authentication for securing the control protocol exchange. • When operation finished and response received, results are stored in IP SLA MIB on source and retrieved using SNMP. • IP SLA operations are defined by target devices. If operation is something such as DNS or HTTP, target device might be any suitable computer. For operations such as testing the port used by a database, organization might not want to risk unexpected effects and would use IP SLA responder functionality to have a router respond in place of the actual database server. Responder functionality can be enabled in a router with one command and requires no complex or per-operation configuration.

48. IP SLA Source and Responder • IP SLA source is where all IP SLA measurement probe operations are configured either by CLI or through an SNMP tool that supports IP SLA operation. Source is also the Cisco IOS device that sends probe packets. Destination of probe might be another Cisco router or another network target, such as a web server or IP host. • Although destination of probe can be any IP device, measurement accuracy is improved with IP SLA responder. IP SLA responder is device running Cisco IOS and is configured as IP SLA measurement responder with the ip sla monitor responder configuration command.

49. IP SLA Operation with Responder • Network manager configures IP SLA operation by defining a target device, protocol, and port number on IP SLA source. Network manager can also configure reaction conditions. Operation is scheduled to be run for a period of time to gather statistics.

50. IP SLA Responder Timestamps • IP SLA responder timestamps are used in round-trip calculations. • IP SLA source sends test packet at time T1. • IP SLA responder includes receipt time (T2) and transmitted time (T3).