Network Troubleshooting

Network Troubleshooting Accessing the WAN– Chapter 8

Objectives • Establish a network baseline • Describe troubleshooting methodologies and troubleshooting tools • Describe the common issues that occur during WAN implementation • Troubleshoot enterprise network implementation issues

Establish a Network Baseline • Explain the importance of network documentation

Establish a Network Baseline • When you document your network, you may have to gathr information directly from routers and switches. • Commands that are useful to the network documentation process include: • The ping command is used to test connectivity with neighboring devices. Pinging to other PCs in the network also initiates the MAC address auto-discovery process. • The telnet command is used to log in remotely to a device for accessing configuration information. • The show ip interface brief is used to display the up or down status and IP address of all interfaces. • The show ip route command is used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols. • The show cdp neighbor detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

Establish a Network Baseline • Explain the purpose for measuring normal network performance when creating a baseline

Establish a Network Baseline • Describe the steps for establishing a network baseline

Describe Troubleshooting Methodologies and Troubleshooting Tools • When you document your network, you may have to gather information directly from routers and switches. • Commands that are useful to the network documentation process include: • The ping command is used to test connectivity with neighboring devices. Pinging to other PCs in the network also initiates the MAC address auto-discovery process. • The telnet command is used to log in remotely to a device for accessing configuration information. • The show ip interface brief is used to display the up or down status and IP address of all interfaces. • The show ip route command is used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols. • The show cdp neighbor detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

Describe Troubleshooting Methodologies and Troubleshooting Tools • There are three main methods for troubleshooting: • Bottom-Up Troubleshooting Method • In bottom-up troubleshooting you start with the physical components of the network and move up through the layers. • Bottom-up troubleshooting is a good approach to use when the problem is suspected to be a physical one. • Top-Down Troubleshooting Method • In top-down troubleshooting your start with the end-user applications and move down the layers of the OSI model. • Use this approach for simpler problems or when you think the problem is with a piece of software. • Divide-and-Conquer Troubleshooting Method • In divide-and-conquer troubleshooting you start by collecting user experience of the problem, document the symptoms and then, using that information, make an informed guess as to which OSI layer to start your investigation. • For example, if users can't access the web server and you can ping the server, then you know that the problem is above Layer 3. • If you can't ping the server, then you know the problem is likely at a lower OSI layer.

Describe Troubleshooting Methodologies and Troubleshooting Tools • To quickly resolve network problems, take the time to select the most effective troubleshooting method. • Use the process shown in the figure to help you select the most efficient troubleshooting method. • For example: Two IP routers are not exchanging routing information. The last time this type of problem occurred it was a protocol issue. So you choose the divide-and-conquer troubleshooting method.

Gathering Symptoms • Step 1. Analyze existing symptoms • Analyze symptoms gathered from the trouble ticket or users to form a definition of the problem. • Step 2. Determine ownership • If problem is within your system, move onto next stage. • If the problem is outside the boundary of your control, for example, lost Internet connectivity you need to contact an administrator for the external system. • Step 3. Narrow the scope • Determine if the problem is at the core, distribution, or access layer of the network. • Step 4. Gather symptoms from suspect devices • Use knowledge and experience to determine if the problem is a hardware or software problem. • Step 5. Document symptoms • Sometimes the problem can be solved using the documented symptoms. If not, begin the isolating phase of the general troubleshooting process.

Gathering Symptoms • Use the Cisco IOS commands to gather symptoms about the network. • Although the debug command is an important tool for gathering symptoms it generates a large amount of console message traffic and the performance of a network device can be noticeably affected. • Make sure you warn network users that a troubleshooting effort is underway and that network performance may be affected. • Remember to disable debugging when you are done.

Describe Troubleshooting Methodologies and Troubleshooting Tools • NMS Tools • Network management system (NMS) tools include device-level monitoring, configuration, and fault management tools. • Network monitoring software graphically displays a physical view of network devices, allowing network managers to monitor remote devices without physically checking them. • Examples are CiscoView, HP Openview, Solar Winds, and What's Up Gold. • Knowledge Bases • On-line network device vendor knowledge bases have become indispensable sources of information. • When vendor-based knowledge bases are combined with Internet search engines like Google, a network administrator has access to a vast pool of experience-based information.

Software Troubleshooting Tools • Baselining Tools • For example they can help you draw network diagrams, help you to keep network software and hardware documentation up-to-date and help you to cost-effectively measure baseline network bandwidth use. • Many tools for automating the network documentation and baselining process are available. • The figure shows a screen chapter of the SolarWinds LAN surveyor and CyberGauge software. • Protocol Analyzers • A protocol analyzer decodes the various protocol layers in a recorded frame and presents this information in a relatively easy to use format. • The figure shows a screen capture of the Wireshark protocol analyzer. • Most protocol analyzers can filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured.

Hardware Troubleshooting Tools • Network Analysis Module • A network analysis module (NAM) can be installed in Cisco Catalyst 6500 series switches and Cisco 7600 series routers to provide a graphical representation of traffic. • Digital Multimeters • Digital multimeters (DMMs) are test instruments that are used to directly measure electrical values of voltage, current, and resistance. • Cable Testers • Cabling testers can be used to detect broken wires, crossed-over wiring, shorted connections, and improperly paired connections. • These devices can be inexpensive continuity testers, moderately priced data cabling testers, or expensive time-domain reflectometers (TDRs). • TDRs are used to test the distance to a break in a cable. • TDRs used to test fiber optic cables are known as optical time-domain reflectometers (OTDRs).

Hardware Troubleshooting Tools • Cable Analyzers • Cable analyzers are multifunctional handheld devices that are used to test and certify copper and fiber cables for different services and standards. • The more sophisticated tools include advanced troubleshooting diagnostics that measure distance to performance defect (NEXT, RL), identify corrective actions, and graphically display crosstalk and impedance behavior. • Portable Network Analyzers • Portable devices that are used for troubleshooting switched networks and VLANs. • By plugging the network analyzer in anywhere on the network, a network engineer can see the switch port to which the device is connected and the average and peak utilization. • The analyzer can also be used to discover VLAN configuration, identify top network talkers, analyze network traffic, and view interface details.

Describe the Common Issues that Occur During WAN Implementation • WAN technologies function at the lower three layers of the OSI reference model. • A communications provider normally owns the data links that make up a WAN. • The links are made available to subscribers for a fee and are used to interconnect LANs or connect to remote networks. • WAN data transfer speed (bandwidth) is considerably slower than the common LAN bandwidth. • The charges for link provision are the major cost element, therefore the WAN implementation must aim to provide maximum bandwidth at acceptable cost.

Describe the Common Issues that Occur During WAN Implementation • WAN connectivity is important to business and expensive, these are the steps for designing or modifying a WAN: • Step 1. Locate LANs - Establish the source and destination endpoints that will connect through the WAN. • Step 2. Analyze traffic - Know what data traffic must be carried, its origin, and its destination. • Step 3. Plan the topology - A high requirement for availability requires extra links that provide alternative data paths for redundancy and load balancing. • Step 4. Estimate the required bandwidth - Traffic on the links may have varying requirements for latency and jitter. • Step 5. Choose the WAN technology - Suitable link technologies must be selected. • Step 6. Evaluate costs - When all the requirements are established, installation and operational costs for the WAN can be determined and compared with the business need driving the WAN implementation.

WAN Traffic Considerations • The table in the figure shows the wide variety of traffic types and their varying requirements of bandwidth, latency, and jitter that WAN links are required to carry. • To determine traffic flow conditions and timing of a WAN link, you need to analyze the traffic characteristics specific to each LAN that is connected to the WAN.

Describe the Common Issues that Occur During WAN Implementation • Describe the considerations for designing a WAN topology

Describe the Common Issues that Occur During WAN Implementation • Describe common WAN implementation issues

Describe the Common Issues that Occur During WAN Implementation • Describe the recommended steps for troubleshooting a WAN

Troubleshoot Enterprise Network Implementation Issues • Explain how network diagrams are used for troubleshooting

To isolate problems at the physical layers • Check for bad cables or connections • Verify that the cable is properly connected and is in good condition. Your cable tester might reveal an open wire. • Check that the correct cabling standard is adhered to throughout the network • Verify that the proper cable is being used. For example, in the figure, the Fluke meter detected that a cable was good for Fast Ethernet, it is not qualified to support 1000BASE-T. • Check that devices are cabled correctly • Check that cables are connected to their correct ports. • This is where having a neat and organized wiring closet saves you a great deal of time. • Verify proper interface configurations • Check that all switch ports are set in the correct VLAN and, speed, and duplex settings are correctly configured. • Check operational statistics and data error rates • Use Cisco show commands to check for statistics such as collisions and input and output errors.

Symptoms of Data Link Layer Problems • Common symptoms at the data link layer include: • No functionality or connectivity at network layer or above • Some Layer 2 problems can stop the frames across a link. • Network is operating below baseline performance levels • There are two types of suboptimal Layer 2 operation: • Frames take an illogical path to their destination but do arrive. An example of a problem which could cause frames to take a suboptimal path is a poorly designed Layer 2 spanning-tree. • Some frames are dropped. An extended or continuous ping also reveals if frames are being dropped. • Excessive broadcasts • Excessive broadcasts result from one of the following: • Poorly programmed or configured applications • Large Layer 2 broadcast domains • Underlying network problems, such as STP loops. • Console messages • In some instances, a router recognizes a Layer 2 problem has occurred and sends alert messages to the console. • The most common console is line protocol down message.

Troubleshooting Layer 3 Problems • In most networks, static routes are used in combination with dynamic routing protocols. • Improper configuration of static routes can lead to less than optimal routing and, in some cases the network to become unreachable. • Here are some possible problems involving routing protocols: • General network issues • Often a change in the topology, such as a down link, may have affects on other areas that might not be obvious at the time. • Connectivity issues • Check for any equipment problems, cabling, and ISP problems. • Neighbor issues • Check if there are any problems with the routers forming neighbor. • Topology database • Check the topology table, for any missing or unexpected entries. • Routing table • Check the routing table for anything missing or unexpected routes. • Use debug commands to view routing updates and maintenance.

Transport Layer Troubleshooting: Access List Issues 1. Selection of traffic flow • ACL must be applied to the correct interface, and correct traffic direction must be selected to function properly. • If the router is running both ACLs and NAT, the order in which each of these technologies is applied is important: • Inbound traffic is processed by the inbound ACL before being processed by outside-to-inside NAT. • Outbound traffic is processed by the outbound ACL after being processed by inside-to-outside NAT. 2. Order of access control elements • The elements ACL should be from specific to general. 3. Implicit deny all • Forgetting about this implicit access control element may be the cause of an ACL misconfiguration. 4. Addresses and wildcard masks • Complex wildcard masks provide significant improvements in efficiency, but are more subject to configuration errors. • The address 10.0.32.0 and wildcard mask 0.0.32.15 to select the first 15 host addresses in either the 10.0.0.0 or 10.0.32.0 network.

Transport Layer Troubleshooting: Access List Issues 5. Selection of transport layer protocol • When configuring ACLs, it is important that only the correct transport layer protocols [TCP, UDP] be specified. 6. Source and destination ports • Address and port information for traffic generated by a replying host is the mirror address and port from the source host. 7. Use of the established keyword • If the keyword is applied to an outbound ACL, unexpected results may occur. 8. Uncommon protocols • Uncommon protocols that are gaining popularity are VPN and encryption protocols. • Troubleshooting Access Control Lists • A useful command for viewing ACL operation is the log keyword on ACL entries. • This keyword instructs the router to place an entry in the system log whenever that entry condition is matched.

Application Layer Overview • The most widely known application layer protocols: • Telnet - Enables users to establish terminal session connections with remote hosts. • HTTP - Supports the exchanging of text, graphic, sound, video, and other multimedia files on the web. • FTP - Performs interactive file transfers between hosts. • TFTP - Performs basic interactive file transfers typically between hosts and networking devices. • SMTP - Supports basic message delivery services. • POP - Connects to mail servers and downloads e-mail. • Simple Network Management Protocol (SNMP) - Collects management information from network devices. • DNS - Maps IP addresses to the names assigned to network devices. • Network File System (NFS) - Enables computers to mount drives on remote hosts and operate them as if they were local drives.

Symptoms of Application Layer Problems • A problem at the application layer can result in unreachable or unusable resources when the physical, data link, network, and transport layers are functional. • It is possible to have full network connectivity, but the application simply cannot provide data. • Another type of problem at the application layer occurs when the physical, data link, network, and transport layers are functional, but the data transfer and requests for network services from a single network service or application do not meet the normal expectations of a user. • A problem at the application layer may cause users to complain that the network or the particular application that they are working with is sluggish or slower than usual when transferring data or requesting network services.

Troubleshooting Application Layer Problems • The steps for troubleshooting application layer problems are: • Step 1. Ping the default gateway. • If successful, Layer 1 and Layer 2 services are functioning properly. • Step 2. Verify end-to-end connectivity. • If Layers 1-3 functioning properly, the issue exist at a higher layer. • Step 3. Verify access list and NAT operation. • If the ACLs and NAT are functioning as expected, the problem must lie in a higher layer. • Step 4. Troubleshoot upper layer protocol connectivity. • Upper layer protocol, such as FTP, HTTP, or Telnet ride on top of the basic IP transport but are subject to protocol-specific problems relating to packet filters and firewalls. • Troubleshooting an upper layer protocol connectivity problem requires understanding the process of the protocol. • This information is usually found in the latest RFC for the protocol or on the developer web page.

Correcting Application Layer Problems • The steps for correcting application layer problems are: • Step 1: Make a backup. • Ensure that a valid configuration has been saved. • Step 2: Make an initial configuration change. • Make only one change at a time. • Step 3: Evaluate each change and its results. • If the results of any problem-solving steps are unsuccessful, immediately undo the changes. • Step 4: Determine if the change solves the problem. • Verify that the change actually resolves the problem without introducing any new problems. • If the problem is not solved, undo all the changes. • Step 5: Stop when the problem is solved. • Step 6: If necessary, get assistance from outside resources. • This may be a co-worker, a consultant, or TAC. • Step 7: Document. • Once the problem is resolved, document the solution.

Summary • Network Baseline How a network is expected to perform under normal conditions • Network documentation should include: • Network configuration table • End-system configuration table • Network topology diagram • Planning for the 1st baseline • Determine what type of data to collect • Identify devices and ports of interest • Determine baseline duration

Summary • 3 stages of the troubleshooting process • Gather symptoms • Isolate problem • Correct problem • 3 main methods for troubleshooting a network • Bottom up • Top down • Divide & conquer

Summary • Software troubleshooting tools • Cisco view • Solar winds • HP Open view • Hardware troubleshooting tools • Network analysis mode • Digital multi-meters • Cable testers • Network analyzer

Summary • Common WAN implementation issues include • QoS • Reliability • Security • Latency • Confidentiality • Public or Private • Using a layered approach to troubleshooting aids in isolating and solving the problem

Network Troubleshooting