Shaking Your Network To Bits 2

1. Shaking Your Network To Bits 2 Getting Your Answers - bit By bit -

2. Room Protocols and Specifications • Please set your pager or phone to ‘silent’ or vibrate mode. • Do not save your questions for the end of the day, even if its off topic. Ask your questions when they pop into your head. • There are no ‘dumb questions’. • Don’t be afraid to ask ‘any’ question; even if you feel it is off topic. • I always provide a morning and afternoon break along with lunch, but make sure to let me know if you feel you need a break. I also have feedback links on my website.

3. Standards And Protocols – And Why Bother? What is a Standard/Protocol? • A set of rules to abide by. • Used to ensure interoperability. • Use the OSI standards model as a consistent approach to troubleshooting. • Much like everyone who drives to work, they do so on a road, with speed limits, rules of etiquette and lane dividers. • And of course everybody follows the rules don’t they?? • The most important network issues revolve around performance. • ‘If it ain’t broke, don’t fix it!’ mentality is dangerous and a 10 year old Networking myth. • Things don’t typically break anymore, they slow down! Please stop saying, ‘But it works, right??!!’

4. Protocol and Standards De Facto • Are standards that are created when a majority of the user community purchases product. • Little documentation for troubleshooting and reference. • The vendor can modify the protocol without consulting the user community • (i.e. IBM’s SNA, Microsoft/Red Hat’s TCP/IP Stack, Novell’s IPX) De Jure • Standards derived from collaboration or via a committee. • Documentation readily available for troubleshooting and reference. • (i.e. IEEE, ISO, CCITT, ANSI, TCP/IP RFC’s) License Agreements • When a vendor offers a protocol for use at a cost. • Depending on the vendor, documentation may be available. • (i.e. Xopen SMB, SUN NFS, Novell’s IPX/SPX)

5. OSI - Layer 1 - Physical - Repeaters/NIC Repeater Information • Repeaters operate at the bit level. This means that a repeater has no concept of any layer of addressing. A repeater is a physical layer device that extends the network length and topology by regenerating and retiming the signal one bit at a time. • A repeater repeats every signal that comes in on one port onto every other port. A repeater does not isolate traffic or collisions which is why you use a mini hub to insert between a station and a half duplex switch port when troubleshooting. • A repeater is transparent to other stations on the network. A repeater is not addressable. It does not store and forward data. • A repeater adds latency and should be factored when designing networks. • A 10BASE-T hub acts as a multiport repeater. • Take some time to review what statistics the vendor can provide and what they mean. • Some hubs are not hubs, but switches.

6. OSI - Layer 1 - Physical - Tips • When analyzing a problem, be aware of device location. • Increase traffic on suspect device to correlate errors. • Disconnect or reset a device when you are certain it is the source of your problem. More importantly, record its behavior before and after a reset. • Always try to “Divide and conquer”. • Test cabling to eliminate suspicions. • Get away from ‘Auto’ settings and manually configure ports for specific speeds, duplex mode and connector types. • Always refer to vendor specifications when troubleshooting. • Problems at this layer include; collisions, burst errors, beacons, cable plant difficulties, opens or shorts, SQE, Link Enable, Repeater, RFI, EMI, bad transceivers, environmental problems. • Any cable, punch-down block or patch panel that are not within specification will result in problems and void any certification. • Always physically check Type 1 and BNC connectors since they may visually seem O.K. • A cable exposed to a source of heat or electrical interference will have its’ physical properties changed resulting in physical layer errors.

7. Grabbing The Problem By The Wire • Successful network design, support and implementation starts with layer 1. • How do you ensure the foundation of your network is solid? NetTool screenshots

8. Collisions • When two stations have data to transmit they may both transmit resulting in a collision. • The transmitting adapters senses a collision, stops transmitting, output a 32-bit jam signal, and wait a random amount of time before retransmitting. • If there are repeated collisions, the adapter tries again (up to a total of 16 times). • Each time it retries, it waits a random amount of time. • BackoffTime = RandomNumber multiplied by SlotTime. • SlotTime = time to propagate 512 bits (i.e., 51.2 µseconds). • RandomNumber is greater than or equal to 0 and less than 2n. • n = number of times it has tried for first 10 times or n = 10 for the 11th through 16th try. • After 16 tries, report error to the upper-layer protocol and give up trying to transmit that frame. • Collision/Backoff has no priority mechanism or preference parameters. • Collisions alone do not constitute a problem. Collision? Try Again Later. Transmit Transmit

9. Hex to Decimal to Binary and Back Convert the following A) Dec 125 to Bin ________ B) Dec 11 to Hex________ C) Dec 127 to Bin ________ D) Bin 01010101 to Dec ______ E) Bin 01010101 to Hex _______ F) Bin 10101010 to Hex ________ G) Hex 1A to Dec _______ H) Hex 12 to Bin ________ I) Hex 5 to Dec _____ Better question: Why is it important to know how to do this? A) 1111101 B) B C) 1111111 D) 85 E) 55 F) AA G) 26 H) 10010 I) 5

10. What Does a 10 Mb Collision Look Like? • The preamble from on of the stations gets embedded within another frame. • The alternating zero and ones are displayed as AA’s or 55’s.

11. The preamble from a station gets embedded within another frame. The signaling type of 100Base-T uses 4B5B, which changes the way information is sent over Fast Ethernet. If the 4-bit equivalent of 43 43 is put together, the 5-bit result is a string of 0101010101010101's! Here's a chart that compares Manchester to 4B5B: Decimal 4-bit data group 5-bit symbol ------- ---------------- ------------ 0 0000 11110 1 0001 01001 2 0010 10100 3 0011 10101 4 0100 01010 5 0101 01011 6 0110 01110 What Does a 100 Mb Collision Look Like?

12. NDIS And Transmitting Packets • It is not recommended that an NDIS module be used to transmit packets since the transmit rate will likely be below the specified transmission rate and the transmission of error packets is not supported. • Since NDIS interface filters out frames with errors, only "good" Ethernet frames are captured. The error counter supported through the NDIS interface are those counters supported by the network adapter. Some vendors do not support any error counters. • The minimum and maximum values for the Ethernet Packet Size field are 64 and 1518 bytes (including the 4 Byte CRC). • Packet gaps are not supported. • A monitor on the transmitting station may report a higher utilization than what is actually on the line.

13. OSI - Layer 2 - Datalink • Bridges operate at this level. • At this layer, we talk in Bytes and/or frames. • MAC layer addressing is used at this layer. • Layer 2 switches use the MAC address for forwarding decisions and are basically multi-port bridges. • Since MAC addresses are used, error checking [CRC/FCS] checks are used. • Be aware of various encapsulation types. • Avoid any Auto configuration settings. • When a packet is addressed to either a Broadcast or Multicast address, a bridge forwards the packet to all ports. • Find out if your switch has a mirror port and if the physical level errors are forwarded. • Find out if your MAC drivers pass on physical level errors to your application. • Be careful and specific when discussing Bytes and bits. • For example, 100 Mega Bytes is not 100 Mega bits. • Note any protocols that modify MAC addresses (i.e., Multicast, HSRP, etc..) • For example, HSRP uses the following MAC address, except Token Ring: • 0000.0c07.ac** (where ** is the HSRP group number)

14. Broadcast Storms… The Truth! • Many technologists believe that by installing more bandwidth and collapsing network architectures’, the threat of a broadcast storm disappears. • When several segments are collapsed into one large one, the chance of a broadcast storm increases. • In summary, when you collapse many separate segments into one large one, the workstations/servers will have to process more broadcast packets. So how do you identify and minimize the source of your broadcast packets? • You should plug a protocol aware tool into any switch port configured for a customer VLAN and observe the protocols in use. NO spanning or mirroring is necessary. • Why bother with a little broadcast packet? • With the consolidation of multiple segments and the behavior of automatic settings, the number of broadcasts have been increasing over the past few years. • Many applications rely on broadcasts to locate servers. • Every Broadcast packet requires an interrupt on the listening station for processing. Regardless if the client acts on the packet or not.

15. Impact of Broadcasts % Processor Time Interrupts/sec

16. Getting The Full Picture • SNMP is an excellent way to get visibility back after implementing layer 2-switched networks.

17. What Client Misconfigurations?

18. Windows 2000/XP Bindings Cleanup Default Bindings ‘Clean’ Bindings The IPX boxes should be unchecked since Microsoft clients use IP. IPX is used for Netware/Novell Clients, so bind it to the Netware Client.

19. Microsoft Bindings Via OPE • What the analyst sees with a protocol analyzer. NetBIOS Over UDP NetBIOS Over IPX NetBIOS Over NetBEUI

20. Microsoft Bindings Via Network Inspector

21. You ‘Auto’ Know Better • Please pick a Frame Type!!!!!!!!! • If you don’t know which frame type to use, capture some server responses and use that type. • Not all vendors use the the same terms when describing various frame types. • Many applications carry along their own IPX drivers.

22. OSI - Layer 2 - Encapsulation Types IPX Encapsulation Naming Conventions Cisco IOS Cisco Catalyst Novell Software LSAP Description Naming Switch Naming Naming Convention Convention Convention * Ethernet Novell-Ether 8023RAW Ethernet_802.3 (raw) FFFF Ethernet with no LLC or SNAP ARPA Ethernet II (EII) Ethernet_II 8137 Ethernet II s/ type 8137 SAP 8023 Ethernet_802.2 E0E0 Ethernet with 802.2 SNAP SNAP Ethernet_SNAP AAAA Ethernet s/ 802.2 + SNAP FDDI SNAP SNAP FDDI_SNAP AAAA FDDI using 802.2 + SNAP SAP SAP FDDI_802.2 E0E0 FDDI using 802.2 Token Ring SAP n/a Token Ring E0E0 Token Ring w/ 802.2 SNAP n/a Token Ring_SNAP AAAA Token Ring w/ 802.2 + SNAP

23. IPX Frame Types Via NetTool • If you select the PC and Segment ID, you can identify the IPX Frame Type or Encapsulation. • If you don’t require IPX, or had it as a legacy-phased out protocol, ensure it is completely removed. • If you do require IPX, ensure you have the proper frame type selected,

24. IPX Frame Types Via OPE Ether II SAP SNAP RAW

25. IPX Frame Types Via Ethereal SAP SNAP RAW Ethernet II

26. Microsoft Bindings & IPX Frame Types Via NAI NetBIOS Over IP NetBIOS Over IPX NetBIOS Over NetBEUI SAP RAW SNAP Ethernet II

27. Microsoft IPX Bindings Via Optiview These devices have IPX bound to the Microsoft Client and should be cleaned up.

28. Who’s A SAP These devices have IPX bound to the Microsoft Client, which causes these stations to transmit an IPX SAP. Every Novell routing device will collect a SAP and broadcast them out every 60 - 90 seconds.

29. Microsoft NETBEUI Bindings Via Optiview These devices have Netbeui bound to the Microsoft Client.

30. Common NetBIOS NAMES and QUALIFIERS Name Number Type Usage <computername> 00 U Workstation Service <computername> 01 U Messenger Service <\\--__MSBROWSE> 01 G Master Browser <computername> 03 U Messenger Service <computername> 06 U RAS Server Service <computername> 1F U NetDDE Service <computername> 20 U File Server Service <computername> 21 U RAS Client Service <computername> 31 U Modem Sharing Client Service <computername> 43 U SMS Clients Remote Control <computername> 45 U SMS Clients Remote Chat <computername> 46 U SMS Clients Remote Transfer <computername> BE U Network Monitor Agent <computername> BF U Network Monitor Application <username> 03 U Messenger Service <domain> 00 G Domain Name <domain> 1B U Domain Master Browser <domain> 1C G Domain Controllers <domain> 1D U Master Browser <domain> 1E G Browser Service Elections

31. Getting OPE To Speak Your Language

32. OSI - Layer 3 - Routing Analysis Even though we may address devices at layer 3, the lower layer address must be resolved for proper communication. Think about when you address a letter to Jim, but you give the letter first to your mailman. So even though your layer 3 address [Jim] remains unchanged, at layer 2 many other people will handle the mail. At layer 3, you need to understand what or if any values are altered or recalculated. i.e. Time to live, TOS, layer 2 CRC’s, IP checksums, etc. Some protocols will alter the layer 2 addresses, get familiar with these algorithms. i.e Multicast, HSRP

33. OSI - Layer 3 - Routing Operation Command Open File IP = 222.222.222.222 MAC = IBM 123456 IP = 111.111.111.111 MAC = XRCM 123456 IP = ________________ MAC = _____________ ROUTER ROUTER Layer 2 Switch HUB IP = 11.11.11.1 MAC = CSCO 123456 IP = 11.11.11.2 MAC = CSCO 987654 IP = ________________ MAC = _____________ IP = ________________ MAC = _____________ IP = 111.111.111.1 MAC = CSCO ABCDEF IP = 222.222.222.1 MAC = CSCO 99999

34. OSI - Layer 3 - Routing Troubleshooting Tips • To capture traffic that is passing through the segment. • Filter on traffic between two routers’ MAC address. • To capture packets leaving your segment. • Filter on packets with the router as the destination MAC address. • To capture packets entering your segment. • Filter on packets with the router as the source MAC address. • Gain a comprehension of layer 3 protocols as they traverse the network. • IP Time to Live, MAC and CRC changes, etc.. • IPX hop counts • Standby Routing Protocol • Be aware of queue latency, filter latency and fragmentation issues. • Trace route packets and document for visualization. • Be careful of multiple serial linked protocols and load balancing applications. • Attempt to establish a baseline for reference. • Protocols at this layer are generally connectionless. (i.e. IP and IPX)

35. Why should I ‘Pathping’? • Provides information about network latency and network loss at intermediate hops between a source and destination. • Transmits multiple ICMP Echo Request messages to each router between a source and destination over a period of time and then computes results based on the packets returned from each router. • Because pathping displays the degree of packet loss at any given router or link, you get an idea which routers or subnets might be having network problems. • Pathping performs the equivalent of the tracert command by identifying which routers are on the path. • It then sends pings periodically to all of the routers over a specified time period and computes statistics based on the number returned from each. • If you type pathping without parameters, you display the helpscreen.

36. Pathping Syntax pathping [-n] [-h MaximumHops] [-g HostList] [-p Period] [-q NumQueries [-w Timeout] [-T] [-R] [TargetName] Parameter Definition; • -n Prevents pathping from attempting to resolve the IP addresses of intermediate routers to their names. This may speed up displaying pathping results. • -h MaximumHops Specifies the maximum number of hops in the path to search for the target (destination). Default = 30 hops. • -g HostList Specifies that the Echo Request messages use the Loose Source Route option in the IP header with the set of intermediate destinations specified in HostList. With loose source routing, successive intermediate destinations can be separated by one or multiple routers. The maximum number of addresses or names in the host list is 9. The HostList is a series of IP addresses (in dotted decimal notation) separated by spaces. • -p Period Specifies the number of milliseconds to wait between consecutive pings. Default = 250 milliseconds (1/4 second). • -q NumQueries Specifies the number of Echo Request messages sent to each router in the path. Default = 100 queries. • -w Timeout Specifies the number of milliseconds to wait for each reply. The default is 3000 milliseconds (3 seconds). • -T Attaches a layer-2 priority tag (for example, 802.1p) to the Echo Request messages that it sends to each of the network devices along the route. This helps to identify network devices that do not have layer-2 priority capability. This switch is used to test for Quality of Service (QoS) connectivity. • -R Determines whether each network device along the route supports the Resource Reservation Protocol (RSVP), which allows the host computer to reserve a specified amount of bandwidth for a data stream. This switch is used to test for Quality of Service (QoS) connectivity. • TargetName Specifies the destination, which is identified either by IP address or host name. • /? Displays help at the command prompt.

37. Pathping Notes • Pathping parameters are case-sensitive. • To avoid network congestion and to minimize the effects of burst losses, the period parameter must be used with caution. • When using the -T parameter Enabling layer-2 priority on the host computer allows packets to be sent with a layer-2 priority tag, which can be used by layer-2 devices to assign a priority to the packet. Devices that do not recognize layer-2 priority will discard these packets, since they appear to be malformed. • When using the -R parameter An RSVP reservation message for a nonexistent session is sent to each network device on the route. If the device does not support RSVP, it returns an Internet Control Message Protocol (ICMP) Destination Unreachable-Protocol Unreachable message. If the device does support RSVP, it returns an RSVP Reservation Error message. Some devices might not return either of these messages. If this occurs, a time-out message is displayed. • Unlike ping, packet size may not be specified.

38. OSI - Layer 4 • Protocols at this layer can be connection oriented or connectionless. • A list of TCP and UDP ports can be found at http://www.isi.edu/in-notes/rfc1700.txt. • Hard wired time-outs, retransmission of missing or late packets, overflow of buffer windows can cause problems. • Learn how a protocol determines if an acknowledgment was sent. And the corresponding retransmission algorithm. • Document any legacy UDP or TCP port numbers in use. • Document any Layer Four defaults for various operating systems • i.e TCP window size for win 98 is 8192 and the TTL is 128 • Remember that different applications use different TCP ports. When a client reports that they can not login, get a utility that can check application ports. • Find out how if your protocol analyzer can add ports to its configuration file.

39. Sample Packet Breakout Data Link Network Transport Application

40. Network Documentation Methods Notes and Observations Include date, people involved, any correspondence, application or problem details, troubleshooting methodology, any bug reports or follow up points. Vendor Information Trace Files rfc’s Departmental Server Storage Media. Make a copy for anybody who was involved.

41. Latency Analysis • To truly understand if you have an issue with latency, you must have a point of reference to compare your results to. • Over time, you will learn you true latency values

42. Example Of Latency Analysis • In this example, we know (from our chart) that the theoretical transmit time for 1 1514 (without the CRC) packet is approximately 1.21 milliseconds on 10 Mb Ethernet. • We all know we will never achieve this speed, but lets see how we do. • Can we determine the real source of latency?

43. TCP Three-way Handshake Initiating Application Port Number (Port 80) HTTP Sequence Number + 1 Sequence Number + 1 The delta value between frames 2 and 3 can be used as a TCP transport connect baseline value. Other important information gathered from this handshake: • Window Size • SACK • Maximum Segment Size • Window Scale Option value

44. TCP Flow Control Kicks In

45. Discover And Change TCP Window Size http://www.dslreports.com/front/drtcp.html

46. Effect of Modifying TCP Window Size

47. TCP Window Information • The Expert reports “Window Frozen” statistics. • Determine the highest TCP Window Size and MSS value (TCP 3 way handshake). • Then compare is to the ‘Frozen’ value. • If the Window size is less than twice the MSS (1460 is Ethernet’s default), then get concerned. • In the “Silly Window Syndrome,” the receiver keeps advertising a small window and sender keeps filling it with small packets. • Can you configure it to use a larger size? Typical misconfigured TCP stacks will be configured for a 512 byte MSS. • “Zero Window” symptom alerts you to stations that have closed their window. • Don’t worry if the window closes briefly at the beginning of a connection, then opens and maintains a reasonable size. • Do worry if a host frequently closes the window for long periods of time. • TCP MAY keep its receive window closed indefinitely. As long as the receiving TCP continues to send acknowledgments in response to the probe segments, the sending TCP MUST allow the connection to stay open.

48. What is a Frozen Window and is it ‘BAD’? • Well first we have to understand what the message is telling us and how it is triggered. • If you click on the question mark and search for ‘Frozen Window’, you’ll find the following information. Huh ?

49. How Is It Triggered? • When we investigate the Expert Settings, you can see that the Frozen Window timer is set for 5 seconds. • To change this setting, double-click on the number 5 and enter a new value.

50. What Does It All Mean? • When a station (using TCP) receives data, it will Acknowledge that data segment. • Along with that information, it will add the current value of it’s TCP receive buffer. • If the Window size transmitted is consistent for longer than the 5 second threshold, the “Frozen’ alarm is triggered. • If you go back to the TCP 3 way handshake, you can determine what the initial TCP window is. • In many cases you’ll find there’s nothing to worry about it.