The Art of Network Troubleshooting

1. The Art of Network Troubleshooting How to Fix Any Problem

2. Two Dozen Simple Rules • We’re all smart people • But we sometimes repeat old mistakes • Or forget old lessons • So let’s examine and review some old troubleshooting techniques. • Lest we forget again

3. Isolate the Problem C Si

4. Write Things Down • Keep a notebook or written log • Electronic PDA logs work well • When you attack a problem, start an entry • Values of logging: • You have a record when the problem reoccurs • You buy yourself some time • You wake up other parts of your brain when you write it down • It forces you to express the problem

5. What Changed?? • What have you installed lately? • What did your network admin change? • What updates have you loaded? • What AV software have you loaded? • What anti-Spyware have you installed? • What has been downloaded? • Any hardware been added? • Any software installations or upgrades of any kind?

6. Use Your References • Odie • Microsoft Knowledge Base • Google It! / Google groups. • www.eventid.net • Other Vendors support sites • Online forums • www.windowsitpro.com

7. Record the Exact Error Message • Look at Event Logs • System and Application • Write down Event ID and description • Write down any error codes in message • Use for knowledge base queries

8. Double-check Antivirus • Users shut them off and forget to turn them back on • Check DAT file dates • http://housecall.antivirus.com can be used to scan hardware • Ultimately the best AV tool is. . . A working brain

9. Wait 15 Minutes • Microsoft favorite time interval • WINs • Server AD replication • Group policies

10. Double check the obvious • Are things plugged in? • I mean really plugged in? • Network cables, I/O, power • Switches, hubs, routers

11. Assemble your Toolkit • Software tools • Hardware tools • Vendor phone numbers • List of URL’s • Service packs, patches • Resource kit tools / Support tools • BartPE • Linux rootkit Virus checkers • Always “cold boot” when using a bootable CD

12. Check IP Connectivity • Ping • Pathping • Tracert • Use tracert –d ipnumber to avoid DNS • If ICMP is filtered use portqry (KB 310099) • Always ping the IP address instead of the DNS name.

13. Portqry Syntax • Portqry –n targetsystem [options] • Options: • -e ntry port number n • -p tcp, -p udp, -p both protocol to use • -r n:m range of ports • -o a,b,c,d list of ports • -i don’t reverse-resolve

14. Portqry Examples • Check for a web server • portqry –n 10.0.0.2 –e 80 -i • portqry –n 10.0.0.2 –o 80,443 –I • Scan TCP ports 130-139 • portqry 10.0.0.2 –r 130:139 -i

15. No-Ping Ping Test C:\>ping –n 1 207.46.134.222 Pinging 207.46.134.222 with 32 bytes of data Request timed out C:\>portqry –n 207.46.134.222 –e 80 –I Querying target system called: 207.46.134.222 TCP Port 80 (http service): LISTENING

16. Separate the Name Resolution • WINS vs. DNS • For DNS use nslookup or dnslint • For WINS use nblookup • Don’t forget about the local lookup files • HOSTS • LMHOSTS

17. Check the Logs • Windows does not necessarily write error messages to the console • Look at all the logs • Consider enabling the security logs • Search Microsoft for eventcombmt tool

18. Simplify the Problem • Remove “in between” components • Firewalls, AV, Anti-spyware • Remove extraneous components • extra protocols? • Investigate binding order • Remove name resolution by trying to access by IP number

19. Simplify the Problem (cont) • Does turning something off make the problem go away? • What’s the client scope? Single client? Group of clients? All clients? • What’s the server scope? One server? All servers? Internet access? • Are the failing machines related by network segment? by physical location? by Active Directory tree or subtree?

20. Hardware Breaks • We tend to blame the software. • Lightning, surges, heat, etc. can cause flaky hardware problems • Corollary: if something is going to break it’ll probably break when it’s new. • Power issues can be back-breakers. • Power bricks, low voltage, power cables, etc.

21. Best way to fix hardware • All too often hardware doesn’t die, it just gets a little sickly. • The best and often only way to diagnose this is to swap it out. • Spare switches, cables, are essential

22. Reboot! • If you make a change and it doesn’t’ take • REBOOT • For network devices like routers, modems, switches • Turn them off, count to five, turn them on. • To force GPOs it may take two or even three reboots. • Windows Updates often require a reboot

23. Know Your Network • Map and list IP addresses, MAC addresses, OSes, software revision levels • Document WAPs, hubs, switches, routers • Build the map when the network is functioning---BEFORE it breaks

24. Know What “Normal” Is • Observe the network devices when things are good. • What indicators are on? • What should they look like? • Take a digital picture of a working network device and keep them somewhere handy.

25. Make One Change at a Time • The possibilities grow exponentially with multiple changes. • With one change (A), it could only be due to A or something that was going to happen anyway (status quo (S)) • With two (A,B) it could be A, B, an AB interaction, or S • With three: A, B, C, AB, AC, CB, ABC, S • And so forth. . .

26. Consider using a Network Monitor • Windows Network Monitor • Ethereal (www.ethereal.com) • WildPackets • You don’t need to be an expert

27. Keep an External Address • Ultimate test is whether you can reach the “outside” and they can reach you. • An outside email address can also be helpful (hotmail, yahoo, gmail, etc.)

28. Check Security and Permissions • Windows “Rights” as well as permissions • Consider auditing “processes” to see if something can’t run because of permissions.

29. Walk around the block and/or explain the problem to someone.