Troubleshooting Active Directory: Advanced Design & Techniques

1. Advanced Active Directory Design and Troubleshooting Ed Whittington Principal Software Engineer HP Business Critical Call Center Oct. 06, 2002

2. Topics • Troubleshooting Basics • Troubleshooting Tools • DNS Troubleshooting • Troubleshooting Replication • Troubleshooting DCPromo • Troubleshooting FRS Replication and DFS • Troubleshooting Group Policy • Troubleshooting in .NET

3. Troubleshooting Basics

4. Basic Troubleshooting Steps • Define the problem (make sure there is one) • What’s failing? • Client authentication and security • Group policy application. • Replication. • Name resolution. • Errors and warnings in event logs. • FRS/DFS • Application • How is the problem replicated? • One or multiple machines? • Narrow the variables

5. Basic Troubleshooting Steps • MPSReports_DS (from HP or Microsoft) • Get the Log files • Event logs • http://www.eventid.net • %windir%\debug\usermode\Userenv.log • %windir%\debug\DCPromo*.log • Turn on Verbose Logging • Run NetDiag, DCDiag (verbose) • Get status report from Replication Monitor.

6. Basic Troubleshooting Steps • Check DNS. • Resolver on ALL computers. • Name Server Properties (forwarding, etc.). • Monitoring tab – test name resolution. • Nslookup, ping to test name resolution. • Ping SRV records. • Check Replication. • Force replication. • Identify who isn’t replicating to whom. • Outbound vs. inbound.

7. Basic Troubleshooting Steps • If all else fails, try demoting. • Really cleans up a lot of problems… If problem is isolated to one DC. • If replication isn’t working, demotion won’t work. • Reinstall to remove the AD, then clean up AD • Ntdsutil to remove server object. • Delete server object from Sites & Services. • Delete FRS server object from System container. • Can manually demote a DC.

8. Manual Demotion of a DC • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet • \Control\ProductOptions • Product Type= • ServerNT (when the computer is a Member Server) • LanManNT (when the computer is a Domain Controller) • Change from LanManNT to ServerNT • It’s now a “dirty” member server • Clean server objects from the AD (Ntdsutil) • Clean up the disk and Registry • Create new Forward Lookup Zone – Bogus.com • Run DCpromo – create new forest for Bogus.com • Demote and eliminate Bogus.com • Wait for Replication • Promote back into domain – use same name if desired • Tool in Windows .NET

9. Troubleshooting Tools Gathering Information

10. Netdiag.exe • NETDIAG.EXE • /v - verbose – always turn this on. • /l - log – writes netdiag.log to default directory. • /d:domain controller – finds DC in domain. • /test: - runs only specified tests. • /skip: - skips specified tests. • Can’t execute remotely. • C:>netdiag /v /l

11. Netdiag.exe • Domain Controller Discovery • Bindings, IP address, Default Gateway tests • DNS tests • NBTstat and WINS ping • Netstat • Route • Trust • Kerberos

12. Dcdiag.exe • DCdiag /v • Domain controller functions of netdiag • More domain-specific • FSMO roles • Connectivity • Replications • Domain controller locator • Intersite “health” • Topology integrity

13. Nltest.exe • /server:servername Sets default server • /dsgetdc:domainname Dsgetdcname API • [ /gc /timeserv /ldap ] • /dclist:domainname Lists DCs in domain • /parentdomain Lists parent domain • /dsgetsite Lists site of server • /dsgetsitecov Lists DC “covering” site • /dcname:domainname Lists PDC for domain • /dcpromo Tests potential success of DCPromo • /whowill:domain user Returns name of DC that will authenticate user

14. Netdom.exe • /join • /add • /reset • /resetpwd • /query FSMO • /trust

15. NTDSUtil • Built-in utility. • Directly accesses Active Directory. • Authoritative Restore. • Can restore an older version of the AD and force it on all DCs to correct variety of problems. • Entire AD or single tree. • Can’t restore the schema. • FSMO Roles. • List, Transfer, Seize roles. • Better than UI – can manipulate all roles in forest and all domains from one utility..

16. NTDSUtil • Metadata Cleanup • Delete orphaned objects. • Servers • Domains • The UI can and will lie to you! Don’t trust it. • Useful tool for listing contents of the AD • Sites, domains, servers, FSMO role holders. • Domains in site. • Servers in domain, servers in site. • Q216364, Q216498, Q230306

17. Gpresult.exe • Run on client • Returns: • Security group membership • User and Computer policy info • GPOs applied to each • Registry settings set in the GPO • Client-side extensions set • Scripts applied • Remember • Policy is cached – reboot / login to clear • Note who authenticating server is • Environmental Variable “logon server” • Much Improved in .NET!

18. GPOtool.exe • Run on domain controller. • Returns: • Analysis of all GPOs in domain. • GUID and friendly name of all GPOs. • DS and Sysvol versions. • Errors encountered. • Good group policy troubleshooting tool. • May take a long time to process (#GPOs)

19. ADSIedit.exe • GUI much like Users & Computers snap-in /Advanced features. • Graphical view of AD. • Like LDP.exe but: • Easier to browse. • Can modify attribute values • Don’t confuse with Users & Computers!

20. LDP.exe • Takes time to set up: • Connect • Bind • View – Tree • Enter DN to start (blank for default) • Exposes attributes quickly, easy to see. • Faster than ADSIedit – no GUI to traverse. • LDAP searches. • Can delete and modify, but not as easy as ADSIedit. • Can execute remotely.

21. DCPromo.log, DCPromoui.log • Located in %systemroot%\debug. • Logged every time dcpromo runs. • DCPromo.log • Shorter. • Appended (read bottom up). • DCPromoUI.log and DCPromoUI.xxxx.log • Results of what is seen in the UI – longer. • Find: Results of getdsdcname, DNS query, Time service sync, authentication, replication, Site info. • Error (0x0) = success – no error . • Error reporting different – read both logs.

22. Userenv.log • Located: %systemroot%\debug\usermode • User environment info: • Group policy (registry) • Client side extensions • Scripts • Security • Increase verbose logging (Q221833) • Take time – read and study and you may be surprised at what you can find!

23. Additional User Mode Logs • Client-side extensions • Registry see Q216357 HKLM\software\Microsoft\WindowsNT\currentversion\winlogon\ GPExtension • Errors created in %windir%\debug\user mode • Named after the .dll • Scripts = Gptext.dll = gptext.log • Folder Redirection = fdeploy.dll = fdeploy.log • Security = scecli.dll = winlogon.log • Q245422 • Produced automatically on error (except winlogon.log) • Check User Mode directory for these files • Invaluable in debugging. Use them!

24. Client Side Extensions (registry)

25. Windows .NET Troubleshooting Tools

26. Remote Desktop Resource Redirection • Client Resources Available when using Terminal Services Remote Desktop • File System – Local drives and Network drives on Local Machine available on Remote machine • Audio – Audio streams such as .wav and .mp3 files can be played through the client sound system. • Port – Applications have access to the serial and parallel ports • Printer – The default local or network printer on the client becomes the default-printing device for the Remote Desktop. • Clipboard – The Remote Desktop and client computer share a clipboard • Terminal Services Virtual Channel Application Programming Interfaces (APIs) are provided to extend client resource redirection for custom applications.

27. WMI • Computer management • Active Directory • Provider: MicrosoftActiveDirectory • Classes: • Replication - See replprov.mof %windir%\system32 • Trust health • Provider: MicrosoftHealthMonitor • Classes: see system32\wbem\trusthm.mof • DNS • Provider: MicrosoftDNS • Classes: system32\wbem\dnsprov.mof • Cluster • MSCluster • Also look in CIM Studio in MSDN

28. WMIC Sample Commands • Look in %windir%\system32\wbem *.mof files for names of providers, classes, etc. • Active Directory • Provider: MicrosoftActiveDirectory • wmic:/namespace: \\root\microsoftactivedirectory PATH msad_replneighbor (shows replication partners) • wmic:/namespace:\\root\rsop\user path RSOP_GPO (lists GPOs with User settings)

29. Admin Tool Improvements • Users and Computers snap-in • Drag and drop. • Multi-select and edit user objects. • Heavily revised object picker. • Users and Computers, Sites and Services, DNS Snap-ins • Saved queries. • Viewing Saved DS, DNS, FRS eventlogs on non-DCs! • .NET Adminpak (only on XP)

30. Command Line Tools • GPresult • Enhanced reporting • DCDiag • dcdiag /test:DCPromo • Repadmin – enhanced reporting • Netdom – computername for DCrename • Others • Shipped on • Service Pack 2 CD (install manually) • .NET Server, AdvSvr CD

31. Windows .NET Improvement to NTDSUtil • Change Offline, DS Repair Mode Password While Online! • NTDSUtil • Set DSRM Password (main menu) • Increases server up-time limited by password change interval in Win2K. • (Had to reboot to DS Repair mode to change.) • Q223301 (Win2K limit) • Cool error message! • Setting password failed. WIN32 Error Code: 0x6ba Error Message: The RPC server is unavailable. See Microsoft Knowledge Base article Q271641 at http://support.microsoft.com for more information.

32. Errors in Windows .NETKinder, Gentler and Report to Microsoft

33. Active Directory Load Balancing Tool • Does the job of branch office deployment. • KCC chooses BHS for connection objects – choose the same one. • Tool allows you to spread the load to other DCs in the site (that have that NC). • ADLB tool modifies the Hub DC’s replication schedules to spread it out over time. • Generates a log – like replmon’s status log. • For Deployments with hundreds of branch offices all replicating to a single hub.. • Tool=no benefit to sites with only one DC per domain.

34. Future: Graphical Replication Monitoring Tool • Very much like ‘Age of Directories’ • Ability to make configuration changes • Not in .NET - maybe Longhorn or Blackcomb?

35. Troubleshooting DNS

36. DNS Resolver Configuration • Win2K clients, servers point to Win2K DNS Name Server that is SOA for their zone. • Don’t point to ISP, other Internal NS. (even as “additional”.) • Keep it simple. • Win2K Name Servers forward to ISP or internal name server hosting registered domain.

37. DNS Name Server Configuration Basics • Dynamic updates = Yes. • Active Directory Integrated Zone • Select one “Primary” • All other ADI Primary NS point to it for DNS • Win2k Name Servers can: • Forward to ISP or Internal NS. • Use root hints (or modify root hints). • Reverse Lookup Zones NOT required • Needed only for tools - NSLookup

38. ADI Primary and Standard Secondary mixed zone • Only a DC can host an ADI primary zone • Member Servers can host Secondary zone • Synch off of an ADI Primary ADI Primary Secondary Secondary ADI Primary ADI Primary

39. DNS Case Study Forwarding corp.net na.corp.net sa.corp.net eu.corp.net na.corp.net Zone xfers Secondary zones sa.corp.net eu.corp.net

40. DNS Case Study corp.net na.corp.net sa.corp.net eu.corp.net eu.corp.net find na.corp.net sa.corp.net na.corp.net

41. With Conditional Forwarding FeatureIn Windows .NET Server… corp.net na.corp.net sa.corp.net eu.corp.net find na.corp.net

42. Problem: SRV records only in Root domain Location of SRV: PDC GC Cname w2k.net corp.com corp.com = Zone Xfer = Forwarder EU.w2k.net NA.w2k.net

43. Solution: Delegate _msdcs zone Location of SRV: PDC GC Cname corp.com _msdcs _tcp _sites _udp w2k.net _msdcs = Delegation = Forwarder EU.w2k.net NA.w2k.net

44. DNS Hotfix • Symptom: Replication breaks • Configuration: Using Secondary Zones for root _msdcs at child domains. • Problem: Serial Number of Secondary zone is higher than the primary – zone transfers stop. • Hotfix Q304653 • The Serial Number Is Decremented in DNS When You Reboot • Solved in .Net

45. DNS Troubleshooting Basics • Check DNS event log (and others). • Check Location of DNS servers. • Usually want Name Server in remote sites. • Check population of SRV records. • _msdcs; _tcp; _udp; _sites • Need Kerberos, LDAP records for each DC. • Correct address, etc. • Can delete, repopulate by restarting netlogon. • Check Delegations – correct names, IP.

46. DNS Troubleshooting Basics • Use of Active Directory Integrated (ADI) zones. • Put standard secondary zones on mbr svrs. • Can clear problems by switching to Std Pri. • Ping DC by SRV record: • ping <guid>.site._msdcs.compaq.com. • Clear the server cache. • Negative Caching problems. • Test – Server Properties – Monitoring tab. • Test – Ping names, NSLookup.

47. Troubleshooting AD Replication

48. Replication Troubleshooting Tools • Event logs – Directory Services, System • Sites and Services snap-in • Age of Directories (AOD) – HP • Replication Monitor • Aelita Event Admin • NetPro Directory Analyzer • Command Line (Support Tools & Res Kit) • DCdiag, Netdiag • Repadmin.exe

49. Event Logs for Replication Troubleshooting • Directory Services Log • 5778 - Subnets not mapped. • Will break client’s “site awareness.” • 1311 - serious - Not enough connectivity. • Connectivity, traffic issue. • Sites with DCs and no site links. • Site topology incorrectly defined. • DNS Lookup failure. • 1772 – RPC Server is unavailable. • Physical connectivity. • DNS.

50. Event Logs for Replication Troubleshooting • System Log • Netlogon errors • Authentication • Trusts • Secure channel • w32Time errors • Kerberos authentication required for replication • DCs must be no more than five minutes out of sync. • Watch time zones!