MSG323 Exchange Transport Monitoring and Troubleshooting

1. MSG323Exchange Transport Monitoring and Troubleshooting Max CiccotostoProgram Manager - Exchange Server Microsoft Corporation

2. Why Are We Here? • Learn about Transport in Exchange 2003 • Common Scenarios • Messages are not routed • <X> queue keeps growing • NDRs are returned for unidentifiable reason • Client has problems with messages/content • Authentication failures • Monitoring is essential to preventing problems, catching them before they happen

3. Agenda • Transport in Exchange • What’s new in Exchange 2003 • Exchange Mailflow • Common Troubleshooting Scenarios • DNS and Connectivity • Working with Queues • Routing and Connectors • Content and Message Properties • Dealing with NDRs • Tracking Down Messages • Setting up Counters and Logging • Monitoring Best Practices • Q&A

4. New in Exchange 2003 • Easier to support: • More Queues (Hidden Queues) • Improved Message Tracking • Improved Logging (DSN Logging) • New Internet Email Connection Wizard • New DNS Resolver tool • Routing Performance and Stability improvements • New filtering and antispam features • Query-based Distributed Groups (QDGs) • Journaling is BCC enabled

5. Transport in Exchange

6. What Is The Transport? • Handles message delivery and routing details, examples: • Look up users in AD, expand DLs • Dynamic routing logic • Picks up and delivers from/to Store • Handles SMTP protocol • Handles queuing of messages • Touches every message, even local-to- -local user!

7. Transport Essentials • Server-Server transport • In Exchange 2003: SMTP native transport • Interoperability • Exchange 2003 can talk to Exchange 5.5 via RPC • Support X.400 connectors, EDK (foreign) gateways • No IMC/IMS needed for Exchange 2000 – Exchange 5.5 interoperability • MTA still there • Used for X.400 and RPC interoperability

8. RPC RPC SMTP RPC RPC RPC 5.5 Server 2003 Server 2003 Server 5.5 Server Exchange 2000 within an Exchange 5.5 site

9. SMTP Service • Uses Windows® 2000 SMTP Service • Protocol events - extend the SMTP protocol • Transport events - extend the function of the Transport Core • Multi-threaded, high-performance

10. Transport Core Categorizer • Categorizer: Component that resolves sender and recipients against Active Directory • Limited Categorizer ships with Windows 2000, disabled • Can do some Directory access, such as expanding mail-enabled Groups • Enhanced Categorizer ships with Exchange 2000 • Adds Exchange features, such as Recipient Limit checking, reading Home-MDB

11. Transport Core Routing • Advanced link-state based routing engine replaces RID Server and GWART used by Exchange 5.5 MTA • MTA uses same engine when it needs to compute next hop, so X.400/RPC connectors get benefit • Size, priority, sender of message; cost and state of links used to compute path

12. Transport Core Store Driver • The interface between Transport and the Store • Uses “ExIPC” for inter-process communication, IFS for large data transfer (message body) • Windows 2000 SMTP Service: NTFS store driver • Exchange 2000: Exchange Web Storage store driver

13. Transport Dependencies • Check these as necessary: • Transport looks up User info in AD • Routing configuration read from AD on startup, link state info kept in memory • DNS used to resolve names, even internal servers! • DS2MB replicates info to MB for Core SMTP • Transport queues messages in Store, delivers to/from store • End client – e.g. auth problems. NOTE: Client-DC issues are not Exchange!

14. Queues – Basic Flowchart NTFS RoutingEngine Active Directory Remote Delivery Queue SMTPProtocol SMTPProtocol Categorizer Queueing Remote Delivery Queue Local Delivery Queue MAPI / OWA Client MAPI / OWA Client InformationStore InformationStore MTA (X400) MTA (X400)

15. A More Advanced View…

16. Topic 1:Internet Connectivity

17. Email From Internet • Possible reasons: • Internet DNS is mis-configured • Recipient Policy does not contain the domain • You maybe filtering the specific domain/IP • Tips: • Use a tool such as www.dnsreport.com • Check the type of NDR that sender gets • Specific domains or everyone? • Check SMTP Logs • There is a lot of information in the KBs!!! • Search for Shared Domain, SMTP and DNS • SMTP Greenbook

18. Send Mail to The Internet • Possible reasons: • You cannot reach the Internet DNS • Smarthost mis-configured (permissions, IP) • Domain not properly registered with DNS (Reverse Lookup enabled on the recipient SMTP) • Tips: • Use a tool such as www.dnsreport.com • Check the type of NDR that sender gets (5.7.1) • Verify info/status with your ISP • Verify if you can connect to remote server • Check Real-time denied lists • There is a lot of information in the KBs!!! • Search for Shared Domain, SMTP and DNS • SMTP Greenbook

19. Topic 2: Working with Queues

20. Queue Problems • Symptoms: • Queue growth ‘abnormal’ – monitoring tools report queue grows beyond threshold • Messages “stuck” in Queue – these messages stay in queue, do not get delivered • Messages waiting to be delivered to external domains (DNS problems) • Internal queues grow (waiting AD lookup, local delivery)

21. TroubleshootingQueues – What are they for?

22. TroubleshootingQueues – Local Delivery • Local Delivery Queue • Messages awaiting delivery to the Information Store • Make sure store is mounted • Could indicate a performance issue • “Poison” Message

23. TroubleshootingQueues – Pre-Submission • Pre-Submission Queue • Messages waiting to be processed by Transport • Exposed for event sink developers • Could indicate a store performance issue or issue with 3rd party event sink

24. TroubleshootingQueues – Pre-Categorization • Pre-Cat Queue • Messages waiting to be processed by Categorizer • Categorizer resolves addresses • Could indicate an issue when talking to Global Catalog • Could indicate a permissions issue

25. TroubleshootingQueues – Pre-Routing • Post-Cat / Pre-Routing Queue • Messages waiting to be routed • Slowdowns usually due to expensive restrictions • Messages w/ Unreachable Destination (not shown) • Indicates Routing failed to find a path for the message

26. TroubleshootingQueues – Remote Delivery • Remote Delivery Queue • Messages being sent to a remote location • Note: Messages may be physically on disk or in the Information Store! – It depends on where the message originated • Use the error message to help focus your troubleshooting (netmon, nslookup, etc.)

27. TroubleshootingQueues – Remote Delivery Queue Error Message

28. TroubleshootingQueues – New in Titanium • Goal was to expose “hidden” queues • Queues • DSN Awaiting Submission • NDR messages that are being submitted • Failed Message Retry Queue • Messages that failed conversion • Deferred Delivery • Messages that have deferred delivery specified by Outlook clients • Standard actions are exposed

29. Fixing Queue Issues • To monitor and Troubleshoot: use WMI or ESM queue viewer • To manipulate queues: use ESM • Verify Dependencies: • DNS external/internal • GC Availability • Store • Routing • Ensure there are no “stuck” messages, if so freeze or delete

30. Tool: Queue Viewer In ESM • Shows queue state and performs actions through ESM, under: • SMTP Protocol • X.400 Service • Actions: Freeze, Delete, Disable queue • WMI access • “Stuck” messages can be frozen or deleted

31. Queue Viewer

32. Topic 3:Routing and Connectors

33. Routing Problems • Symptoms: • Links / Connectors are marked “down” • Topology changes / breaks message path • “Currently unreachable” queue grows • External mail is not routed • NDRs, delayed delivery • Routing specific errors in event logs • Mail “disappears” or gets queued up • Only some mail gets delivered to end-user

34. Fixing Routing Problems • Check topology status: • Confirm routing configuration has not changed (Did you uninstall IIS?) • Ensure master is up, routing service is running Monitor queues, set up counters • Are bridgeheads up? • Are routing groups connected? • DNS internal: check Network-Address AD attribute of destination server, try to resolve name • Use WinRoute to debug topology, link state information • New “Routing and LinkState Whitepaper”

35. Tool: WinRoute • Available: • On CD in SUPPORT\UTILS (Exchange 2000) • Now on “Exchange Tools Site” • Read Q281382: “How to use WinRoute” • Connects to specified host, acts as read-only slave • Displays Link State info packet decoded • Resolves GUIDs against AD • If no AD access, see GUIDs and states • Configure DC hostname if running outside the domain • Can save link state information to file (*.rte files)

36. WinRoute

37. Topic 4: Content and Message Properties

38. Content And Properties • Symptoms: • Messages do not appear correctly to client • Content is not preserved outbound/inbound • Need to check advanced message properties (X-Headers, FROM, TO) • Mail message fidelity is lost • Hard to monitor, rely on user info • To troubleshoot, investigate original message(s), use Archive Sink tool • Don’t forget “Global Settings”

39. SMTP vs. MAPI Submission • SMTP submission happens on protocol level (port 25) • MAPI submission through MAPI client (Outlook), directly to store • Both submission paths go through Transport (Archive Sink) • SMTP mail is MIME encoded, MAPI is MS-TNEF encoded • Archive Sink captures both

40. Tool: Archive Sink • Available: • On CD in SUPPORT\UTILS (Exchange 2000) • Now on “Exchange Tools Site” • Hooks on 2 possible Transport Events and dumps message properties (regkey) • New version: • three files per message: xml, .eml, p1 stream • Works on multiple Virtual SMTP servers • Archives both MAPI and SMTP messages • Not to be confused with Journaling – meant to be troubleshooting tool

41. Archive Sink

42. Topic 5: Dealing with NDRs

43. Non Delivery Reports • It is a symptom • Always check returned NDRs when troubleshooting • Sent to end-user – but can have NDRs also sent to another account(s) • We added many codes • In Exchange 2003 • New log category for DSN (Sev 0-5). • X-Header “Error number” • Added regkey to enable pre-E2K behavior

44. TroubleshootingDelivery Status Notifications Original message (may get “Send Again” form in Outlook) Recipient Check NDR online Server reporting the problem DSN error code

45. NDR Troubleshooting • General steps to follow: • Is it permanent or transient? 4.x.x or 5.x.x? • Check specific diagnostic code (e.g. 5.4.0) • Reference cause/solution KB Q284204 • Is it a client or server problem? • What’s the reporting server? • Can you reach the reporting server? • Can other users send messages? • Worst case: use tools such as Queue viewer, Message tracking or Winroute

46. Topic 6: Tracking down messages

47. Message Tracking • “I sent an email to John two days ago, he as not received it yet” • Useful for: • Diagnosing “missing” or “lost” messages • Discovering the message path – so you can increase logging / tracing • Recording successful / failed deliveries • Gathering statistical data from tracking logs • Tools available: • Message Tracking Center in ESM • Do-it-yourself scripts • Third party products

48. Tracking Details • Per-server • Writes plain text logs to share \\servername\servername.log • Enabled on server object, option to log subject • Turned off by default • In Exchange 2003 we added extra logs • Reference • Use KB Q246959 • Make sure NOT to manually modify logs – can lead to corrupt data

49. Message Tracking UI and Logs

50. Monitoring and Troubleshooting Best Practices