Troubleshooting Exchange Transport Service

1. Troubleshooting Exchange Transport Service Miha Pihler MVP – EnterpriseSecurity Microsoft Certified Master | Exchange 2010

2. About • Speaker / Trainer • Author

3. Agenda • Understand how Exchange Transport Service Works • One of the most important services • Without it there is no e-mail at all • (Also no spam) ;-) • Troubleshoot common Exchange Transport Service

4. Microsoft Exchange Transport Service

5. Exchange Transport Service (cont.) • MSExchangeTransport.exe • Exchange 2010 and Exchange 2013 • It is parent service and it spawns a child service • EdgeTransport.exe • Child service is actually listening in on TCP port 25 • If child service fails parent notices and respawns service

6. Exchange Transport Service (cont.) • If child service fails multiple times (twice) parent will check messages in queue and it will move problematic message(s) to special queue • Queue is called Poison Queue and you can see it with e.g. get-queue • Message will stay in poison queue until it expires or until Administrator performs some action on it • Removes the messages • Re-submits them

7. Exchange Transport Service (cont.) • EdgeTransport.exe.config.xml • C:\Program Files\Microsoft\Exchange Server\V14\Bin • Settings from the file are applied at start or re-start of the service

8. Exchange queue DB • Queue DB is ESE database • Same rules apply as for any other ESE DB • Exchange queue • C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue • C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue • You can export messages from queue • export-message • Export-Message srv-exch1\366055\230652 | AssembleMessage -Path "c:\temp\email.eml„ • Place the .eml file into Pickup folder • C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Pickup • http://technet.microsoft.com/en-us/library/aa997214.aspx

9. Exchange queue DB (cont.) • Delete queue DB and generate new one • Stop Exchange Transport Service • Rename Queue folder • Start Exchange Transport Service • This will create new clean Queue DB

10. Important log files • If you enabled filtering agents on your Exchange than first log to check should be • C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\AgentLog • What to expect from Agent log • Recipient does not exist • If enabled • Can be enabled/disabled per Accepted Domain • RBL (Realtime Block List) events (if RBL enabled) • SenderID events • E.g. domain does not exist

11. Examples of Agent log events • Youhave to checktheselogs on allservers!

12. Important log files (cont.) • FSEAgentLog • You only need to check this logs if you are using Forefront Protection for Exchange • Similar to AgentLog C:\Program Files (x86)\Microsoft Forefront Protection for Exchange Server\Data\FSEAgentLog

13. Important log files (cont.) • Protocol logs • SmtpReceive • SmtpSend • Must be enabled on connector (disabled by default) C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLog C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog

14. ExamplesofProtocollogs

15. Whatelse to lookfor in protocollogs • Authentication errors • This will happen on internal connectors mostly • It will tell you that your receive AND/OR send connectors are not configured correctly • If you have multiple HUB servers they will always require an authentication before they exchange messages • Make sure that send and receive connectors are configured to send and accept authentication

16. Whatelse to lookfor in protocollogs • Authentication errors • If these connectors are not configured correctly there will be no e-mail going in or out and it will be queued at the last / first server • I see this very often when administrators create new connectors • It is also confusing when selecting Internet or Internal Connector from the wizard

17. Pipeline [PS] C:\Windows\system32>Get-TransportPipeline Event TransportAgents ----- --------------- OnConnectEvent {Connection Filtering Agent, Protocol Analysis ... OnHeloCommand {} OnEhloCommand {} OnAuthCommand {} OnEndOfAuthentication {} OnMailCommand {Connection Filtering Agent, Sender Filter Agent} OnRcptCommand {Connection Filtering Agent, Address Rewriting ... OnDataCommand {} OnEndOfHeaders {Connection Filtering Agent, Address Rewriting ... OnEndOfData {Edge Rule Agent, Protocol Analysis Agent, Atta... OnHelpCommand {} OnNoopCommand {} OnReject {Protocol Analysis Agent} OnRsetCommand {Protocol Analysis Agent} OnDisconnectEvent {Protocol Analysis Agent} OnSubmittedMessage {Address Rewriting Outbound Agent, FSE Routing ... OnResolvedMessage {} OnRoutedMessage {Address Rewriting Outbound Agent} OnCategorizedMessage {}

18. Pipeline with 3rd party add-on [PS] C:\Windows\system32>Get-TransportPipeline Event TransportAgents ----- --------------- OnConnectEvent {} OnHeloCommand {} OnEhloCommand {} OnAuthCommand {} OnEndOfAuthentication {} OnMailCommand {} OnRcptCommand {} OnDataCommand {} OnEndOfHeaders {} OnEndOfData {} OnHelpCommand {} OnNoopCommand {} OnReject {} OnRsetCommand {} OnDisconnectEvent {} OnSubmittedMessage {Exclaimer Mail Disclaimers Routing Agent, Text... OnResolvedMessage {} OnRoutedMessage {Exclaimer Mail Disclaimers Routing Agent, Tran... OnCategorizedMessage {Exclaimer Mail Disclaimers Routing Agent}

19. Agentsand 3rd partyagents • You can disable 3rd party agents • You can‘t disable some built in agents [PS] C:\Windows\system32>Get-TransportAgent Identity Enabled Priority -------- ------- -------- Exclaimer Mail Disclaimers Routing Agent True 1 Transport Rule Agent True 2 Text Messaging Routing Agent True 3 Text Messaging Delivery Agent True 4

20. Problems on the sending side.. • What if the problem is on the sending side… • E.g. problem with script/program sending the e-mail • One way is to use WireShark • But what if we use SMTP with SSL (remember slides before)? • Well first we have to disable SSL

21. Pipeline Tracing • Helps solve some of the hardest problems (without using WireShark and other changes) • Can help you view the message in original state as it enters your organization • Must be enabled and set for specific sender

22. Pipeline Tracing • Set-TransportServer –PipelineTracingSenderAddressmiha.pihler@telnet.si –PipeLineTracingEnabled $True • C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\PipelineTracing\MessageSnapshots\ • Will show up onceyouget a „hit“ on set sender

23. Pipeline Tracing • Log Example

24. Pipeline Tracing • Can be used for other troubleshooting purposes • For example we want to find specific header information

25. Performancecounters • A number of usefull performance counters • For different Connectors (send / receive) • For different agents • …

26. Otherusefullthings to look at • Get-ReceiveConnector • Tarpit • Protocol Errors settings • Get-TransportServer • Configuration or specific transport server • Get-TransportAgent and Get-TransportPipeline • See what agents are installed and where they are used • Get-TransportConfig • Mostly configuration for internal mail flow

27. Summary • When troubleshooting mail delivery: • Know what logs to look at • Agent Log should be first if you are using different filters • It will list ALL e-mail that touched your servers • FSE (Forefront) logs should be next • Don‘t forget to check these logs on ALL your servers • If needed enable and use Pipeline tracing • This is useful if you are troubleshooting delivery into your organization • See how e-mail looks as it arrives to your systems

28. Q&A • Tomorrow morning I have another session Understanding and troubleshooting Kerberos miha.pihler@telnet.si