XenApp Escalation Case Studies

1. XenApp Escalation Case Studies Rick Berry Senior Escalation Engineer November 2010

2. Escalation process Case studies review: Intermittent SSON issue on Windows XP XenApp 6.0 Citrix Licensing magic trick XenApp 6.0 hotfix install issue Resources and Tools Open Forum - Questions Citrix Confidential - Do Not Distribute Agenda

3. Escalation Process

4. The 4 W�s: Who Who is affected (end users, servers, etc)? What What are the symptoms, errors displayed, what has changed? Where Where are the errors seen (PC, server, web server)? When When does the error occur, when did the issue start? Citrix Confidential - Do Not Distribute Escalation process

5. Case study #1:Intermittent SSON issue on Windows XP

6. Problem: Intermittently, end users would log in to their workstations and once the Citrix Program Neighborhood Agent would launch (PNAMain.exe), end users would see the following message Environment: Windows XP workstations (SP2 and SP3), locked down configurations Citrix plug-in version 11.2, PNA site configured for pass thru and prompt user Only certain end users would see issue and issue only seemed to occur at certain times Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

7. Who End users on Windows XP workstations (SP2 and SP3) What Error messages displayed when PNA is first launched and will also occur if end user launches published app Where On the end users workstation When Intermittent, after updating to 11.2 plug-in (from 9.233) Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

8. Troubleshooting process: Determine if ssonsvr.exe is running during event Action: Enable process audit tracking Caution: Beware of security log gotchas Enable userenv logging to review startup Action: Enable userenv logging via KB221833 and set UserEnvDebugLevel to 0x00030002 Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

10. Extracted security logs when ssonsvr.exe is working: Security Success Audit process has been created C:\WINDOWS\system32\winlogon.exe Security Success Audit process has been created: C:\WINDOWS\system32\mpnotify.exe Security Success Audit process has been created: C:\Program Files\Citrix\ICA Client\ssoncom.exe Security Success Audit process has been created: C:\Program Files\Citrix\ICA Client\ssonsvr.exe Security Success Audit process has exited: C:\Program Files\Citrix\ICA Client\ssoncom.exe Security Success Audit process has been started C:\Program Files\Citrix\ICA Client\pnagent.exe Security Success Audit process has been started C:\Program Files\Citrix\ICA Client\pnamain.exe Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

11. Extracted security logs when SSON wasn�t working: Security Success Audit process has exited: C:\WINDOWS\system32\winlogon.exe Security Success Audit process has been created: C:\Program Files\Citrix\ICA Client\pnagent.exe Security Success Audit process has been created: C:\Program Files\Citrix\ICA Client\pnamain.exe First major clue: In the non-working scenario, Mpnotify.exe was never launched by Winlogon.exe and therefore Ssoncom.exe and SsonSvr.exe would never run causing the end user to be prompted for their credentials when Pnamain.exe would run (or wfcrun32.exe etc). Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

12. Why isn�t MPNotify.exe launching? The usual suspects: Reviewed network provider order, Citrix SSON at top Checked Gina chain for login Need more details/clues: End users seeing issue were �Power Users� with more locally installed applications and hence more files on workstation End users seeing issue usually saw issue first login of the work week (systems were usually left up over the weekend) Logging out and back in resolved issue for end users (or rebooting workstation) Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

13. Anti-Virus software (full scan task) was now deemed a likely culprit Workstations were configured for full system dumps using DumpConfigurator.hta (Codeplex) Microsoft and Anti-virus vender were engaged to review system dumps of workstations taken during event Anti-virus vendor found nothing conclusive in memory dumps Microsoft analysis showed issue related to a memory management issue in Windows XP and winlogon Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

14. Key takeaways from case: Issue only occurs on Windows XP workstations (Windows Vista & Windows 7 ok) Microsoft stated no fix for memory management issue will be released due to the current support status for Windows XP and the issue is resolved in Windows Vista and Windows 7 Mitigation: Prior to end user login, run empty.exe and pass it winlogon.exe to clear free memory working set: empty.exe winlogon.exe Anti-Virus software didn�t allow process to be run after scheduled scan, so using windows scheduled task was implemented by most customers seeing issue Citrix article: http://support.citrix.com/article/CTX124186 Citrix Confidential - Do Not Distribute Case #1 � Intermittent SSON issue with WinXP

15. Case study #2:XenApp 6.0 Citrix Licensing magic trick

16. Problem: Customers were installing a XenApp 6.0 hotfix on their servers with Citrix Licensing installed. Upon hotfix install and server reboot, Citrix licensing errors started popping up on the console (also seen in the event logs as eventid 9018 errors). Environment: Windows 2008 R2 x64 XenApp 6.0 with Citrix Licensing installed Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

17. Who Server administrators and end users connecting to servers via ICA What Citrix licensing popup on console (also seen in event logs as eventid 9018 errors) after XenApp 6.0 hotfix install Where On the XenApp server When Consistent after hotfix install and server rebooted Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

18. Details: Customer was installing XA600W2K8R2X64003 (http://support.citrix.com/article/CTX126211) to resolve server reboot hangs Fix description from readme: Servers running XenApp 6 can become unresponsive while shutting down. The issue occurs when the picadm.sys driver encounters certain error conditions that prevent it from shutting down gracefully. Upon reboot, it was found Citrix Licensing was no longer present on the server causing the issue (only the service was gone, not the binaries). Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

19. Troubleshooting process: Since customer is installing a hotfix, we need the hotfix install logs XenApp 6.0 hotfixes write logs to %TEMP% folder by default unless verbose logging enabled Verbose logging preferred and we had customer install hotfix via cmd line: msiexec /update C:\XA600W2K8R2X64003.msp /norestart /passive /l*v C:\HFX003.log We also used Windows Sysinternals tool Process Monitor tool to collect trace log during hotfix install Issue was reproduced internally at Citrix Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

20. Data analysis from Process Monitor: We see msiexec deleting the Citrix Licensing performance keys: MsiExec.exe 1896 RegDeleteKey HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance SUCCESS MsiExec.exe 1896 RegDeleteKey HKLM\System\CurrentControlSet\services\Citrix Licensing SUCCESS Then msiexec recreates them: MsiExec.exe 1896 RegCreateKey HKLM\System\CurrentControlSet\Services\Citrix Licensing\Performance REPARSE Desired Access: Set Value MsiExec.exe 1896 RegCreateKey HKLM\System\CurrentControlSet\Services\Citrix Licensing\Performance NAME NOT FOUND Desired Access: Set Value MsiExec.exe 1896 RegCreateKey HKLM\System\CurrentControlSet\services\Citrix Licensing SUCCESS Desired Access: Maximum Allowed, Granted Access: None MsiExec.exe 1896 RegCreateKey HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance SUCCESS Desired Access: Set Value Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

21. Data analysis from Process Monitor: You see values being created under the performance key: MsiExec.exe 1896 RegSetValue HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance\Library SUCCESS Type: REG_SZ, Length: 28, Data: mflicperf.dll MsiExec.exe 1896 RegSetValue HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance\Open SUCCESS Type: REG_SZ, Length: 30, Data: MFLicPerf_Open MsiExec.exe 1896 RegSetValue HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance\Close SUCCESS Type: REG_SZ, Length: 32, Data: MFLicPerf_Close MsiExec.exe 1896 RegSetValue HKLM\System\CurrentControlSet\services\Citrix Licensing\Performance\Collect SUCCESS Type: REG_SZ, Length: 36, Data: MFLicPerf_Collect But never any values under the root Citrix Licensing key, hence the failure of the service after the reboot. Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

22. Key takeaways from case: The root cause came from an issue with the registration of the mflicperf.dll file during the hotfix install, the issue would also occur if you performed a repair on the base XenApp 6 product The core issue and mitigations were documented in http://support.citrix.com/article/CTX126713 The issue was corrected with in XA600W2K8R2X64002 The hotfix should be installed prior to other hotfixes if possible Citrix Confidential - Do Not Distribute Case #2 � XenApp 6.0 Citrix Licensing magic trick

23. Case study #3:XenApp 6.0 hotfix install issue

24. Problem: Customer was installing XenApp hotfix XA600W2K8R2X64002 via the interactive method (double clicking or right clicking and selecting Install) and was getting an error within seconds of launching the hotfix: Environment: Windows 2008 R2 x64 XenApp 6.0 with Citrix Licensing installed Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

25. Who Server administrators installing hotfix XA600W2K8R2X64002 via interactive method What Error dialog pops up immediately when installing hotfix Where Administrator was logged in to XenApp server via RDP Console session When Consistent error message when installing hotfix Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

26. Troubleshooting process: Since customer is installing a hotfix, we need the hotfix install logs XenApp 6.0 hotfixes write logs to %TEMP% folder by default unless verbose logging enabled Verbose logging preferred and we had customer install hotfix via cmd line: msiexec /update C:\XA600W2K8R2X64002.msp /norestart /passive /l*v C:\HFX002.log We also used Windows Sysinternals tool Process Monitor tool to collect trace log during hotfix install Have customer test via automated install (using msiexec) and install via console session (non RDP) Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

27. Data analysis: Automated hotfix install works fine using \unattended or \quiet msiexec switches Session type doesn�t matter (e.g. RDP, physical console) Review on windows installer log showed that customer was logging in via domain account (that had administrative rights) Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

28. Data analysis: Review of windows showed that UAC is enabled (user account control at highest setting � �Always Notify�): MSI (c) (50:84) [15:52:01:043]: Doing action: ErrorUACEnabledAndHotfix Action 15:52:01: ErrorUACEnabledAndHotfix. Action start 15:52:01: ErrorUACEnabledAndHotfix. MSI (c) (50:64) [15:52:01:048]: Note: 1: 2262 2: Error 3: -2147287038 DEBUG: Error 2869:� The dialog ErrorDialog has the error style bit set, but is not an error dialog MSI (c) (50:64) [15:52:01:053]: Font created.� Charset: Req=0, Ret=0, Font: Req=MS Shell Dlg, Ret=MS Shell Dlg Internal Error 2869. ErrorDialog Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

29. Data analysis: From http://msdn.microsoft.com/en-us/library/aa372835(VS.85).aspx we see that error 2869 is: The dialog [2] has the error style bit set, but is not an error dialog. Not too helpful�. Testing showed that with UAC disabled, issue didn�t occur, so reviewing UAC further was necessary Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

30. It was determined by reviewed Microsoft UAC documentation that UAC is definitely involved in this issue (see http://msdn.microsoft.com/en-us/library/aa511445.aspx) When a domain account is logged in with administrative rights and UAC is enabled, the account doesn�t full administrative rights and would need to self-elevate to install a hotfix This combined with our hotfix package not presenting a proper dialog advising the administrator of the need to elevate was core to the issue Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

31. Key takeaways from this case: Mitigations if UAC is enabled on W2K8 R2: Install hotfix using unattended switches Install hotfix using local administrator account Issue is documented in: http://support.citrix.com/articles/CTX127124 Hotfix packaging being reviewed for further enhancements regarding UAC installations Citrix Confidential - Do Not Distribute Case #3 � XenApp 6.0 hotfix install issue

32. Resources and Tools

33. Citrix Knowledge Center articles: CTX124186: Case Study - Ssonsvr Process not Running Intermittently CTX368624 - Troubleshooting Citrix Pass-through Authentication (Single Sign-On) CTX126713 - Citrix Licensing Error: Citrix XenApp cannot contact the license server localhost. CTX111880 - How to enable process audit tracking CTX127124 - Internal dialog error 2869 Citrix Confidential - Do Not Distribute Resources discussed

34. Hotfixes: Hotfix XA600W2K8R2X64002 - For Citrix XenApp 6.0 for Windows Server 2008 R2 Hotfix XA600W2K8R2X64003 - For Citrix XenApp 6.0 for Windows Server 2008 R2 LIMITED RELEASE - Hotfix XA600W2K8R2X64009 - For Citrix XenApp 6.0 for Windows Server 2008 R2 Citrix Confidential - Do Not Distribute Resources discussed

35. Citrix Tools: CDFControl Quick Launch Other Tools: Crash Dump Configurator Windows Sysinternals Process Monitor Windows Sysinternals site Live Windows Sysinternals site W2K3 Resource Kit tools (for Empty.exe) Citrix Confidential - Do Not Distribute Resources discussed

36. Open Forum � Questions