500 likes | 1.05k Views
XenApp 6 Case Studies and Troubleshooting. Rick Berry, Escalation Engineer Mark Callahan, Escalation Engineer May 24 th , 2011. Agenda. Case study for UPM issue on XenApp 6 Case study on XenApp 6 filtered policy issue Questions and wrap-up. Case study for UPM issue on XenApp 6.
E N D
XenApp 6 Case Studies and Troubleshooting Rick Berry, Escalation Engineer Mark Callahan, Escalation Engineer May 24th, 2011
Agenda • Case study for UPM issue on XenApp 6 • Case study on XenApp 6 filtered policy issue • Questions and wrap-up
Problem Definition • Customer was experiencing hung sessions at logon • Some users could log in, others could not
Symptoms • “Black Hole” • User Profile Manager process still running • Logged in users would eventually be affected Citrix Confidential - Do Not Distribute
Functional Overview - Logon Local Windows Devices XenApp Servers XenDesktop Streamed/Delivered Desktops Profiles stored via File Share My Settings File Servers Profile management Service
Active Directory Functional Details File Server User Logon \profiles\UserName\ Intelligent Sync Collect Configuration XenApp Server [User Logon Event Location] Folder Redirection via Network File Server \HKLM\Software\Policies\Citrix\UserProfileManager. \\server\UserHome\ Profile management Service
Troubleshooting Methodology • Complete System Dump • PerfMon • User Profile Manager Logs Citrix Confidential - Do Not Distribute
Troubleshooting MethodologyComplete System Memory Dump • Examine Kernel memory • Examine Winlogon process Citrix Confidential - Do Not Distribute
Troubleshooting MethodologyPerformance Monitor • Performance Monitor – monitor User Profile Manager and Winlogon threads PROBLEM NORMAL
Troubleshooting MethodologyUser Profile Manager Logs [PID];WaitUntilChangeJournalIsProcessed: Waiting to finish change journal processing of partition: C Ah Ha! A suspicious log entry!
Troubleshooting Methodology • NTFS change journal was showing an increased size of the identification field. SCREENSHOT
Resolution • Based on the data learned from the NTFS change journal examination, a code change was made to handle changes to the size of the Update Sequence Number record and a hotfix was developed.
Resources – Citrix Profile Manager Citrix Profile Manager Edocs Site Citrix Profile Manager Logon Diagram Citrix Profile Manager Logoff Diagram CTX119791- Profile Management FAQ CTX12559- Citrix Profile Manager Upgrade FAQ CTX124455- How to Capture CDF Startup Traces on UPM 3.0
Resources – Citrix Profile Manager Log Parser for Citrix Profile Management Memory Dump File Not Being Generated on Provisioned Target Microsoft Windows Change Journals
Problem definition • Customer had a new XenApp 6 farm in place • XenApp 6 Citrix policies (both computer and user settings) were being applied via Active Directory Group Policy Objects (GPOs) • Some of the Citrix policy settings were filtered for Access Gateway connections and others were filtered by client IP • When end users connect to the XenApp 6 server from an Access Gateway site, the filtered policy settings were not applying to the session
XenApp 6 Group-based administration XenApp Farm • Manage XenApp servers collectively by grouping servers into worker groups • You can assign published applications and Citrix policies to worker groups • Servers added to worker groups inherit settings Published Application: Notepad.exe Worker Group 1 Citrix Policy: Enable Client Drive Mapping Worker Group 2 Worker Group 3
Applying Citrix Policies to Worker groups • Worker Group is a new filter for applying Citrix policies • Automatic configuration of new XenApp servers by placing them in an existing worker group
Citrix policy creation and administration Create policies as Citrix IMA-based policies using Delivery Services Console (Used if AD does not exist or access is limited) Create policies as Active Directory-based policies using Group Policy Management Console (GPMC) Note: All Citrix policy settings are configurable using either administration method
Citrix policies via the Delivery Services Console • Citrix policies added via the DSC are stored in the datastore • Two types of policies categorized by computer policies and user policies • Can be “filtered” for granular control or “unfiltered” to apply to all servers or users • Policy settings are stored in the servers registry
Filtered versus unfiltered policies • Filtered policy • Applies to specific group of users or servers • Uses a variety of filters (IP, AG, Groups, Client name) • Use case: Disable CDM for the Marketing domain group • Unfiltered policy • Applies to all servers or users • Used when filters or granular control isn’t necessary • Use case: Specifying the license server that all farm servers will use
Citrix policy extension • Allows integration of Citrix policies into the Windows GPO engine • Adds a Citrix node in the Group Policy Management Console and Group Policy Object Editor • Installed with Delivery Services Console • Must be installed on the same machine whereGroup Policy Objects are administered • Can be installed on a standalone machine used for administrative purposes
Citrix policy settings on the server • Computer policies • Enables or disables server settings that were once under the farm and server properties in previous versions • Registry location: • 32-bit components: HKLM\Software\Policies\Citrix • 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix • User policies • Enables or disables specific features for user sessions • Registry location: • 32-bit components: HKLM\Software\Policies\Citrix\<SessionID> • 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix\<SessionID>
GPO processing and precedence OU Group Policy Objects Domain Group Policy Objects Site Group Policy Objects PROCESSING PRECEDENCE Citrix Group Policy Objects Local Policies
Citrix policies general roubleshooting checklist • Identify how the policies are being applied (e.g. Active Directory, DSC, both)? • Are the Citrix policy files present on the server? • What does the group policy results wizard show? • CDF Tracing results (see CTX113199 for modules). • Setup and review Citrix policy debugging logs. • Are the Citrix policy registry settings in place?
Troubleshooting Methodology • Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?
Troubleshooting methodology for the case • Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly? • What does output from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.
Troubleshooting Methodology • Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly? • What does out from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server. • Enable Citrix policy debugging (see CTX128413)
Setting these values to 0xFFFFFFFF writes the debug information to a log file: %SYSTEMROOT%\Temp\CitrixCseEngine.log Setting these values to 0x0000FFFF writes the debug information to a debugger such as DebugView NOTE: The same values have to be written to HKLM\SOFTWARE\Wow6432Node\Citrix\GroupPolicy For more details see CTX128413
Troubleshooting Methodology – Debug logs • Reviewing %SYSTEMROOT%\Temp\CitrixCseEngine.log we need to verify the logged in user User Name = REDGETLAB\rickbeuser1, SID = S-1-5-21-3992822370-2973014269-1922904879-1172, Session ID = 3 Computer Identity - Name = 60426497M1 • Next we search on the display name of our policy so we can get the GUID since the GUID is referenced more in the log Name={52243C73-ED52-4539-B484-02098F5A88F4}, DisplayName=Test Policies, Link=LDAP://OU=RickBe,DC=REDGETLAB,DC=CTX
Troubleshooting Methodology – Debug logs • We know that the Access Gateway filter on the policy was using a wildcard (apply to any Access Gateway site), so for the Access Gateway filter we can search on AGInUse FullArmor.GroupPolicyFramework:And(Citrix.Policy.Templates:AGInUse.isValid, Citrix.Policy.Templates:AGFarm.isValid Citrix.Policy.Templates:WildcardMatch("*" Citrix.Policy.Templates:AGTags.value,"*",true
Troubleshooting Methodology – Registry review • Our session in question was session 3: HKLM\SOFTWARE\Policies\Citrix\3\Events "LastUpdate"="2011-03-27 04:12:12Z“ • Looking at the Evidence key: HKLM\SOFTWARE\Policies\Citrix\3\Evidence “AGFarm”= "AGInUse"=dword:00000000 These are issues!!
Root cause isolation • Reviewing the debug logs and comparing this to the registry entries being made allowed us to narrow down the issue to how the policy filters were being evaluated • Through our analysis it was determined that there was an issue with the filter expression logic when the Access Gateway filter was being used
Resolution • The investigation into this issue resulted in code change for the Delivery Services Console which was tested successfully by the customer • This code change is currently being packaged into a hotfix for the Delivery Services Console
Resources – Citrix Policy Architecture CTX125152 - Citrix Group Policy Engine Facts in XenApp 6 CTX127612 - How Policies are Applied when an ICA Session Connects to XenApp 6.0 CTX127611 - How Citrix IMA Policies for XenApp 6.0 Fit in to Microsofts GPO Processing and Precedence Model CTX124241 - Technical Guide for Upgrading/Migrating to XenApp 6 Citrix Blog Site - XenApp 6 Policies Deep Dive
Resources – Citrix Policy troubleshooting CTX128413 - XenApp 6 and XenDesktop 5 Group Policy Tracing CTX111961 - CDFControl Tool CTX113199 - IMA Modules to Select When Obtaining a CDF Trace for a Policy Problem
Session surveys are available online at www.citrixsummit.com starting Thursday, May 26 Provide your feedback and pick up a complimentary gift at the registration desk Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account Before you leave…