430 likes | 2.37k Views
Debugging Citrix XenDesktop & XenApp. Jamie Baker Manager, US Escalation Team Kapil Ramlal Sr. Software Maintenance Engineer May, 2010. Agenda. Troubleshooting Theory Brief Architectural Review XenApp and XenDesktop Common Components Common Problem Types
E N D
Debugging Citrix XenDesktop &XenApp Jamie Baker Manager, US Escalation Team KapilRamlal Sr. Software Maintenance Engineer May, 2010
Agenda • Troubleshooting Theory • Brief Architectural Review • XenApp and XenDesktop • Common Components • Common Problem Types • Debugging Tools and Techniques Citrix Confidential - Do Not Distribute
Troubleshooting Theory Citrix Confidential - Do Not Distribute
Good Facts of Life Bad Compare them Both
Troubleshooting • Good • Know how the system is supposed to work • Determine how to collect the data you need • Bad • Collect data from the bad system • Compare the Good system with the Bad
Putting It All Together Find “best” virtual desktop Acquire license and determine settings Authenticate Start VM Desktop Delivery Controller SAN PVS Register PXE-boot VM and stream OS Connect using ICA Log in XenServer Virtual Machines Full range of authentication methods supported through web interface technology Apply profile Deliver apps Full support for SmartAccess and ICA session policies Active Directory with roaming profiles XenApp Citrix Confidential - Do Not Distribute
The Desktop Delivery Controller • Desktop Delivery Controller (Broker) Farm • Uses core XA technology • IMA, licensing, WI, ... • Delivers and controls access to virtual desktops • VDA technology agnostic • User authentication / single sign on • VM power management • ICA policy decision Licensing Web Service WI DDC Servers VM Host (XenServer, Hyper-V, VMware) User LDAP ICA WebService LDAP AD VDAs (VMs, Blade PCs) Citrix Confidential - Do Not Distribute
The Desktop Delivery Controller • Worker / controller design • Few controllers / many VDAs • DDC scales to 1000s of VDAs • DDC not in connection path • Dependent on Active Directory • User authentication • Communication security • Controller discovery Licensing Web Service WI DDC Servers VM Host (XenServer, Hyper-V, VMware) User LDAP ICA WebService LDAP AD VDAs (VMs, Blade PCs)
The Desktop Delivery Controller Core Architecture DDC Server Web Svc DomainController VM Host HTTP Pool Manager XML Service LDAP 80, 443 WI VDA(VM or blade PC) DCOM,WCF Workstation(VM or blade PC) Workstation(VM or blade PC) Controller Service MFCOM / IMAProxy WCF 2514, 8000, DCOM Desktop Service PortICA Svc AMC 8080 IMA IMA CGP Svc PortICA Drivers 2513 2512 PSC Licensing ADO IMA DataStore LicenseServer DDCServers
Virtual Desktop Agent • “Virtual Desktop Agent” • Collection of services, drivers, ... • “PortICA”: ICA connectivity • “Desktop Service”: web service interface communicating with DDCs • How does it relate to XenApp? • Majority of ICA code is shared • Does not use Terminal Services • Major changes in: WinLogon integration, session management, USB support Citrix Confidential - Do Not Distribute
XenDesktop and Active Directory • XenDesktop relies on AD for • Authentication of end users and admins • Mutual authentication of DDCs and VDAs • Encryption of network traffic • Discovery and authorization of DDCs by VDAs • Each DDC farm can have an OU • Only used for purpose 4. • May (but need not) contain computer accounts • Need not be configured at root OU level • Alternative Discovery Method • Configure DDC identity in VDA registry (see CTX118976) Citrix Confidential - Do Not Distribute
Common Components The same ICA client can be used to access both XenDesktop and XenApp Citrix Web Interface can also be used for both Active Directory XML IMA DDC/ZDC (Although roles are a bit different) Citrix Confidential - Do Not Distribute
Common Problem Types XenDesktop XenApp VDA Registration Failure VDA Connectivity / Reconnecting failure Hyper visor Issues Service hangs (DDC/Poolmgr) .NET Global Assembly Cache Exceptions CPU Memory • Data Store • Server Connectivity / Reconnecting failure • Load Balancing • Black Hole • Service hangs (IMA/ZDC) • CPU • Memory Citrix Confidential - Do Not Distribute
Debugging VDA Registration Debugging Use XDPing to check for time sync issues Check port connectivity (Telnet, XDPing, CtxPrtChk) Check resultant set of policy for AD inconsistencies Check Event Viewer Capture remote CDF trace using CDFControl (CTX111961) DEMO: Capturing a remote CDF trace with CDFControl Citrix Confidential - Do Not Distribute
Debugging VDA Connectivity Leverage XDPing (CTX123278) to rule out common causes Capture remote CDF trace Check for display driver switching issues Ensure no WDDM display driver is being used Try reducing screen resolution and color depth Citrix Confidential - Do Not Distribute
The Global Assembly Cache (GAC) • Stores assemblies specifically designated to be shared by several applications • Citrix XenDesktop VDA and DDC services use .NET, and rely on the GAC for shared assemblies Citrix Confidential - Do Not Distribute
The Global Assembly Cache (GAC) Debugging When? • .NET components fail to start or encounters an exception, such as: How? • Enable Fusion Logging (Registry Based Setting) The GAC Utility • Comes with Microsoft Visual Studio IDE • Can be used to reinstall GAC components FileLoadException FileNotFoundException BadImageFormatException Citrix Confidential - Do Not Distribute
The XenApp “Black Hole” Problem • Connections are routed to the Least Loaded Server in the Farm • An underlying problems exists on the Least Loaded server • Least Loaded Server still responds to IMA heartbeat pings • ZDC gets the pings and routes to the broken server, causing a "Black Hole” effect Citrix Confidential - Do Not Distribute
The XenApp “Black Hole” Problem • What are the 3 most important XenApp servers, from a connecting user's standpoint? • How to quickly validate the health of these 3 servers? • How to monitor Farm Health using MedEvac (CTX119899) • Runs tests against: • Terminal Services • RPC • XML • Least Loaded Server Citrix Confidential - Do Not Distribute
What about XenDesktop? DEMO: Xnapshot (Sneak Peek) Citrix Confidential - Do Not Distribute
A typical customer support engineer A reasonable customer I got a problem I need information A Here it is When can you fix it? I need information B I need information X “He does not know what he is doing”
A typical customer support engineer A capable customer I got a problem Have you changed anything ? Hmmm. Not really “I better not tell him what I did”
Xnapshot tool makes it easy Take-it-easy support engineer Next generation customer “Smart – he knows what he is doing” We got all your information and we know what were changed.
Capturing Post-Mortem Memory Dumps User and Kernel Space Windows uses 2 levels of protection to restrict access to areas of memory System memory is divided into 2 spaces: • User Space • Kernel Space Applications run in User Space Operating System code and Drivers run in Kernel Space
Capturing Post-Mortem Memory Dumps User Dump Capture Setup a default post-mortem debugger to catch crashing applications How to Set the NT Symbolic Debugger as a Default Windows Postmortem Debugger (CTX105888) Check for the managed debugger under: • HKLM\Software\Microsoft\.NETFramework • Value: DbgManagedDebugger
Capturing Post-Mortem Memory Dumps System Dump Capture Configure in Startup and Recovery settings Ensure pagefile can store dump See MS Article cc976050 for registry settings
Capturing Post-Mortem Memory Dumps System Dump Capture Dedicated Dump Drive (Windows 7+) • Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl Name: DedicatedDumpFile Type: REG_SZ Value: Dump path, such as D:\dedicateddumpfile.sys How to Recover Windows Kernel Level Dump Files from Provisioned Target CTX123642 Citrix Confidential - Do Not Distribute
Capturing Post-Mortem Memory Dumps System Dump Capture Tools The NMI switch (MS KB927069) Keyboard Initiated (MS FF545499) SystemDump 3.1 for 32-bit and 64-bit platforms (CTX111072) Citrix Confidential - Do Not Distribute
To Hook or not to Hook • Try capturing a CDF trace on the particular component for deeper insight • Example: MF_Hook_SCardHook • Try excluding the application from the hook to see how it behaves • If it runs fine based on testing, then leave it excluded CTX107825 – HOW TO DISABLE CITRIX HOOKS ON A PER APP BASIS Citrix Confidential - Do Not Distribute
XenDesktop & XenApp Core Services XenDesktop XenApp DDC: Pool Manager Service (CdsPoolMgr.exe) XML Service IMA Desktop Delivery Service (CDSController.exe) VDA: Workstaton Agent PortICA CtxSvcHost XML ZDC: • IMA • XML Service MEMBER XA SERVER: • IMA • XML Citrix Confidential - Do Not Distribute
The TechEdge survey will be emailed out to end-user customers If you complete the survey, you will be entered to win a $250 Amazon gift card. The winner will be announced June 1st. View TechEdge videos & PPTs on the Knowledge Center by Monday, May17th http://support.citrix.com/techedge2010 TechEdge Survey, Video Postings & PPTs