Ramon Scott – Lead Escalation Engineer

1. Configuring & Troubleshooting XenDesktop Sites SUM408 May 2013 Ramon Scott – Lead Escalation Engineer

2. Presentation Goals • Provide an Understanding of the Architecture • Instruct on How to Configure • Provide Proven Troubleshooting Methodologies and Resources

3. High-Level XenDesktop Database And Services Architecture

4. Database

5. XenDesktop 5 Database Overview • Supported Databases: • SQL Server 2008 SP1 / 2008R2(including Express) • Database Schema • Full Relational Schema • Tables, Views, Stored Procedures • Single Database (for core product) • Multiple SQL ‘Schemas’ in Database • ‘Schemas’ map onto Windows services running on Broker Broker Database Broker

6. Setup Process Single Admin Separate Admins XD Admin Broker Broker XD Console XD Console 1. Schema 1. Schema XD Admin “Export” (SQL script) 2. Schema 4. Verify 3. Verify 2. Schema XD Admin credentials used Database Database SQL Server Console 3. Schema SQL Admin SQL Admin credentials used

7. Database Access Security Access Model • Network Service Account “NT AUTHORITY\NETWORK SERVICE” • Computer Account “DOMAIN\MACHINE$” SQL Login per Broker Restricted permission set • Brokers do not have rights to change schema Controller Broker Service Controller Database Controller Broker Service Database Controller Database

8. Database High-Availability • Broker is critically dependant on Database • Existing connections not impacted • Creating new connections and reconnecting to desktops impacted • Database Failure = Broker Failure • Supported Database H/A Options: (expected popularity order) • SQL Mirror • Virtual Machine H/A • SQL Cluster Citrix Confidential - Do Not Distribute

9. Database Schema Roles and Permissions

10. Health Checks: XDDBDiag • Provided consistency data check on the data • Provides connectivity verification It also provides the following: • Virtual Desktop Agent Information • Hypervisor Connections Information • Policy Information • Controller Information • Desktop Groups Information • SQL Information • Current Connections / Connection Log XDDBDIAG

11. Services

12. XenDesktop 5 Services Architecture Desktop Studio Desktop Director PowerShell PowerShell WCF [80] WCF [80] WinRM 2.0 [5985/5986] Controller Machine Creation Service Host Service Machine Identity Service AD Identity Service Broker Service Configuration Service Virtual Desktop Agent (VDA) Windows Communication Foundation (WCF) SQL Server Machine Creation Services Broker Service Infrastructure Services

13. Service Status

14. Machine Creation

15. Profile PvD PvD Profile Profile Profile PvD PvD PvD Profile PvD Profile Desktop Catalog models App App Image • Existing • Dedicated • Pooled • Pooled with personal vDisk • Streamed • Streamed with personal vDisk Profile Profile Base Image with Apps Image App App Streamed Base Image with Apps Base Image Streamed Base Image Base Image with Apps Profile Profile Image App App Profile Profile *Image Streamed from Citrix Provisioning Server (PVS) *Image created with Machine Creation Services (MCS) *Image created outside of XenDesktop

16. Desktop Catalog models * Behaves like pooled-static

17. MCS – ID Disk, Difference Disk, Base VM This is what the user sees as Drive C:\ This is hidden from the users view Windows 7 Master VHD Chain VHD Chain VHD Chain Diff Disk Diff Disk Diff Disk ID Disk ID Disk ID Disk Virtual Desktop 2 Virtual Desktop 1 Virtual Desktop x Storage Subsystem

18. MCS with PvD – ID Disk, Diff Disk, Base VM, PVDisk Windows 7 Master VHD Chain Diff Disk ID Disk Virtual Desktop 1 Personal vDisk • This part is hidden from user • Merged with the Diff Disk • Seen by user as Drive C:\ • E.g. Installed apps • Seen by the user as Drive P:\ • USERDATA e.g. My Documents • Free space is the split allocation • PVDisk auto-created during catalog creation by copying PvD template from Base VM • 10GB by default with 50 / 50 split for App Data / User Data

19. PVS – Streamed vDisk, Cache, Base VM This is what the user sees as Drive C:\ Visible file on another disk, typically D:\ Windows 7 Master PVS Stream PVSStream PVS Stream Streamed vDisk Streamed vDisk Streamed vDisk Write Cache Write Cache Write Cache Virtual Desktop 2 Virtual Desktop 1 Virtual Desktop x Storage Subsystem

20. PVS with PvD–Streamed vDisk, Cache, Base VM, PvDisk Windows 7 Master PVS Stream Streamed vDisk Write Cache Virtual Desktop 1 Personal vDisk • This part is hidden from user • Seen by user as Drive C:\ • E.g. Installed apps • Seen by the user as Drive P:\ • USERDATA e.g. My Documents • Free space is the split allocation • PvDisk auto-created during catalog creation by copying PvD template from Base VM • 10GB by default with 50 / 50 split for App Data / User Data

21. Where are some of the common Issue ? • Hypervisor communication • Domain permissions • Previously failed attempts still present in database • Host Connection configured with incorrect storage • Naming convention on the host

22. What logs do we need for this issue ? Desktop Studio PoSH WCF [80] Broker Machine Creation Service Host Service Machine Identity Service AD Identity Service Broker Service Configuration Service SQL Server Machine Creation Services Broker Service Infrastructure Services

23. Troubleshooting Methodology • Understand issue history • Verify configuration, error logs and alerts • Gather and review log data of issues • Compare data to working environment

24. Enabling Log from the Command Line Service –LogFile <Location> Citrix.MachineCreation.SdkWcfEndpoint.exe -Logfile“c:\xdlogs\MCS-PVSvm.log” Citrix.ADIdentity.SdkWcfEndpoint.exe -LogFilec:\xdlogs\AD.log Citrix.MachineIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\mi.log

25. Case Study 1 Machine Creation Services

26. Case Study 1: MCS Fails after wizard Background: New Deployment Latest Hotfixes Full Administrator account used Worked before they rebuilt environment Case Study Walk Through

27. Log Analysis: Desktop Studio Logs Case Study 1: Machine Creation Service fail after wizard 24/04/13 02:37:10.7603 : DesktopStudio: [6] Script SetActionMetaData(402): [RES] Value: Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog. Search Terms: [Time of Issue] Fail | Error | Exception | Denied

28. Log Analysis: Machine Creation Service Logs Failed to copy disk. Reason : SR_HAS_NO_PBDS ManagedMachineException: Failed to copy disk. Reason : SR_HAS_NO_PBDS Concluding job d5ea54c6-b7f1-4d45-ac08-2e2abae39e48 with state DiskConsolidationFailed. WorkflowAddMetadata(, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.) Case Study 1: Machine Creation Service fail after wizard Search Terms: [Time of Issue] Fail | Error | Exception | Denied

29. Root Cause analysis: Misconfiguration • Failed to copy disk Reason : SR_HAS_NO_PBDS • Hypervisor Connection’s did not include correct storage for the Master Image • Target device disk could not be copied due to this Hypervisor -Storage misconfiguration *Definitions: SR - Storage Repositories PBD - Physical Block Devices

30. VDA Startup and Registration

31. VDA Registration Registered VDA Controller WCF Desktop Service Broker Service Database VDA DDC LDAP ListofDDC Active Directory Controller

32. XDPing Log Basic Checks Logs: Workstation Agent Logs Broker Logs Network Trace Troubleshooting VDA Startup and Registration Controller VDA Desktop Service Broker Service 1011011010 SSL 1011011010 SSL 101101

33. XDPING • Can be run on both the DDC and VDA • Used to collect data related to basic components • Will verify if the components are working correctly • Verify Domain Membership • Network Interfaces • WCF Endpoints • Services • DNS lookup • Time difference between machine and Domain Controller XDPING

34. Basic Checks • Check the Network: Ping , Telnet and NetStat, Firewall • Ensure Services started without errors • Listening on the correct port • Check time • Check configured list of DDCs in registry

35. Case Study 2 Startup and Registration

36. Case Study 2: New Catalog Fail to Register Background: Locked down environment Special configuration needed to manually enable needed services Worked in the Proof of Conference Lab but failed in production Case Study Walk Through

37. Log Analysis: Workstation Agent Service Logs Failed to register with http://FTLRSCOTT2RHONE.lab.net:80/Citrix/CdsController/IRegistrar. WCF Fault with detail CallbackCommunicationError, message 'Fail worker callback using SPN host/RS2-SynPool01.lab.net and IP address 10.19.196.945' Register FAILURE: HighAvailabilityActive = False, InHighAvailabilityMode = False, _firstRegistrationAttemptTime = 05/18/2013 13:54:31, HighAvailabilityRegistrationTimout = 00:05:00 Message following Error pattern Could not register with any controllers. Waiting to try again in 9407 ms Case Study 2: New Catalog Fail to Register Search Terms: [Time of Issue] Fail | Error | Exception | Denied

38. Log Analysis: Broker Service Logs Broker:TestWorkerComms failed for worker S-1-5-21-1123877020-465626563-3648135752-1267 caught exception: System.ServiceModel.Security.SecurityNegotiationException: The caller was not authenticated by the service. ---> System.ServiceModel.FaultException: The request for security token could not be satisfied because authentication failed. Case Study 2: New Catalog Fail to Register Search Terms: [Time of Issue] Fail | Error | Exception | Denied

39. Root Cause analysis: Misconfiguration • The DDC was not authorized the initiate a connection to the VDA • “Access To Compute From The Network” Computer Policy did not have an entry for the Controlled and the default everyone was removed in production. Resolution: Customer added explicit entry to a Group that included all the Brokers as members

40. Troubleshooting and Support • PVD maintains logs in the base of the volume attached to the VM • (alongside the VHD containing the PVD user-installed applications) • These logs contain a wealth of information that should be captured and provided to support/engineering if you experience problems • Most frequently seen PVD support cases … • Failure of PVD to start virtualization (PVD can’t locate volume/VHD, etc.. …) • Customers trying to install unsupported apps • Customers trying to move PVDs between VMs

41. Troubleshooting and Support (cont’d) • Desktop Director has helpdesk-facing PVD metrics and support • % of application area in use / total size • % of user profile area in use / total size • PVD reset • PVD reset allows the helpdesk to reset the application area while leaving the user’s data intact • Aka “revert to factory default” • Useful to reset PVDs that become wedged due to users installing broken applications

42. VDA Launch

43. VDA Launch Preparing New Session Idle Controller #1 VDA WCF Broker Service Desktop Service ICA Service SQL VDA Policy Settings Broker signals worker to Prepare for a Session Launch Request XML broker queries DB for a ready worker User Clicks to launch session WI DDC

44. VDA Launch (cont’d) Active Connected Controller #1 VDA WCF Broker Service Desktop Service ICA Service SQL VDA Policy Settings Request to Validate Ticket sent Controller Validates Ticket Validates License Policies Ticket is ValidAuthNTicket Portica gets License ICA file is sent to Endpoint Work State: Connected Work State: Active WI DDC

45. What Happened ?

46. Troubleshooting VDA Launch • Event Logs (Web Interface, Controller, Storefront) • Desktop Studio • Broker Logs • Workstation Agent • Portica Logs • Network Packet tracing

47. Case Study 3 VDA Launch

48. Case Study 3: Launch Failure 1030 Background: They recently converted all images to a Citrix PVS image The original image worked All streamed images including the golden image failed to launch Case Study Walk Through

49. Search: Prepare

50. Troubleshooting :VDA Launch • Search Strings: Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect