NxPrep: Installing Virtual Machine Images on Engine

1. XenClient Enterprise 4.5 NxPrep: Installing Virtual Machine Images on Engine

2. Table of Contents

3. NxPrep Overview

4. NxPrep: Installing Deployed Virtual Machines This is what happens when a new version of a Virtual Machine (VM) image is deployed to a computer from the Synchronizer. The VM update is downloaded from Synchronizer. The new version of the VM image is installed on Engine. After installation, the new VM image version is ready to run. The “Installing Update” process is called NxPrep. During NxPrep, the Engine prepares a new version of a VM image to run on the Engine.

5. NxPrepVirtual MachineBoots • During NxPrep, Engine will boot the new VM version in the background. • Windows XP VMs: Three background boots. • Other VM types: Two background boots. • The term “background boot” means there is no display attached and the user cannot see what is happening (and is less able to interfere with the process). • What happens in the VM during the background boots: • Windows discovers new XCE virtual devices, which are different from Hyper-V virtual devices, and is given a chance to adjust to the new device set. • The XCE PV drivers package is installed. • The user and local disks are initialized (if necessary). • The Microsoft iSCSI initiator service is configured. This is how the optical drive is shared between the Engine and VMs. • Miscellaneous Windows configuration tasks. • The NxPrepExtend script is run (if it exists). • For a more detailed account of what happens during the background boots, refer to this log file in a running VM: • C:\NxTop\Logs\nxprep.log

6. Windows Domain Membership • NxPrep Configures Domain Membership • This is only done when a VM is initially installed on Engine. • For VM updates, the previous configuration is carried forward. • For a restored VM, domain configuration is included in the backup. • Synchronizer Coordination • Engine never communicates with Active Directory (AD) directly. • Synchronizer creates a computer account in the domain for the VM. • Engine injects computer account settings into the VM during NxPrep. • AD Required for First VM Boot • Windows (in the VM) needs to contact AD to: • Establish a trust relationship with the domain controller. • Authenticate domain login credentials for the user. • Domain logins will fail if the VM cannot access AD. • Windows will cache credentials after the first successful login. • This is standard Windows behavior. • AD Issues can Break NxPrep • Synchronizer must be able to create a computer account for the VM. • If AD integration is misconfigured in Synchronizer, NxPrep will fail. • Usually caused by an incorrect password. • Can also be caused by network problems.

7. PV Drivers Installation • The PV Drivers are installed into the VM during NxPrep. The version of the PV Drivers will match either the Synchronizer or Engine version as outlined below. • XCE Engine Version 4.1 (and earlier) • NxPrep will install PV Drivers from a kit that was injected into the VM image when the VM image version was published in Synchronizer. • Therefore, the installed VM will have PV drivers that match the version of the Synchronizer when the deployed version of the VM image was published. • This might be different from the version of the Synchronizer that exists when the VM image is being installed on Engine. • XCE Engine Version 4.5 (and later) • The Engine will inject the PV Drivers installation kit into the VM before the first NxPrep boot. • Therefore, the installed VM will have PV drivers that match the Engine version. • Any PV Drivers installation kit previously injected into the image by Synchronizer is ignored.

8. NxPrepExtend • C:\NxPrepExtend.cmd • If this script exists in the VM image, it is executed within the VM near the end of the NxPrep process. This feature can be used to customize an installed VM. Since the script is run during NxPrep, the results of the script will persist across snapback. • Best Practices • Keep NxPrepExtend scripts short and simple. • All NxPrepExtend commands must complete with no user action. • When modifying the registry with reg.exe, use the “/f” switch. • Limitations • There is no networking available when the NxPrepExtend script is run. • The user and local disks (U: and L: drives) might not be available. • NxPrep can fail if a NxPrepExtend command hangs or returns an error. • NxPrepExtend Results • A new or modified NxPrepExtend script would not take effect in deployed VMs until: • The VM image is republished in Synchronizer. • The new VM image version is deployed to the computer.

9. NxPrepExtend Example: Disable Reboot Warning This warning appears when a user logs into a VM for the first time. It alerts the user to restart the VM so the user profile can be relocated from the system disk to the user disk. Until this is done, user data will not be included in VM backups uploaded from Engine to Synchronizer. For owned or assigned computers, it is strongly recommended to keep this warning intact. But for unowned or kiosk computers where user data is not being backed up to Synchronizer, the warning might be disabled. To disable the warning, create a file “C:\NxPrepExtend.cmd” in the VM image in Synchronizer. This script will be executed at the end of NxPrep. The command to include in the script is: reg delete HKLM\Software\Microsoft\Windows\CurrentVersion\Run /v "NxTop Notify" /f This change would not go into effect until the VM image is republished and the new version installed on Engine.

10. NxPrep Timeout and Retry • NxPrep Timeout • Engine terminates NxPrep after 90 minutes. • NxPrep should never take this long to complete. • If it takes this long, the NxPrep VM is probably blocked. • NxPrep Retry • For most NxPrep failures, Engine will retry NxPrep once. • If the second NxPrep also fails, Engine will give up. • The VM status will go to “Error: Contact Administrator”. • NxPrep Retry Reset • Engine will reset the NxPrep retry count when: • The computer is restarted. • The VM image is redeployed to the computer. • A new VM version is deployed to the computer. • The Engine management endpoint service (mepd) is restarted.

11. NxPrep Memory The Engine requires memory to run the NxPrep VMs. Memory used for NxPrep depends on the OS type. This VM status means Engine cannot install a VM update because there is insufficient memory for the NxPrep VMs. To free up memory, shutdown any running VM. This includes the Dock (previously known as the Connect VM) if it is enabled.

12. NxPrep, Quick Launch, and Smart Memory • Quick Launch and Smart Memory affect memory allocation for VMs. • These features might not leave sufficient memory for NxPrep. • For Quick Launch this should only happen for a 64-bit VM and only if the first NxPrep attempt fails. • For Quick Launch, the solution is to reboot the computer. NxPrep should begin after the computer restarts. The user might have to wait until NxPrep completes before the VM can be accessed. • For Smart Memory, the solution is to shutdown a VM to free up memory. Quick Launch is configured in the Synchronizer Engine policy. Smart Memory is configured in the Engine control panel.

13. Disk Layering, VHD Files, and Snapback

14. VHD Layering for Running Virtual Machines • The system disk for a running VM is backed by a chain of VHD files. • The VHD files can be divided into logical layers as illustrated in the following diagram. • Some layers come from Synchronizer, others are created on the Engine. Runtime Layer VM NxPrep Layer Created on the Engine. Captures all changes made to the system disk while the VM runs. Reset by snapback. Publish Layer Created on the Engine. Contains the results of the NxPrep operation. This layer sets the baseline for snapback. System Disk (C: Drive) Downloaded from Synchronizer. Contains the results of the publish operation when a VM image version is published. System Layer Downloaded from Synchronizer. Contains the VM image as authored and configured within the Synchronizer. Might include multiple VHD files.

15. System and Publish Layers • The system and publish layers are downloaded from Synchronizer. • System Layer • Contains the VM image as authored and configured in Synchronizer. • Will often include multiple VHD files (one per VM image version). • Publish Layer • Contains the results of the publish operation in Synchronizer. • Always a single VHD file. system-2-diff.vhd Contains the results of the version 2 publish operation. VHD VHD VHD Publish Layer System Disk (C: Drive) Before NxPrep system-2.vhd Contains the differences between versions 1 and 2. System Layer system-1.vhd Base VHD file for version 1 of the VM image.

16. NxPrep Layer • During the NxPrep process: • Engine creates the NxPrep layer linked to the Publish layer. • All system disk changes during NxPrep are captured in this layer. • Publish and System layers are read-only during NxPrep. • NxPrep VMs run in the background with no console access. • Unless the console is enabled for troubleshooting. Write NxPrep Layer VM Read Publish Layer Read Read System Disk (C: Drive) System Layer

17. Runtime Layer • When the VM is started after NxPrep: • Engine creates a new Runtime layer linked to the NxPrep layer. • All system disk changes are captured in the Runtime layer. • NxPrep, Publish, and System layers are read-only. • The VM starts with console access available to the user. Runtime Layer Write NxPrep Layer Read VM Publish Layer Read Read System Disk (C: Drive) Read System Layer

18. Snapback • Shared VMs are snapped back every time they are restarted. • Custom and Local VMs do not participate in snapback. • Before snapback: • Runtime layer contains all changes made to the system disk by the VM. • NxPrep layer is unchanged since the VM was installed. • Publish and System layers are consistent with the Synchronizer. • After snapback: • Runtime layer is discarded and replaced with a new empty VHD file. • NxPrep, Publish, and System layers remain unchanged. • The system disk is “snapped back” to the results of NxPrep. Runtime Layer NxPrep Layer The Runtime layer might grow large while the VM runs. It is reset during the snapback process. This “snaps back” the system disk to a state consistent with the results of NxPrep. VM Publish Layer System Disk (C: Drive) These layers are read-only when the VM runs in user mode and cannot be modified by the VM. System Layer

19. Concurrent NxPrep • NxPrep can run in the background while the VM is running. • Two copies of the VM are running at the same time. • Only possible if sufficient memory is available. VM version 2 (NxPrep) VM version 1 (Running) Runtime Read/Write for Version 1 Read/Write for Version 2 Version 2 NxPrep Version 1 NxPrep Version 1 Publish Version 2 Publish Read-Only Read-Only The VM is running while an update is being installed. Version 2 System Two versions of the VM are running at the same time, starting from different locations in the VHD tree. There is no conflict between running versions because all common VHD files are read-only. Version 1 System

20. NxPrep Troubleshooting

21. NxPrep Failure Status • Error: Contact Administrator • NxPrep started but failed part-way through. • Usually caused by problems within the VM image. • Install Update When Server Available • NxPrep failed due to a problem with the Synchronizer. • Most often caused by AD integration problems.

22. NxPrep Failures Caused By Virtual Machine Image Issues • Application or Service Conflicts • Some applications or services can conflict with NxPrep. • Even if they are installed and working properly in the Hyper-V VM. • Potentially problematic applications or services include: • Antivirus, Encryption, USB-over-IP, or VPN software. • Software that creates devices or installs drivers. • VMWare or XenServer tools from a V2V conversion. • Windows Inconsistencies • NxPrep can fail due to Windows inconsistencies such as: • Windows updates that were not completely applied. • A Windows configuration change that requires a restart to complete. • An application or service in a partially-installed state. • To avoid these issues, verify Windows is in a consistent state before publishing. • NxPrepExtend • A NxPrepExtend script can cause NxPrep to fail if the script: • Runs a command that blocks waiting for user input or user action. • Becomes unresponsive or returns an error. • Terminates the NxPrep VM or prevents it from shutting down as expected.

23. NxPrep Failures Caused By Synchronizer Issues • NxPrep requires coordination between Engine and Synchronizer. • Synchronizer problems can cause NxPrep to fail. • Network problems: • Between Engine and Synchronizer. • Between Synchronizer and the Database or Active Directory. • Between a remote Synchronizer server and the primary server. • Active Directory configuration problems: • Incorrect/changed/expired password or account disabled. • Insufficient privileges (most commonly, unable to create computer accounts). • AD configuration can be tested in the Synchronizer console.

24. NxPrep Best Practices • Sizing Computers for XCE Engine • Include a memory allowance for NxPrep. • 512 MB if only 32-bit VMs will be run. • 1024 MB if any 64-bit VMs will be run. • Otherwise the VM might be unavailable to the user during NxPrep. • Getting Started with VM Images in Synchronizer • Start with a VM image containing a very basic Windows installation. • Verify the basic Windows image can publish, download, install, and run as expected. • Then build up the image incrementally with applications and services. • Before Publishing a New VM Image Version • Ensure there are no pending updates or configuration changes in the image. • Verify all applications are installed and working properly. • Run the VM image through a clean start/login/logout/shutdown cycle.

25. NxPrep Best Practices (continued) • Always Publish for Staged Deployment • Assign the staged version to test users and verify: • The new image version can download and install properly. • The new image version contains the desired changes. • Applications and services in the image are working properly. • Do not deploy the new version to production users until it has been tested. • Logging Into Hyper-V VMs • When managing VM images, always login as one of two users: • The local Administrator. • A domain account created for image management. • User profiles created in the image will carry over to deployed VMs. • VM Image Management • Turn off the Windows “system protection” setting for the system disk. • Avoid copying large files to the system disk. • Even if they are deleted later, they will cause the image to grow. • Install or update software from network shares. • Turn off all automatic update mechanisms for Windows and applications. • Antivirus software services should be disabled for publish and NxPrep.

26. NxPrep Logs in Running Virtual Machines Publish and NxPrep Log Folder In a running VM installed on Engine, the folder C:\NxTop contains log files generated when the VM image was published (in the Synchronizer) and installed (on the Engine). This is a hidden folder. NxPrep Log Files The log file C:\NxTop\nxprep.log contains a detailed record of what was done during the NxPrep process. It can be useful to review if NxPrep succeeds but the installed VM is inconsistent in some way. For example this log can be checked to ensure that a NxPrepExtend script was run successfully. Log Files for Installed Version Only The C:\NxTop folder will only contain log files for the NxPrep operation that successfully produced the running version of the VM. If the NxPrep process fails while trying to install a newer version of the VM image, the results of the failed NxPrep would not be reflected in the log files within the running VM.

27. NxPrep Logs in Engine Diagnostics NxPrep logs are included in the standard Engine diagnostics. Each VM will have its own set of NxPrep logs. The logs are for the “last completed” NxPrep cycle, meaning that NxPrep has run through to completion, or both NxPrep attempts have failed. Diagnostics will not include logs for NxPrep in progress. You must wait until both NxPrep attempts fail. This might mean that you must wait for NxPrep to be timed out twice. If you want Engine diagnostics for a NxPrep problem, do not collect them while NxPrep is in progress.

28. Foreground NxPrep

29. Background vs. Foreground NxPrep • Background NxPrep • This is the default mode for NxPrep. • The console for the NxPrep VM is not visible to the user. • NxPrep should complete automatically with no user action. • Displaying the NxPrep VM console could be confusing to the user. • Foreground NxPrep • Sometimes it is desired to run the NxPrep VM in the foreground. • So the console is visible and accessible. • This should only be done when necessary for troubleshooting. • For NxPrep failures caused by problems within the NxPrep VM.

30. Enabling Foreground Install • Foreground install must be enabled in two places: • In the Synchronizer Engine policy. • This is usually done for a specific user by overriding the default policy. • In the Engine control panel. • This can only be done after the Engine policy has been updated to allow foreground installs.

31. Enabling Foreground Install in Synchronizer Select the Policies tab, then the Support section. Locate the user in the navigation panel. Enable the “Allow Foreground Install” option, then save the policy changes. The change will go into effect when the Engine checks for updates with Synchronizer.

32. Enabling Foreground Install in the Engine Start the Engine control panel and choose the “Tools by Category” view. Start the Activity Center applet. Click the link under “Related Tasks” to enable foreground installation. This can only be done if foreground installation is enabled in the Engine policy. If the “Turn on Visible Installation” option is not displayed, ensure foreground installation is enabled for the user or computer in the Synchronizer.

33. Foreground NxPrep Example After the update is downloaded… The VM would normally go into an “Installing Update” stage… But with foreground installation enabled, the NxPrep VM should start in the foreground like an installed VM. When the NxPrep VM is in the foreground, pressing the Ctrl-Down hotkey should show that the VM is in the “Installing Update” phase.