Patch Management

Patch Management

Key Terms • WUA: Windows Update Agent – Microsoft’s Patch Engine • LAN Share, Patch Repository, or File Server: In reference to patching, this is the endpoint that will host a shared folder and distribute patches to endpoints • Endpoint: Any single machine on which a Kaseya agent is installed • System Server or KServer: The server hosting the Kaseya front-end application • WindowsUpdate.log: This file records all actions conducted by WUA

Patch Management: Windows Update Agent

Windows Update Service v.Windows Update Client • Windows Update SERVICE is accessed via services.msc and must be enabled, started, and set to start automatically: • The Windows Update CLIENT the user-facing GUI and is accessed via the Control Panel or Program Menu on the endpoint. Kaseya recommends that the client be disabled:

What is WUA and when is it used? • WUA is Microsoft’s patch engine, the same background utility used when patching a home computer using Control Panel > Windows Updates. • Kaseya leverages WUA by invoking WUA.api, Microsoft’s exposed portions of the WUA utility • WUA is used for all patch scans and some or all patch installations (depending on File Source Configuration) • Scan: WUA negotiates with Microsoft to determine the patches that are missing from and applicable to the endpoint • Install: WUA negotiates all patch installations when the File Source is configured as “Download from Internet” OR when a patch is flagged as “Internet Based Install Only” • WUA depends on two services (Start > Run > services.msc) • Windows Update or Automatic Update (wuauserv) • Background Intelligent Transfer Service (BITS)

Windows Auto Update • Patch Management > Windows Auto Update is used to configure the Windows Update client on the local machine • Ensure Group Policy is not configured to enable the Windows Update Client. Group Policy will overwrite Kaseya’s setting. An entry will be made in the Configuration Changes log if this occurs. Review http://support.microsoft.com/kb/328010 for details on configuring Windows Update via Group Policy • Windows Action Center can also allow the user re-enabled Windows Automatic Updates and will result in an entry in the Configuration Changes log.

Patch Management: File Source

File Source Configuration: Options See KKB000794 for details on configuring a LAN Share for patching

File Source Configuration: Download from Internet Benefits • Endpoint negotiates directly with Microsoft • All WUA activity, including install failure codes, recorded into windowsupdate.log (eases troubleshooting) • Individual downloads may be smaller. WUA will download only the components of a patch that the endpoint needs. Considerations • Patches will be downloaded multiple times (each endpoint will download its own patch). This can be an issue in locations with limited bandwidth or where data rate overages are incurred.

File Source Configuration: Pulled from System Server Benefits • Endpoint does not need rights to download files from internet • Patches will only need to be downloaded the first time – any endpoint needing the same version of a patch will not invoke a separate request to the internet Considerations • Patch files are stored on Kserver (%KaseyaRoot%\WebPages\ManagedFiles\VSAPatchFiles) – consider HDD resources • KServer will distribute patches to endpoints during patch install processes which will consume processing resources. Ensure your KServer is scaled to account for this increased workload.

File Source Configuration: Pulled from file server using UNC (via Internet) Benefits • Endpoint server does not need rights to download files from internet • Patches will only need to be downloaded the first time – any endpoint needing the same version of a patch will not invoke a separate request to the internet. Considerations • Endpoints should be on the same Local Area Network as the LAN Share machine • You must use a Set Credential • Patch files will be stored on LAN share in defined patch repository folder. Ensure the server has sufficient resources to store the persistent data. • Patch file downloads may be larger than the WUA-installed version of the same patch because all components of the patch will be downloaded when using a LAN Share.

File Source Configuration: Pulled from file server using UNC (via System Server) Benefits • Endpoint does not need rights to download files from internet • LAN Share server does not need rights to download files from internet • Patches will only need to be downloaded the first time – any endpoint needing the same version of a patch will not invoke a separate request to the internet Considerations • You must use a Set Credential • Patch files are stored on Kserver (%KaseyaRoot%\WebPages\ManagedFiles\VSAPatchFiles). Patch files will ALSO be stored on LAN Share in the defined patch repository folder. Consider HDD space for KServer and LAN Share. • Patch files will be transferred up to three times: 1: from internet to KServer; 2: From KServer to LAN share; 3: From LAN share to endpoint. Consider amount of time the process will take to transfer these files multiple times and network bandwidth availability. • KServer will distribute patches to LAN Share during patch install processes which will consume processing resources. LAN share then distributes the patch to the endpoint. Ensure your KServer is scaled to account for this increased workload.

Patch Management: Patch Scan

Agent > Set Credential • You must use a credential if the endpoint’s File Source is configured as a LAN share • Credential Test validates that the provided account exists, is active, is not locked out, and that the password is correct. • Credential Test status reflects the results of the most recent test • The Credential should be explicitly added as a member of the local administrator group. In some environments, a Domain Admin may not provide sufficient privileges. • In some environments, it is necessary add the credential to the working directory permissions with Full Control. You may be advised by Kaseya Support to do this as part of troubleshooting. • Review KKB000733 for instruction on configuring the credential for appropriate access on the local machine.

Patch Status > Patch Test • Patch Test completes the same steps as the Credential test but also ensures that the credential has the access to the File Source, can download a test patch file from the File Source, and can execute that file. A Patch Test can fail with a credential-related error even when Credential Test is passing. • If your file source includes the option “Copy packages to the working directory on the local drive with most free space,” the patch test (and patch downloads) will complete against whichever drive has the most available space as of the most recent audit. • If your working directory is c:\kworking but the e:\ drive has more free space, the patch test files will write to e:\kworking (the folder will be created if it does not already exist). If the patch test cannot write to the e:\ drive, you may see a Patch Test failure with an error indicating that the test failed to write to e:\kworking. Ensure the drive still exists (portable drive was not removed), has sufficient space available, and that the Set Credential has Full Control of this folder. Re-run an audit if necessary to gather most current data. • If the Patch Test fails indicating the patch file could not be downloaded or executed, verify access to the five websites and/or to the LAN share, and that the credential has sufficient privileges to perform the action. • In some cases, patches can install successfully even when Patch Test is failing. The reverse can also be also true. Patch Test is a good indication of whether installs will be successful, but the possibility of failure still does exist. E:\kworking c:\kworking C:\ 5GB free E:\ 8GB free

Patch Scan Process A patch must be release by Microsoft into the Update Catalog (catalog.update.microsoft.com) AND detected as needed via a patch scan by at least one endpoint in your environment to populate in the VSA. Review KKB000676 for more detail.

Patch Scan Results • Enters records in database tables that hold the patch-specific information retrieved from Microsoft • Associates the unique patch to the machine where the patch scan discovered the patch as missing • KB number is not a unique identifier – there can be multiple patches with the same KB number. Each version of a patch will have a unique ID assigned by Microsoft. The “Update Identifier” is a four-part key that is found on the patch data pop-up for that patch. Click a hyperlinked KB number within the VSA to see the Update Identifier: Patch Revision Patch ID Download ID Download Revision

Primary v. Alternate Data Source • Primary Data Source: Microsoft’s online patch repository • Alternate Data Source: Microsoft’s wsusscn2.cab offline scan file, distributed to the endpoint as needed • Kaseya leverages WUA for patch scans. If WUA cannot access the Microsoft sites to complete the scan, WUA will complete the scan against the wsusscn2.cab. The .cab file contains an extremely limited number of patches: only the most recent, active Service Packs, Security Updates, and Update Rollups. Review http://support.microsoft.com/kb/926464 for additional detail. • WUA only “finds” patches that are missing from and applicable to the endpoint • Patches are determined to be missing if the WUA scan “finds” the patch • Scans against the .cab file can lead to patches incorrectly reported as Failed. Patches are determined to be failed if the WUA scan “finds” a patch that was previously scheduled for install, the install attempt occurred, and the patch is still “found” by WUA. Add/Remove programs is not referenced in regard to successful/failed installs. • Any scan that completes against the Alternate Data Source will return limited or no results • Endpoints without access to the Microsoft websites will always complete against the alternate data source. • If endpoints should have access to the internet, then indication of the use of the alternate data source is something that must be investigated. • Review KKB000781 for information on troubleshooting use of the Alternate Data Source

Primary v. Alternate Data Source (cont.) Review Agent > Agent Logs > Agent Procedure Log for detail regarding the patch scan Patch scan against the Primary Data Source: There are no indications of the Alternate Data Source. The scan completed against the Primary Data Source (Microsoft’s online catalog). Patch scan against the Alternate Data Source: Search online to determine the meaning of this error The “Result” code listed is the WUA failure code

Common WUA Error Codes • 0x80070422: BITS service is disabled • Enable BITS • 0x80248014: c:\Windows\SoftwareDistribution is inaccessible to WUA • Stop wuauserv service, delete SoftwareDistribution, restart wuauserv service • 0x80070643: .NET framework may be corrupt • Review http://support.microsoft.com/kb/976982 for automated and manual resolution options • 0x80072EFD: Microsoft reports this can be due to temporary internet connection problems, but we see this most often when a firewall, proxy, or other security hardware/software/appliance is blocking access to the MS sites. • Review http://support.microsoft.com/kb/836941 for additional detail. • Add the five websites to firewall rules, allow anonymous browse access via proxy, update web filter, etc. to allow appropriate access. • 0x0024002: A service is stopped. This error is most often thrown when the wuauserv is stopped or disabled. • Start the Windows Update/Automatic Update service (services.msc or cmd > net start wuauserv) • 0x8024402c: The proxy server or target server cannot be name resolved • Check cmd > netshwinhttp show proxy to verify proxy address and port are correct • 0x80240020: Interactive user is required to complete installation. This error is most often found in the WindowsUpdate.log file after a failed installation. • 0x80240021: Operation Timeout. This can occur when network infrastructure is throttling traffic, particularly when downloading larger patches. This error is most often found in the WindowsUpdate.log file after a failed installation. • Resources: • http://technet.microsoft.com/en-us/library/cc720432(WS.10).aspx • http://inetexplorer.mvps.org/archive/windows_update_codes.htm

Patch Management: Patch Policy

Patch Approval Levels • Approved: the patch will install when running Initial Update or Automatic Update • Denied: The patch will not install via Initial Update or Automatic Update but CAN be installed using Machine Update or Patch Update • Pending Approval: Functionally, Pending Approval = Deny. This approval level allows admins to more easily identify newly discovered patches in their VSA. Patches Pending Approval will not install via Initial or Automatic Update. Patches Pending Approval can be installed using Machine or Patch Update

Patch Approval Levels • All patches have two categorizations: Classification and Product: • A patch must be approved by both classification and product to be approved. On the Approval by Policy page, you can toggle between Classification and Product using the Policy View dropdown: • If there is a conflict between the Classification and Product, the more restrictive setting will take precedence: • Approved + Denied = “Denied” • Approved + Pending Approval = “Denied (Pending Approval)” • Default Approval Status applies only when a patch is first discovered by the VSA. Any patches already known to the system at the time the policy is created or Default Approval Status is changed will follow the approval level that was in place at the time the patch was first discovered • Use Patch Management > Approval by Patch to approve or deny a patch for all patch policies • Use Patch Management > KB Override to approve or deny all patches matching a single KB article ID regardless of patch policy settings. Using KB Override, you can deny a patch that has not yet been discovered by the VSA (you must know the KB number).

Creating Patch Policies • There are multiple approaches to creating patch policies. Each organization will have its own unique needs. Consider layering patch policies to avoid needing a large number of unique policies that need to be managed • Approach Patch Policies considering which patches you want to EXCLUDE/DENY rather than which you want to INCLUDE/APROVE. For example: • Create a Base policy that would apply to all machines. Deny all patches you do not want to deploy to any machines, regardless of organization or machine type • Create machine type-specific policies (Server, Workstation). Deny only those that are specific to patches you want to deny for the machine type and approve ALL others. • Create product-specific policies. Deny only those that apply to the product and approve ALL others. • This creates a series of filters that will only allow the patches approved in ALL associated policies to deploy

Layering Patch Policies • By layering the patch policies, admins can more easily manage patches and can quickly determine what kinds of patches are restricted for a specific machine • Because Deny will always take precedence, consider which patches you want to exclude and approve all other patches. Create policies that address single issues/types of patches

Creating Patch Policies

Patch Management: Reboot Action

Reboot Action • The “correct” Reboot Action will vary based on your users and business requirements • Reboot Immediately after Update: Machine is rebooted without warning • Reboot every day at 12:00am after install: will reboot on the selected day and time without warning, but will only occur if a patch installation cycle has run • Warn the user: A notification is displayed on endpoint indicating a reboot will occur in the defined number of minutes. The user cannot dismiss this reboot. • Skip reboot if user logged in: If the VSA detects a user is logged into the endpoint, the endpoint will not be rebooted. Patches will remain in pending until the endpoint is rebooted and a post-reboot scan runs. • If user is logged in, ask every x minutes: If no user is logged in, the endpoint will be rebooted immediately. If a user is logged in, a reboot nag will appear on the screen at the defined interval. The user can opt to “Continue Working” or “Reboot Now”. The nag will continue until a reboot and rescan occur. • If user is logged in, ask; Reboot if no response: A reboot will occur if no user is logged in. If a user is logged in, a notification will appear. If the user does not respond within the defined number of minutes, the reboot will occur without further warning. • If user is logged in, ask; Do Nothing: Same action as above, but no reboot will occur if the user does not respond to the notification. Patches will remain in Pending until a reboot and rescan occurs. • Do Not Reboot: The endpoint will not be rebooted. A preliminary rescan will run. If all patches previously schedule for install are no longer listed as Missing by the WUA scan, the pending flag will be cleared. Otherwise, patches will remain in pending until a reboot occurs. You can optionally generate an email to notify an admin that the machine requires a reboot. Note that the email will send regardless of whether the rescan is able to clear the pending flag.

Reboot Action (cont.) • All installation processes except Initial Update honor Reboot Action settings. • Initial Update will reboot without warning and as often as necessary to install all Missing Approved patches. Use only for non-production machines. • In 6.0 and later, the dynamic patch installation scripts are generated immediately before the cycle begins. 5.x scripts generate at midnight the day of scheduled install. If the reboot action is changed after the scripts are generated, the reboot for the scheduled cycle will the settings in place at the time the scripts were generated. The new reboot action will be honored for the next install cycle. • The next install cycle will be skipped if the previous cycle does not complete – this includes the reboot and rescan. • A manual reboot of an endpoint (either using the “Reboot” agent procedure OR Start > Restart does not complete the cycle. A Patch Reboot (by Reboot Action or clicking “Reboot Now” on the Patch Status page) schedules an immediate rescan. If a machine is rebooted manually, run a Machine Scan on the endpoint to clear the Reboot Needed flag and to stop reboot nag notifications for the end user.

Patch Management: Installation Overview

Patch Install Process Download From Internet LAN Share or System Server File Source: LAN Share or System Server • Automatic Update will always use the defined file source • Machine Update and Patch Update do not invoke WUA to install patches. The full executable file will be downloaded. • Patches will remain with a Pending status until the entire process complete – that includes the reboot and post-reboot rescan* *exception: when a preliminary rescan detects all scheduled patches as installed, the Pending flag will clear Script fires to execute each patch GET request is sent to file share or system server to retrieve patches Patch files are downloaded and sent to endpoint’s working directory Reboot Action is followed Post-reboot Rescan Dynamic scripts generated by KServer KServer verifies credentials WUA is invoked to process install WUA updates c:\windows\windows update.log WUA downloads and installs patch components File Source: Download From Internet

Patch Management: Best Practices Overview

Best Practices • All endpoints must have anonymous browse access to the following sites. • update.microsoft.com • download.microsoft.com • download.windowsupdate.com • www.windowsupdate.com • vsaupdate.kaseya.net • If access to the above sites is restricted, many patching functions will be impacted • Any patch repository server (whether a LAN share or the System Server) must have anonymous browse access to go.microsoft.com/fwlink/?LinkID=74689 • This Microsoft site hosts the offline scan file, wsusscn2.cab. This .cab file is the alternate data source. • Proxy cannot require authentication – Windows Update Agent does not support required proxy credentials • Windows Update Service (wuauserv) and Background Intelligent Transfer Service (BITS) must be enabled on the endpoint • Use an Agent > Set Credential that is explicitly a local administrator on the machine. A Domain Admin is not always sufficient as access restrictions can be placed at the domain level. See KKB000733 for details. • Use a customworking directory. Do not use Temp, Program Files, Windows, System, etc. “c:\kworking” is the current Kaseya default. • If using a LAN share as the file source, server should reside within same LAN as the endpoints referencing that sever. In most cases, it is not necessary for a single site to have multiple LAN servers as a patch file source (there are always exceptions to this). If the server is outside of the LAN, consider using “download from internet” as the file source for endpoints as this will often be less traffic. See KKB000794 for instruction on configuring a server to be a patch repository LAN share.

Best Practices (cont.) • Include the “Delete After Install” option in the file source configuration. Remove this option if troubleshooting a failed patch. • Avoid using a Domain Controller as the LAN File share/patch repository • Windows Auto Update shouldbe disabled (the client, not the service) • Reboot Action should be considered for each machine/type of machine. Servers may need a different reboot action than workstations. • Microsoft releases the bulk of their patches on the second Tuesday of each month (“Patch Tuesday”). Smaller numbers of patches are released out of cycle. Running Automatic Update no more often than weekly (no less often than monthly) usually provides the best balance between keeping endpoints up to date and keeping end user impact to a reasonable level. • Run Patch Status > Patch Test any time the File Source configuration is changed • Patch scan is the patch discovery process. Run scans regularly to detect the recent patches – at least once per week. It is not necessary to run scans daily, but plan to run them often enough to discover new patches which will be installed during the next Automatic Update. • Missing Approved patches for an endpoint are a result of the latest patch scan data compared with the endpoint’s Patch Policy membership. Patch score (Info Center > Reporting) is affected by the number of Missing Approved patches. Plan your scan and update schedules to maximize patch score if you are contracted to specific SLAs. • Ensure endpoints have access to the internet so Patch Scans can discover all missing patches. See KKB000781 for information on troubleshooting unexpected scan results • Enable and start Windows Update and BITS services on all endpoints. This can be done via Command Line using the command: net start <serviceName>

Patch Management: Patching through Proxy

Patching through a Proxy • All endpoints must have anonymous browse access to: • update.microsoft.com • download.microsoft.com • download.windowsupdate.com • www.windowsupdate.com • vsaupdate.kaseya.net • Proxy cannot require authentication – Windows Update Agent does not support required proxy credentials. See http://support.microsoft.com/kb/900935 for more information • If you are using a proxy, you may need to configure the Operating System to be aware of the proxy address so WUA can traverse the network. Use Microsoft’s proxycfg or netsh command line utilities to configure the proxy address in the OS: http://msdn.microsoft.com/en-us/library/windows/desktop/aa384069(v=vs.85).aspx • Example: • proxycfg –p <proxy>:<port> OR • netshwinhttp set proxy <proxy>:<port>

Patching through a ProxyConfiguring the LAN Share • If you use a LAN share to distribute patches, you may also need to enter the proxy information for the LAN share within the VSA. This will allow the download of the patch files through the proxy. This does NOT pass proxy credentials through to the endpoint or make those credentials accessible to WUA. To set the proxy within Kaseya to allow downloads, navigate to Agent > Agent Status and access the hidden proxy link to the right of the Select Columns button:

Patch Management: Patch Failures

Patch Failures Patches can fail for several reasons: • Endpoint configuration/resources • File Source configuration • Network security blocking access • Download failures • WUA failures • Invalid command line arguments • Improperly coded detection logic in the patch • WUA patch scan cannot reach Microsoft sites

Patch FailuresEndpoint Configuration/Resources • Ensure the credential is properly configured and that Credential Test and Patch Test are passing • Ensure the endpoint has sufficient resources to complete the installation (available drive space, memory, etc.) • Verify necessary services are started and endpoint can access MS patch sites • Ensure a local Windows Update scan can complete successfully • Review c:\windows\windowsupdate.log for detailed failure codes (only applicable for scan results and when the patch attempted to install via WUA)

Patch FailuresFile Source Configuration • Verify Patch Management > File Source is configured to a valid server/share • Verify the Set Credential AND System accounts have appropriate access to the share and the shared folder • Ensure the drive hosting the shared folder has sufficient available space • Verify downloaded patch files are valid (compare file size on server to file size listed on patch data page within VSA) • Log onto the endpoint as the account defined in Set Credential, attempt to map a drive from the endpoint to the LAN Share, copy a file, and paste the file into the endpoint’s working directory. • Review KKB0009794 for detail on configuring a LAN Share as a patch repository

Patch FailuresNetwork Security Blocking Access • Verify firewall rules allow access to • update.microsoft.com • download.microsoft.com • download.windowsupdate.com • www.windowsupdate.com • vsaupdate.kaseya.net • Verify proxy allows anonymous browse access to the above sites • Ensure web filters, ISP, other network infrastructure (including AntiVirus) components are not blocking access or downloads • Group Policy (GPO) can, more rarely, be responsible for blocking access. Verify GPOs are configured to allow the necessary access across your network.

Patch FailuresDownload Failures • Patch can become corrupt in transit. Verify patch at system server, LAN share, and endpoint. • Attempt to manually download the patch from Microsoft. Use Patch Management > Patch Location (visibile only to On-Premise Master accounts) to determine the download URL of the patch. • Remove the “Do Not Delete” option from File Source configuration and attempt to execute the patch. Many patches support creation of a log file using optional switches. Execute the patch using the /? switch to determine command line arguments for forcing log file generation.

Patch FailuresWUA Failures • Launch the Windows Update Client on the local endpoint, run a custom scan searching for ALL patches (not just OS-level patches). WUA can be self-healing and/or will provide a failure code and link to resolution options directly from Microsoft. • Review c:\windows\WindowsUpdate.log for additional error/failure codes. Gather the Patch ID from the Update Identifier and use that to search windowsupdate.log. Patches installed when leveraging WUA will record all installation attempts and the results (failures will include an error code) in this log file.

Patch FailuresInvalid Command Line Arguments • On occasion, a command line switch may be incorrect for a patch. This can lead to a failed install when using a LAN share or the Machine/Patch Update functions (WUA-based installs do not use command line switches). • Download the patch from the Patch Location URL and execute the patch via command line including the /? switch. Review the supported switches with those defined in Patch Management > Command Line for the patch. • If you find the command line may be incorrect, please report this finding to Kaseya Support so we can make the proper adjustments. Note that for many patches /q and /quiet are interchangeable, even when only one is listed as a supported switch.

Patch FailuresImproper Detection Logic • Kaseya relies on the results of the WUA scan to determine if a previously-scheduled patch is still being reported as missing OR is no longer being reported as missing. If a patch is missing after an installation attempt, it is marked as failed • Occasionally, Microsoft does not include the necessary detection logic within the patch or within WUA to determine that a patch is actually installed • In these cases, Kaseya will mark the patch as Failed • An attempt to install the patch via the local Windows Update client will result in a subsequent scan reporting the patch as missing even though the install attempt reported successful • WindowsUpdate.log can provide definitive answers regarding whether this is occurring if WUA is being leveraged for installations • the patch will be listed as an “added update” • the log will include a record that the patch successfully installed • a later scan will still list the patch as an “added update” • When patches are marked as failed due to incorrect detection logic, Kaseya recommends you verify the patch is installed and then Ignore the patch so it does not continue to install (see KKB000891)

Patch FailureWUA Patch Scan cannot access MS • If the patch scan completes against the alternate data source, the scan will not properly detect the status of missing patches • Kaseya relies on the results from the WUA scan to determine if a patch is missing • Any patch that was missing, is then scheduled, and after installation cycle is still missing will be marked as Failed • If patch scan is completing against the Alternate Data Source, address this first. You may find that, once completing against the Primary source, the patches will list as Installed after a scan. If not, reattempt the installation and allow the rescan to complete. Often, this fully resolves the ‘misreporting’ of patches as failed. • Downstream failures will need to be investigated further

Patch Management