Best Practices in Architecting & Implementing Windows Server Update Services (WSUS)

1. SESSION CODE: WSV208 Best Practices in Architecting& Implementing Windows Server Update Services (WSUS) Greg Shields Partner & Principal Technologist Concentrated Technologywww.ConcentratedTech.com

2. Agenda • Topics • Part I: Architecting & Implementing WSUS • Part II: Troubleshooting WSUS • Part III: Tips & Tricks for Using WSUS

3. Architecting & Implementing WSUS Part 1

4. WSUS Product Vision • Simple, zero-cost solution for distributing Microsoft Updates content in a corporation. • A “free” RTW add-on for Windows Server • Solution only distributes Microsoft Updates • Distributing 3rd party patches require purchasing advanced management tools such as SCE or Configuration Manager 2007 • Provides a foundation for Update Management across Microsoft products: SCE, Configuration Manager 2007, MBSA, WU, SBS, Forefront, … • Consistent scan results • Unified client scan mechanism (WUA) irrespective of which server actually manages the updates.

5. WSUS Momentum • Over 500,000 distinct WSUS servers synched with Microsoft Update last month • Used by over 60% medium/large orgs and built into SBS • WSUS 3 released April 30 2007 • Huge improvements in performance, deployment options, reporting and UI • Easy in-place upgrade from WSUS2 • WSUS 3.0 SP1 released Feb 7, 2008 • WSUS 3.0 SP2 released Jan 26, 2009

6. WSUS Lifecycle/Roadmap • Support lifecycle

7. WSUS 3.0 SP1/SP2 Adds Features • WSUS 3 SP1 added the following features: • Installs on Windows Server 2008, integrated with Server Manager (after installing Server Manager update KB940518) • API enhancements for advanced management tools • Bug fixes • WSUS 3 SP2 adds: • Installs on Windows Server 2008 R2 • Supports managing Win7 clients • Support for BranchCache • Auto-approval rules with deadlines • Bug fixes (DSS gets languages from USS, target groups sorted alphabetically, more robust setup upgrade) • Compliance against approved updates

8. New Features in WSUS SP2 Greg Shields Partner & Principal Technologist Concentrated Technology www.ConcentratedTech.com demo

9. Elements of ArchitectureWhy Architecture? • Problems are usually results of improper architecture • A correct architecture will drive a better design • Especially in situations of administrator distrust or insufficient bandwidth • Design your WSUS solution with the same goals as your AD solution • Roaming users should be dealt with separately

10. “Simple” Architecture • Single, well-connected site • WSUS Updates from MU • Clients update from WSUS • Single server can handle 25,000 clients • 50K clients with 2x front-end servers and big SQL back-end • Remote SQL configuration reduces server load • Front-end handles update sync load • Back-end handles reporting load

11. “Simple, with Groups” Architecture • Largest use case in production today • Driving forces to move to Machine Groups: • Differing patching requirements or schedules • Test groups • Servers vs. Workstations • Politics • Not necessarily used for load distribution

12. WSUS Chaining • Chaining involves downstream servers getting updates (and sometimes Group data) from upstream servers • Options for chaining • Distributed vs. Centralized model • “Autonomous Mode” vs. “Replica Mode” • Chaining solves the problem of “mesh” or “fully independent” architectures • Wastes resources and bandwidth • Not that some situations don’t mandate “mesh” or “fully independent” architectures!

13. “Centralized” Architecture • Downstream servers are replicas of primary server • Little downstream control over servers • Downstream admins drop machines into predefined groups • All update approvals and schedule done at primary server

14. “Distributed” Architecture • Downstream servers obtain updates from primary server, except: • Update approvals do not flow down. Assigned at each site individually. • Downstream admins have greater control. Can create groups and assign approvals. • Used for distribution rather than control of updates Combinations of centralized anddistributed possible. Depends onintra-IT trust model.

15. “Disconnected” Architecture • Many environments don’t have Internet connectivity. • Test/dev, government, classified, air gap environments • Data must be imported from “the outside” • Any the previous architectures will work • Manual import process required • Gives CM/QA/Security the option to review updates prior to bringing “inside”. Sneakernet

16. “Disconnected” Architecture • Match advanced options between source and target. • Express installation files & languages must match. • Backup & restore updates from source to target. • Back up C:\WSUS\WSUSContent • Restore to the same location on the target server. • Transfer update metadata from source to target. • Navigate to C:\Program Files\Update Services\Tools • Export metadata using wsusutil.exe export {packageName} {logFile} • Import with wsusutil.exe import {packageName} {logFile} • packageName & logFileare unique names you choose Database validation can take multiple hours to complete!

17. Laptop WSUS Laptops “Roaming” Architecture • Manages updates for external resources • WSUS servers distribute approval metadata • Clients download updates from Windows Update directly. • Extra security for internet-facing WSUS server • Useful separate architecture for mostly off-net clients

18. Laptop WSUS Laptops “Roaming” Architecture • Four Steps to Internet-facing WSUS • Build server in DMZ and position behind ISA proxy • Locate database on server not reachable from Internet • Enable SSL for communications • Host content on Microsoft Update

19. “High Availability” Architecture • WSUS 3.0 includes native support for high availability • NLB Clusters connect multiple WSUS web servers via a single cluster IP • SQL Cluster manages the database • No single point of failure • Critical: This design isuseful for availability,but does little forperformance.

20. Managing Branch Offices • Branch offices are typically managed through replica WSUS servers • Replica servers take all orders from the central server. • Settings at the top flow downward, but take time. • Alternatively, unify architecture through a single “central server” • Single server manages all clients across all offices • Deploy ISA proxy in the branch • Enable BITS peer-caching • Use delta files to reduce network traffic. • 10x more server disk space • 4x less client download

21. Upgrade deployment • WSUS 3 SP1 setup supports in-place upgrade • One-way upgrade (no rollback) • Can’t be done from WSUS 2 on Windows Server 2000 or using SQL 2000 • Alternative is migration upgrade: • Install second server • If original server is WSUS2 SP1: • Perform disconnected replica steps (wsusutil, ntbackup, wsusmigrate) • Switch over client via policy • If original server is also WSUS3 • Configure new server to be a replica of the first and sync • After sync, configure new server to be autonomous • Upgrade hierarchy from top down

22. Troubleshooting WSUS part 2

23. Errors and Error Codes • Numerous WSUS error codes exist. • A complete list of all WSUS error codes is available on-line at http://inetexplorer.mvps.org/archive/ windows_update_codes.htm • For example, 0x8DDD0018 occurs when one of these services is Disabled • Automatic Updates • BITS • Event Log

24. Errors and Error Codes II • 0x80072EE2, 0x80072EFD • This issue occurs because the Windows Update client did not receive a timely response from the Windows Update Web site server. • Likely a proxy configuration, personal firewall, or trusted hosts problem

25. Errors and Error Codes III • 0x80246008, 0x8024402C • Caused by BITS malfunctioning or corrupted. • Download and extract the BITSAdmin tool from the Windows Support Tools CD. • Bitsadmin /util /repairservice /force • If that doesn’t work, try a BITS re-install • Though if you do a BITS re-install, clear out the %SystemRoot%\SoftwareDistribution folder and reboot when done. Its worth mentioning here that thereis no “backup” download process for WUA. …like HTTP or FTP…If BITS is non-functional, so is patching!

26. Errors and Error Codes IV • 0x80244019 • This error is often caused when the Proxy server is not properly configured. • Ensure that your Proxy server allows Anonymous access to these external addresses: • http://windowsupdate.microsoft.com • http://*.windowsupdate.microsoft.com • https://*.windowsupdate.microsoft.com • http://*.update.microsoft.com • https://*.update.microsoft.com • http://*.windowsupdate.com • http://download.windowsupdate.com • http://download.microsoft.com • http://*.download.windowsupdate.com • http://wustat.windows.com • http://ntservicepack.microsoft.com Microsoft doesnot publish the IP’sassociated with theseFQDN’s.So, if you do perimeternetwork security by IPyou’ve gotta’ stayon the ball with these!

27. WUA Client Issues • To enable auto-updates, ensure: • Anonymous access granted to Self Update virtual directory on WSUS server • Auto-updates requires TCP/80 to function on WSUS server • Be aware of GP replication times • 90 to 120 minute GP refresh timing will impact speed of clients becoming visible in WSUS admin tool • Be aware of AU detection frequency times • WUA client set to check with server every 22 hours (minus offset). • When WUA checks in is when it checks WUA version. • Need to do wuauclt /detectnow to force this to occur on-demand.

28. WUA Client Issues II • Known issue with imaged workstations: • If you image your workstations (and who doesn’t these days!), you must change SID • Sysinternals NewSID, Microsoft SysPrep • Not doing this will prevent WUA from contacting WSUS • To fix this problem: • Run one of the above tools to change the SID • HKLM\Software\Microsoft\Windows\ CurrentVersion\WindowsUpdate • Delete PingID, SUSClientID, and AccountDomainSID values • Restart wususerv service • Run wuauclt /resetauthorization /detectnow

29. WUA Client Issues III • Disabling the Automatic Updates Service or the BITS Service at any point in the past prevents it from starting properly when you need it! • Reset permissions on these services to re-enable functionality. • Use the Service Control Resource Kit tool (sc.exe) to do this: • sc sdset bits "D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)" • sc sdsetwuauserv "D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)“ • Every disabled client needs this!

30. Tips & Tricks for Using WSUS part 3

31. Optimize Patch Distribution • In large, multi-site environments low bandwidth may cause problems for remote offices. • Distributing updates to downstream servers is big problem • Potential solutions: • Ensure downloading only the languages you need • Configure patch distribution to occur in the evenings. • Stagger patch distributions between tiered sites • Express installation files can exacerbate this. • The bandwidth savings in express installation files occurs from WSUS server to client, not between WSUS servers. • Throttle BITS

32. Throttling BITS • BITS can be throttled either on the WSUS server or additionally on all the clients. • Alleviates network saturation during update distribution and during client installation • Be aware that this does slow down update distributions! • Throttle BITS in Group Policy: • Computer Configuration | Administrative Templates | Network | Background Intelligent Transfer Service • Two settings: • Maximum network bandwidth that BITS uses • Limit by Kbps based on time of day or at all times • Be aware that Kbps is kiloBITS not kiloBYTES (divide by 8) • Timeout (in days) for inactive jobs

33. DNS Netmask Ordering • Non-centralized architectures can better route clients through DNS Netmask ordering. • Microsoft DNS Round Robin will first provide an IP address in the same subnet as the requestor. • If no IP exists in the same subnet, a random IP will be selected. • All WSUS hosts must respond to the same FQDN. • DNS FQDN record is populated with IP addresses of all WSUS servers in the network.

34. Server Tuning • Run cleanup and DB defrag every few months • Cleanup wizard is a feature in WSUS 3 • Removes stale computers and updates • DB index defrag script available on ScriptCenter • keeps the server running fast • Look out: • Take care to not remove computers that are still active (but having trouble contacting the server) • Populate from AD sample tool can help • In a hierarchy, need to run cleanup on each WSUS server. • Clean computers from bottom-up • Clean updates from top-down (or between sync intervals) • Can be automated through the API

35. Considerations for Updating Servers • Servers require more care than workstations… • A rebuild is usually not an acceptable solution for a failed patch installation. • Outage windows are shorter. • But in some ways servers are easier… • Data and system drives usually separated. • Hardware configuration is usually more stable or well-understood. • Service isolation and redundancy – in larger environments – limits exposure/risk. • People typically aren’t “surfing” on servers. • The RAID 1 Undo Trick…

36. What About Reboots? • I’ve said this before, and I’ll say it again: • “If you have a patch management plan without a reboot strategy, you don’t have a patch management plan.” • Three methods: • Client-initiated • WSUS-initiated • Script-initiated • Two methodologies: • Scheduled reboots vs. rebooting for patch installation I will argue in favor of scheduled, forced rebootsover mid-day reboots.

37. Handling Reboots RebootFile = "computers.txt“ LogFile = "results.txt" Set fso = CreateObject("Scripting.FileSystemObject") Set f = fso.OpenTextFile(RebootFile, 1, True) Set objTextFile = fso.OpenTextFile(LogFile, 2, True) On Error resume next Do While f.AtEndOfLine <> True strComputer = f.ReadLine Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2") If Err.Number <> 0 Then objTextFile.WriteLine(strComputer & " is not responding.") Err.Clear Else Set colOperatingSystems = objWMIService.ExecQuery("Select * from Win32_OperatingSystem") objTextFile.WriteLine(strComputer & " is rebooting.") For Each objOperatingSystem in colOperatingSystems ObjOperatingSystem.Reboot() Next End If Loop

38. Custom Reports • UI supports basic customization (filters) • Advanced customization can be built on • WSUS (.Net) API • Can use of PowerShell scripts to generate reports • Public read-only SQL views • Can use SSRS to generate reports (if full SQL) • Samples available from MSDN • E.g., compliance against approved updates

39. Match KBs to MSRCs • Ever wish you had a nice mapping of knowledgebase numbers to MSRC numbers? • “The Q-numbers to the MS-numbers” • This script outputs a .CSV file that provides just that mapping • Add the name of your WSUS server into the top line of the script: strWSUSServer = “<Enter WSUS Server here>"

40. Match KBs to MSRCs strWSUSServer = “<Enter WSUS Server here>" Set fso = CreateObject("Scripting.FileSystemObject") Set objTextFile = fso.OpenTextFile("OUTPUT.csv", 2, True) objTextFile.WriteLine("MS Number,Q Number") Set conn = CreateObject("ADODB.Connection") Set rs = CreateObject("ADODB.Recordset") dbconn = "Driver={SQL Server};Server=" & strWSUSServer & ";Database=SUSDB" conn.opendbconn strSQLQuery = "SELECT dbo.tbSecurityBulletinForRevision.SecurityBulletinID, dbo.tbLocalizedProperty.Title FROM dbo.tbLocalizedPropertyForRevision INNER JOIN dbo.tbLocalizedProperty ON dbo.tbLocalizedPropertyForRevision.LocalizedPropertyID = dbo.tbLocalizedProperty.LocalizedPropertyID INNER JOIN dbo.tbSecurityBulletinForRevision ON dbo.tbLocalizedPropertyForRevision.RevisionID = dbo.tbSecurityBulletinForRevision.RevisionID WHERE (dbo.tbLocalizedPropertyForRevision.LanguageID = 1033) ORDER BY dbo.tbSecurityBulletinForRevision.SecurityBulletinID" rs.OpenstrSQLQuery, conn, 3, 3 While Not rs.EOF objTextFile.WriteLine(rs.Fields(0).Value & "," & Replace(rs.Fields(1).Value, ",", "")) rs.MoveNext Wend WScript.Echo "Done!"

41. Agent Control • Use WUA API to control the agent • Custom install schedules • Updating servers in web farms • Implementing “install now” functionality

42. On-Demand Patching(You Patch Now!) • Ever wish you had a WSUS “Big Red Button”? • Such a button might automatically download and install all approved patches and reboot if necessary… • How about this VBScript? • Run this script from any server console • Immediately downloads and installs all approved patches. • If a reboot is required, it will then reboot the server.

43. The WSUS Big Red Button Set fso = CreateObject("Scripting.FileSystemObject") Set objAutomaticUpdates = CreateObject("Microsoft.Update.AutoUpdate") objAutomaticUpdates.EnableService objAutomaticUpdates.DetectNow Set objSession = CreateObject("Microsoft.Update.Session") Set objSearcher = objSession.CreateUpdateSearcher() Set objResults = objSearcher.Search("IsInstalled=0 and Type='Software'") Set colUpdates = objResults.Updates Set objUpdatesToDownload = CreateObject("Microsoft.Update.UpdateColl") intUpdateCount = 0 For i = 0 to colUpdates.Count - 1 intUpdateCount = intUpdateCount + 1 Set objUpdate = colUpdates.Item(i) objUpdatesToDownload.Add(objUpdate) Next ‘<<This is only the first half of the script. Add the code from the next page to ‘create the full script>>

44. The WSUS Big Red Button ‘<<Add this half to the code on the previous page!>> If intUpdateCount = 0 Then WScript.Quit Else Set objDownloader = objSession.CreateUpdateDownloader() objDownloader.Updates = objUpdatesToDownload objDownloader.Download() Set objInstaller = objSession.CreateUpdateInstaller() objInstaller.Updates = objUpdatesToDownload Set installationResult = objInstaller.Install() Set objSysInfo = CreateObject("Microsoft.Update.SystemInfo") If objSysInfo.RebootRequired Then Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate,(Shutdown)}!\\localhost\root\cimv2") Set colOperatingSystems = objWMIService.ExecQuery("Select * from Win32_OperatingSystem") For Each objOperatingSystem in colOperatingSystems objOperatingSystem.Reboot() Next End If End If

45. Other API Uses • ISVs use APIs for many other features as well • Distribute 3rd party updates (quite complex) • Gather software and hardware inventory • Distribute updates to non-Windows devices • Your starting point is http://technet.microsoft.com/en-us/wsus/bb466192.aspx • API Samples • Diagnostic Tools • Header Files

46. Summary • WSUS is simple to use, but scales to enterprise • Flexible server deployment options • Single server, scale up, branch office, scale out, disconnected, roaming laptops • Flexible update deployment options • Peer caching, delta patching, auto approval rules, auto-reapprove revisions • Periodically tune the server (defrag + cleanup) • Public API and DB views can be used to extend the base functionality for many advanced scenarios • Starting point for all WSUS information http://www.microsoft.com/updateservices

47. Required Slide Resources Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet • http://microsoft.com/msdn

48. question & answer

49. Required Slide Complete an evaluation on CommNet and enter to win!

50. Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st http://northamerica.msteched.com/registration You can also register at the North America 2011 kiosk located at registrationJoin us in Atlanta next year