Keeping HP-UX Up-To-Date and Patching Best Practices

1. Keeping HP-UX Up-To-Date and Patching Best Practices Dusan Baljevic, HP Customer Education Sydney, Australia

2. Acknowledgements These slides have been used in various presentations in Australia over the last several years. This is a work-in-progress and updates are frequent. I bear full responsibility for any error, even though it is purely unintentional. I cannot claim credits solely, nor can I claim that I know everything about Unix. I consider myself to be a Unix Apprentice. Wisdom of many helped in creation of the presentation (seminars at HPWorld, ITRC/HPSC forums, HP Ambassadors and Unix Profession members, HP Education courses, individual contributions on the Net).

3. HP-UX Network Design • At a minimum, three fully-firewalled, separate networks are recommended for HP-UX servers. It is assumed that such best practice is enforced. • Corporate and Management LAN can be an Auto Port Aggregate (APA). • Management LAN is typically used for protocols like NTP, DNS, LDAP, remote Ignite-UX, remote SD-UX, DHCP for clients, LAN-based backups, and similar. Corporate LAN Console LAN (ILO, GSP) Management (Confined) LAN

4. Seminar Agenda All commands and features listed in the presentation apply to HP-UX 11iv3. Similar would apply to older releases, where applicable. HP-UX Patching Versus Update-UX Update-UX HP-UX Patch Management Concepts Installing, Verifying, Removing, and Committing HP-UX Patches HP-UX Patch Management with SD-UX Depots HP-UX Patch Management with Software Assistant (SWA) HP-UX Patch Management with Dynamic Root Disk (DRD)

5. HP-UX Patching Versus Update-UX

6. HP-UX Patching Versus Update-UX 1 of 3 Full update-ux process is strongly recommended and preferred to standard patching. The update-ux method is quite safe and there are no “loose points”. If possible, we also encourage customers to use Software Assistant (SWA) on a regular basis. Patch bundles will patch existing software, but update-ux will update products (the core O/S, all the drivers and even independent software units that will not be updated during patching).

7. HP-UX Patching Versus Update-UX 2 of 3 The update-ux method is not only used to update from a lower to a higher version (for example, 11i v2 to v3), but also to update from an older to a newer release within the same version. For many reasons, we encourage usage of update-ux with Dynamic Root Disk (DRD). If O/S is upgraded through update-ux process, the best practice recommends cold installs; incremental upgrades might create possibility that some obsolete software and libraries exist afterwards.

8. HP-UX Patching Versus Update-UX 3 of 3 We recommend customers develop a release “cycle” through DRD implementation: Run update-ux every year (18 months or maximum two years is acceptable in some circumstances). Only break this cycle if they must have some new functionality in a bi-annual release. Unless specifically requested differently, the patch/update level should be at latest release, if practicable, or LATEST-1.

9. HP-UX Patch and Update Management • Patch/update management is a quite complex and involved topic. • There is no patch/update management plan that fits all situations. • Every company must determine the plan that fits best in their own environment and meets their business objectives. • A plan should be reviewed periodically because the environment and business objectives change over time, new tools and practices evolve, and operating systems evolve. All of these changes require modifications to existing patch management plans.

10. HP-UX Operating Environment 1 of 4 • HP strongly recommends that only a complete OE be installed and that no removal of Required products and bundles in the OE occur, unless Independent Software Unit (ISU) products are used. • HP-UX 11i OEs have been packaged and tested as complete solutions. • HP-UX 11i releases are delivered bi-annually (for 11iv3 it is typically in March and September).

11. HP-UX Operating Environment 2 of 4 • As of HP-UX 11iv3, ISUs are no longer delivered via the standard patch process or scheduled bi-yearly updates. For ISU products, defect fixes, performance enhancements, and new functionality, are delivered using the ISU model.  • ISUs are additional layered software products. • Each ISU update is cumulative so customers only need to install the latest update to receive all defect fixes, performance enhancements and updated functionality.

12. HP-UX Operating Environment 3 of 4 • A mechanism for handling OE subsets is not available. Installing applications delivered with an OE separate from the entire OE will not include those applications in the OE bundle wrapper, preventing some operations from identifying them as part of the OE. Installing or removing individual products in the OE may also impact the quality of the OE. If you choose to add or remove individual OE products to an 11i system or remove a product from an installed OE, be sure to specify all filesets listed for the target product. • Omitting a fileset will prevent the product (or other products that depend upon that fileset) from functioning and could hang the system.

13. HP-UX Operating Environment 4 of 4 • DRD only supports updating from 11.31.0709, 11.31.0803, or 11.31.0809 to 11.31.0903 or later releases. DRD may not be used to update from 11i v2 to 11iv3 (although it has been shown to work very well). • In a DRD scenario, update can be done with following alternatives. From a active disk run  drd runcmd update-ux, drd will run update on inactive disk. Active disk will not be altered. This option is not officially supported for 11iv2 to 11iv3 update. * Boot the inactive disk (activate the clone) and run update-uxcommand on it.  Active disk will not be altered. Run update-uxon active disk. Inactive disk (clone) will not be altered.

14. Examples How to Check HP-UX OE # swlist | egrep “\-OE” # swlist -l fileset -a install_date | grep OE # swlist -a install_date OS-Core # /opt/ignite/bin/print_manifest 14

15. HP-UX 11i v3 Boot Disk Cloning 1 of 2 If internal disks are used for booting, they should be on different controllers. It is a crucial requirement to allocate one or two disks (or LUNs) for boot disk cloning - Dynamic Root Disk (DRD). Creates a "point-in-time“ O/S image, On-line patching and configuration changes of the inactive O/S, Easier change management approvals because the active O/S is not affected (risk is eliminated), Some tasks make dynamic changes of the O/S during the cloning, without affecting the active O/S, Boot disk mirroring does not prevent disasters caused by human errors, If boot disks are on the same controller, mirroring is not a perfect protection.

16. HP-UX 11i v3 Boot Disk Cloning 2 of 2 With DRD, future upgrades and patching are very easy. It is strongly discouraged to use root volume group for any third-party applications. /var/tmp must have at least 32 MB free (if make_tape_recovery is used, the space is needed for LIF volume assembly).

17. HP-UX Backups Ensure that operating system backups are in place before the server is moved into production. Typically, Ignite-UX based backups, DRD, or SAN-based LUN snapshots are recommended. Ignite-based backups shall not include any non-root volume groups. Examples of Ignite backups to local tape drive and via network: # make_tape_recovery -x inc_entire=vg00 -x exclude=/tmp # make_net_recovery -s srvname -n 3 -P s –x \ inc_entire=vg00 -d "Archive of myclient“ Ensure that all applications and databases are backed up via proper (typically commercial) tools.

18. Update-UX

19. Update-UX Examples 1 of 2 Install updated O/S release from local depot# swinstall –s /mydepot Update-UX # update-ux -s /mydepot/11iv3VSE-OE HPUX11i-VSE-OE Install updated O/S release from local CD-ROM or DVD# swinstall –s /DVD Update-UX # update-ux -s /DVD HPUX11i-DC-OE Install updated O/S release from local depot via DRD# drd runcmd swinstall –s /mydepot Update-UX # drd runcmd update-ux -s /mydepot/11iv3VSE-OE \ HPUX11i-VSE-OE # drd activate ... 19

20. Update-UX Examples 2 of 2 Install updated O/S release from remote depot interactively# update-ux -i -s remsrv:/depot Install updated O/S release from remote depot# swinstall –s remsrv:/depot Update-UX # update-ux -s remsrv:/depot/11iv3VSE-OE \ HPUX11i-DC-OE Install updated O/S release from local depot via DRD# drd runcmd swinstall –s /mydepot Update-UX # drd runcmd update-ux -s /mydepot/11iv3VSE-OE \ HPUX11i-VSE-OE 20

21. HP-UX Patch Management Concepts

22. Why HP-UX Patches? HP releases patches for a variety of reasons: * New functionality, * New hardware support, * Bug fixes (including security issues), * Performance enhancements. • Lack of attention to this topic can lead to data loss, financial loss, exploits of vulnerabilities, damaged reputation, and other negative consequences.

23. HP-UX Patch Best Practices 1 of 4 Unless specifically requested differently, the patch level should be at latest release, if practicable, or LATEST-1. Main reasons for patching: stability and security. Unless specifically requested differently, regular patch audit should be enforced (via Remote Services, Software Assistant, HPSC* Patch Assessment, and similar offerings and tools). Four basic strategies are: * Proactive patch management (patching regularly to avoid problems). * Reactive patch management (patching after problem occurs). * Security patch management. * Install a new system (to replace old or un-patched one) .

24. HP-UX Patch Best Practices 2 of 4 Reactive patch management: * Fix an existing problem or security vulnerability; * Relatively unplanned activity. Proactive patch management: * Avoid potential problems; * Improve system reliability and availability; * Enable new hardware or software features; * Improve system performance; * Planned activity.

25. HP-UX Patch Best Practices 3 of 4 Ideally, the strategy should include proactive patching, reactive patching, and a separate plan for security patches.. Deploying patches should have three distinct processes: * Patch testing. Patches should be installed on one or more levels of preproduction systems and perform testing; * Planning deployment; * Installing patches.

26. HP-UX Patch Best Practices 4 of 4 There are three factors for patch strategy: * Restrictive; * Conservative; * Innovative. The decision must be based on: * Risk levels; * Maintenance window; * Number of local or remote systems involved; * Uniqueness of system configuration; * System and application availability.

27. HP-UX Patch Strategy

28. HP-UX Patch Naming Convention • HP patches follow a naming convention. • Note that PHKL patches usually require a system reboot. • Check patch README before installing. • The Patch name format is: PHxx_yyyyy, where: PH = Patch HP-UX. xx = Area patched: CO - general HP-UX commands. KL - kernel patches. NE - network specific patches. SS - all other subsystems and applications. yyyyy = Unique number (positive four or five-digit integer)

29. HP-UX Patch Supersession Chain • Patches from HP are usually cumulative. • Later patches may “supersede” older patches. • The final patch in a supersession chain provides a superset of the features and fixes provided by its predecessors. • If regular patching is not implemented, it is sufficient to install the latest patches. • Patch numbering scheme does not follow any pattern that ordinary users can understand. • Other vendors might release patches for their own HP-UX products in different formats (tar, cpio, zip, and so on). FOO-RUN PHCO_10237 PHCO_14721 PHCO_26118 superseded by … superseded by … superseded by … 29

30. HP-UX Patch Ratings • HP assigns every patch a rating, indicating how thoroughly the patch has been tested. • Visit the ITRC patch database to determine patch star rating. • Some customers only install 2- and 3-star patches. 30

31. HP-UX Patch Warnings • A patch warning is a notification that a patch causes or exposes adverse behavior. • See the HPSC patch database to review patch warnings. • HP distinguishes between “critical” and “non-critical” warnings. HP suggests a variety of remediation actions: • In some cases, such as if you encounter a critical problem on the system, immediate removal of the patch might be necessary. • In many cases, removal and replacement can wait until the next scheduled maintenance window. • In other cases, such as when the problem does not affect the hardware or software configuration, there is no need for you to take any action.

32. HP-UX Patch Types General Release versus Special Release Patches Critical versus Non-Critical Patches 32

33. HP-UX Patch Dependencies • Some patches require other patches or products in order to function properly. • SD-UX automatically enforces prerequisite, corequisite, and exrequisite dependencies. • Patch README may also describe manual dependencies not enforced by SD-UX. corequisites (may be installed in any sequence, or together) PHCO_10023 PHCO_20246 prerequisites (must install the prereq patches first) PHCO_10023 PHCO_20246 exrequisites (exrequisite patches are mutually exclusive) PHCO_10023 PHCO_20246 33

34. HP-UX Patch Dependencies and Supersession If a superseded patch is required to satisfy a dependency, then any superseding patches should satisfy the dependency too. PHCO_10000 maybe installed concurrently with corequisite patch PHCO_20246 orsuperseding patch PHCO_23109 PHCO_23109 supersedes PHCO_20246 PHCO_10000 corequisites supersedes Superseded patch PHCO_10402 does not meet PHCO_10000 corequisite dependency PHCO_10402 34

35. HP-UX Patch Structure • SD-UX organizes software and patches in hierarchical bundles, products, and filesets: • A fileset is a collection of related files. • A product or patch is a collection of related filesets. • A bundle is a collection of products or patches. Bundle: HPUXMinRuntime Patch Bundle: QPKBase Product: Networking Patch: PHNE_38680 Fileset: Networking.NET2-KRNFileset: Networking.NET2-RUN Fileset: PHNE_38680.NET2-KRNFileset: PHNE_38680.NET2-RUN applied to applied to Product: X11 Patch: PHSS_37226 Fileset: X11.X11-RUNFileset: X11.X11-RUN-MAN Fileset: PHSS_37226.X11-RUNFileset: PHSS_37226.X11-RUN-MAN applied to applied to

36. HP-UX Patch Attributes • Every SD-UX patch or product may have one or more attributes. • Attributes store SD-UX metadata information. • Some of the most useful patch attributes are shown below. What problem does patch PHCO_10000 fix? Are there any special instructions?# swlist –l patch [–s /depot] –a readme PHCO_10000 Will I have to reboot my system if I install or remove PHCO_10000?# swlist –l patch [–s /depot] –a is_reboot PHCO_10000 Which ancestor filesets does PHCO_10000 replace?# swlist –l patch [–s /depot] –a ancestor PHCO_10000 Which patch filesets does PHCO_10000 supersede?# swlist –l patch [–s /depot] –a supersedes PHCO_10000 Do I have a patch that supersedes patch PHCO_10000?# swlist –l patch [–s /depot] –a supersedes | grep PHCO_10000 View all of the attributes for patch PHCO_10000 filesets# swlist –l patch [–s /depot] –v PHCO_10000 View a description of all supported SD-UX attributes# man 4 sd 36

37. The state Attribute • Every fileset has a state attribute that indicates the current installation state. • After installing a patch, verify the patch state=configured Verify patch installation state# swlist –l patch –a state PHCO_10000 37

38. The patch_state Attribute • Patches have an additional patch_state attribute that indicates the status of the patch. • After installing a new patch, verify the patchpatch_state=applied Verify patch_state# swlist –l patch –a patch_state PHCO_10000 38

39. The category_tagAttribute • Every patch has a category_tag attribute containing one or more categories. • Some common tags include: • critical, enhancement, hardware_enablement, firmware • Category tags can be used as filters when listing patches. View a list of all category tags present on this system or depot# swlist –l category [-s /depot] View a specific patch’s list of category tags# swlist –l product [-s /depot] –a category_tag PHCO_1000 List all patches that fix critical defects# swlist –l product [-s /depot] –a category_tag ″PH*,c=critical″ List all enhancement patches # swlist –l product [-s /depot] –a category_tag ″PH*,c=enhancement″ 39

40. HP-UX Patch Sources • HPSC patch database Online database containing all available patches, accessible via FTP and HTTP • BUNDLE11i, HWEnable, and QPK patch bundles Patch bundles containing critical, tested Operating Environment patches • HPSC patch tapes Custom patch tapes available to some customers with support contracts • Local or remote SD-UX depot server Locally managed depot containing patches approved for your environment

41. HP-UX Patch Tools • SD-UX utilities: swinstall, swlist, swremove, swcopy, swverify Standard SD-UX utilities for installing, listing, and removing patches • Software Manager. • HPSC patch database search engine Web-based utility for searching the patch database and downloading patches • Software Assistant (SWA) CLI utility that analyzes an HP-UX system, and recommends and downloads security patches and quality pack patch bundles • Dynamic Root Disk (DRD) CLI utility that minimizes while installing and removing patches • HP Patch Assessment Tool Web-based utility that analyzes an HP-UX system, and recommends and downloads custom patch bundles

42. HP-UX Software Manager (SWM) 1 of 2 • SWM extends the functionality provided by SD-UX. • The major modes are similar to the following SD-UX commands: /opt/swm/bin/swm install swinstall /opt/swm/bin/swm job swjob /opt/swm/bin/swm list swlist /opt/swm/bin/swm oeupdate update-ux • Dry run and preview of a serial depot installation that does not require a reboot # swm install -p -x selection_output=- -x \ perform_analysis=true -s /var/myapp.depot myapp

43. HP-UX Software Manager (SWM) 2 of 2 • Dry run and preview of a serial depot installation that requires a reboot* # swm install -p -x selection_output=- -x \ perform_analysis=true –s /tmp/PHKL_41362.depot \* • Dry run and preview of an installation from a depot source (directory) # swm install -p -x selection_output=- -x \ perform_analysis=true -s /var/opt/mx/depot11 \*

44. Installing, Verifying, Removing and Committing HP-UX Patches

45. Downloading Patches from HPSC 1 of 4 http://h20566.www2.hp.com/portal/site/hpsc/public/ Enter your OSversion here Enter a search string here Specify a searchtype here Click [Search] 45

46. Downloading Patches from HPSC 2 of 4 Note the patchratings Click a patch name toread the .text file Select desired patches Click add to selected patch list 46

47. Downloading Patches from HPSC 3 of 4 Click downloadselected 47

48. Downloading patches from HPSC 4 of 4 Review specialinstructions Choose adownload format Click download Or, downloadindividual patches 48

49. Installing Single Patch from HPSC gzip archive • Do a full backup • Unzip the archive:# gzip -d /tmp/patches.tgz • Untar the archive:# tar -xvf /tmp/patches.tar • Unshar each patch:# sh /tmp/PHCO_10000 • Read the resulting .text file carefully:# more /tmp/PHCO_10000.text • Preview the installation# swinstall –p \ –s /tmp/PHCO_10000.depot \ -x autoreboot=true \ -x patch_match_target=true • Install the patch:# swinstall –s /tmp/PHCO_10000.depot \ -x autoreboot=true \ -x patch_match_target=true tar archive shar archive PHCO_10000.text PHCO_10000.depot

50. Installing Multiple Patches from HPSC PHCO_10000 • Do a full backup • Unzip the archive:# gzip -d /tmp/patches.tgz • Untar the archive:# tar -xvf /tmp/patches.tar • Copy the patches to a depot:# cd /tmp# ./create_depot_hp-ux_11 • Check for dependencies and special instructions # swlist –a readme –s /tmp/depot | more • Preview the installation:# swinstall –p \ –s /tmp/depot \ -x autoreboot=true \ -x patch_match_target=true • Install all of the patches from the depot:# swinstall –s /tmp/depot \ -x autoreboot=true \ -x patch_match_target=true PHCO_21345 PHCO_31104 Depot PHCO_10000PHCO_21345PHCO_31104