IBM System z Technical University – Berlin, Germany– May 21-25

1. IBM System z Technical University – Berlin, Germany– May 21-25 Session code zZS25***** PRELIMINARY VERSION V *****Software Deployment Customer ExperiencesTim Alpaerts, Euroclear

2. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

3. Terminology • Installation system: one particular MVS system dedicated to installing software and installing fixes. • End-User system: all MVS systems other than the installation system. • Dead libraries: product libraries created on the installation system by software installation, (maybe SMP/E installed or non-SMP/E installed) not actively in use by an instance of the product. • Retrofit: after making a local correction on an end-user system, it needs to be incorporated into the dead libraries for future builds. • Version-less dsnames: product library dsnames that do not imply a particular version or release of a software, they do not contain a version release qualifier. • Non-Uniform software stacks: selectively activating only the needed parts of the complete enterprise software stack on every system, mainly for the purpose of reducing the license charge.

4. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

5. Pre-production Brussels The old situation, overview Pre-production Paris Development systems Production systems K-systems each group of systems: Test systems Paris Active MVS target & dlibs Inactive MVS target & dlibs MVS SMPE, within each group 2 target zones, non-IBM DASD was shared with CA-MIM 2 dlib zones products IBM IBM Test systems Brussels SMPE

6. Some issues with the old set-up • Version-release qualifiers in dsnames of product libraries make system upgrades risky. • Enqueue and reserve contention interference between lpars, instability from dead-locks. • No inventory nor overview of what system is using what software releases. • No obvious distinction between software product libraries and databases and work datasets. • Locally maintained PROGxx parmlib members accumulate errors over time • Low confidence when doing IPL.

7. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

8. Objectives for the new design (1) • Software installation needs to create dsnames (dead libraries) different from those the driving system is using. • All software installation work concentrated on one single system dedicated for this purpose. • All product libraries with version-less dsnames, to avoid widespread impact from system upgrades. • Distinction between R/O product libraries and R/W databases immediately apparent from the dsname. • Complete isolation of DASD between lpars in the IODF, no more cross-plex sharing of dasd.

9. Objectives for the new design (2) • All software, OS and all software products are upgraded in one go. • Upfront agreement on scope and planning (IPL slots). • Integrated testing. • All the “hold actions” are performed in parallel and are repeatable. • Provide a ‘perfect’ APF-list and link-list, complete but without any needless entries. • Ensure at all times we have an SMP perfectly in-sync for every software rolled-out on every system. • Upgrade per system if required by a freeze of the environment. • Complete inventory of software installation and deployment states. • Visibility of what is used where (licencing, cleanup of old S/W) • Installation guidelines and documentation is available the same way for all softwares.

10. The new design, overview (1) K-systems Production systems Development systems Test systems each system: &SYSR1 Active IPL-set Inactive IPL-set &SYSR2 6 model 9 vols 6 model 9 vols &SYSR3 &OEMR1 &OEMR2 &OEMR3 End-user systems

11. The new design, the IPL-set • IPL-set is restored from a (virtual) tape that was created on the installation system. • Each sysplex has two IPL-sets, the active set and the inactive set. • First volume of an IPL-set is the load device for the end-user system. • First volume restored from full volume dump to include the IPL text in the dump, the other volumes are dumped logically by dataset. • Second volume contains an ICF user catalog for the VSAM datasets on the IPL-set. • IPL-set is protected against updates by DFP OPEN exit IFG0EX0B, datasets can only be opened for INPUT.

12. The new design, the IPL-set, continued • Datasets on the IPL-set are cataloged with symbolic volsers defined in IEASYMxx parmlib member with substrings from &SYSR1:SYSDEF SYMDEF(&SYSR2.='&SYSR1(1:5)2') SYMDEF(&SYSR3.='&SYSR1(1:5)3') SYMDEF(&OEMR1.='&SYSR1(1:5)5') SYMDEF(&OEMR2.='&SYSR1(1:5)6') SYMDEF(&OEMR3.='&SYSR1(1:5)7')When MVS IPL processing sets &SYSR1, the volser of the IPL device to "IPCAA1", then IEASYMxx sets &SYSR2. = "IPCAA2" &SYSR3. = "IPCAA3" &OEMR1. = "IPCAA5" &OEMR2. = "IPCAA6" &OEMR3. = "IPCAA7"

13. The new design, overview (2) K-systems Production systems Development systems Test systemsLevel-set, 6 model 9 volumes each system: &SYSR1 Active IPL-set Inactive IPL-set &SYSR2 6 model 9 vols 6 model 9 vols &SYSR3 &OEMR1 &OEMR2 &OEMR3Installation system End-user systems

14. The new design, the Level-set • An exact image of the software for the end-user systems. • Generated by the TDSL application on the software installation system. • A ‘contents’ dataset is added with the list of all products and libraries on the LS. • The non-VSAM datasets on the LS are not cataloged, VSAM datasets get a special temporary HLQ. • Dsnames on the LS have no version-release qualifier (limited exceptions are allowed), • Exists only on the software installation system.

15. The new design, overview (3) K-systemsREF-set, SMS storage pool Production systems REFx datasets Development systems Test systems TDSL Level-set, 6 model 9 volumes each system: &SYSR1 Active IPL-set Inactive IPL-set &SYSR2 6 model 9 vols 6 model 9 vols &SYSR3 &OEMR1 &OEMR2 &OEMR3Installation system End-user systems

16. The new design, the REF-set • A large SMS-pool with dead-libraries • All the REF-datasets created by software installation go in this pool • ACS routines direct all allocations of REFx datasets to this pool • All REF-dsnames contain a version-release qualifier • Mostly vendor product libraries, but also Euroclear ‘config’ libraries • Exists only on the central software installation system

17. The new design, RENAME K-systemsREF-set, SMS storage pool Production systems REFx datasets Development systems Test systems TDSL Level-set, 6 model 9 volumes each system: &SYSR1 Active IPL-set Inactive IPL-set &SYSR2 6 model 9 vols 6 model 9 vols &SYSR3 &OEMR1 &OEMR2 &OEMR3Installation system End-user systems

18. The new design, rename rules (1) • Dsnames in the REF-set need to be renamed when they are copied to the Level Set. • Standard processing: rename removes the version release qualifier, i.e.REFN.T119A.OPS.LOAD REFN / SYSN dataset:SYSN.OPS.LOAD one qualifier droppedREF1.T113A.ZOS.LINKLIB REF1 / SYS1 dataset:SYS1.LINKLIB two qualifiers dropped

19. The new design, rename rules (2) The standard rename does not support two or more releases side by side. • Keep-Qualifier processing: rename keeps the version release qualifier, i.e.REFN.T119A.OPS.LOADSYSN.T119A.OPS.LOAD • Keep-Qualifier processing is never donefor SYS1 datasets:REF1.T113A.ZOS.LINKLIBSYS1.LINKLIB

20. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

21. The TDSL Product Catalog • TDSL application handles software deployment on the installation system. • TDSL Product Catalog describes all products and the datasets belonging to each product in SQL tables. • Each product (and version plus maintenance level) is identified by an 8-char acronym. • Product Catalog has attributes at the level of the product and at the level of the dataset. • Products can be duplicated in the Product Catalog to apply maintenance

22. The TDSL Product Catalog, screenshot

23. The TDSL Product Catalog, screenshot LR indicator Filter line provides a fast navigation facility More attributes on the right not in this screenshot Each product is identified by an 8-char acronym

24. The TDSL Dataset List screenshot

25. The TDSL Product Catalog, dataset attributes • Dataset Type:INSTALL product installation jcl SMP the CSI clusters, SMPPTS, SMPMTS, SMPLOG, etc.DLIB SMP dlibs, 100% inert for TDSLTARGET SMP/E target library or other distributable datasetVTARGET VSAM target datasetUTARGET z/fs linear, always keeps version-release qualifier for automount NODIST not distributed, but it contributes to the PROGxx member • APF dataset should be APF on end-user system • LNK dataset should be link-listed on end-user system.

26. The TDSL Product Catalog, dataset attributes, continued • DSTVOL:a symbolic volser like &SYSR1, &SYSR2 ... In the dataset list of the z/OS 1.13 A product: &SYSR1…REF1.T113A.ZOS.NUCLEUS &SYSR1REF1.T113A.ZOS.LINKLIB &SYSR1 &SYSR2REF1.T113A.ZOS.IMW.SIMWSDCK &SYSR2REF1.T113A.ZOS.OMEG.TKANMOD &SYSR3… &SYSR3Why fix the location of target datasets on the Level-set? - to ensure SYS1.NUCLEUS etc. on &SYSR1 volume &OEMR1- fewer catalog updates on the end-user system

27. The TDSL product catalog, screenshot of z/OS 1.13 dataset list

28. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

29. Non-uniform software stacks (1) swap validationproduction systems test systemsdevelopment production K-systems systemsSystems using the same software stack: EB systems EF systems SSE systems K systems Installation system

30. Non-uniform software stacks (2) • Why not deploy all software on all systems? Procurement dept. Licensing all software on all systems needlessly expensive.Audit dept. libraries in APF list for software products that will never be used. • Why not deploy software per-system?Hard to maintain the multitude of parmlib members (i.e. PROGxx members)Many different combinations to validate before a system upgrade. • A solution in-between is needed. All software will be present on all systems, but we make the libraries APF & linklisted only on those systems where it is licensed. To do this we add a where-used attribute to each product in the product catalog.

31. Non-uniform software stacks (3) 7 flags attributed to each product, each flag is on (used in this region) or off (not used in region)SSE PROGEGEF PROGEF the libraries of product CORT660A will be included inEP PROGEP PROGEG, PROGEF, PROGEP, PROGEN and PROGEBENL PROGEN but not in PROGEK or PROGES K PROGEKBSOF PROGEBSY90 PROGES

32. Creating PROG members Why create PROG members with a program? • Software installers know what datasets need to be APF or Link-listed, it’s in the product installation instructions -> can be entered in TDSL. • We keep lists of datasets we don’t distribute, but that need to be in the PROG member. • TDSL has al the info it needs to create APF lists and Link lists for the End-User systems.

33. Creating PROG members (2)

34. Creating PROG members (3)

35. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

36. TDSL Generate steps • Choose software products to go onto a Level-Set. • An SQL query finds all unique DSTVOLs • Engineer specifies the ‘real’ volsers and device numbers for each symbolic DSTVOLs, i.e. &SYSR1 will be build on real volser L13A11 • The dialog produces jcl to build the LS with the selected software products. Only the TARGET datasets will be copied to the LS

37. The TDSL Generate, screenshot

38. The treatment of link list datasets, space allocation The problem: • ALL REF-Set libraries have secondary space so SMP APPLY won’t fail with abend x37. • On the Level-Set, Link list libraries should have zero secondary allocation, to prevent an extent being added when someone adds/updates a member. The solution: • DFDSS dataset copy will copy every library into a single extent. • Copy with ALLDATA & ALLEXCP will preserve the allocated space. • A homegrown utility clears the secondary allocation value in the VTOC of the Level-Set.

39. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

40. Roll-out on the end-user system • Each sysplex has an active and inactive IPL-set with associated master catalogs: ACTIVE ACTIVE INACTIVE INACTIVEMASTER CATALOG IPL-SET IPL-SET MASTER CATALOGIPCAA1IPCAB1 CATALOG.PLXP.IPCAA1.MASTER CATALOG.PLXP.IPCAB1.MASTER • Inactive IPL-set is formatted, ICKDSF INIT • New master catalog is delete/defined • A dump is restored onto the inactive IPL-set • REPRO NOMREGECAT of the active master catalog to the new one • Inactive master catalog is fixed for new- or changed datasets • IPL from the inactive IPL-set.

41. This slide just for the notes

42. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

43. The Challenges of VSAM datasets • VSAM datasets need to be cataloged for all processing.Interferece with the driving system when copying VSAM datasets w same name. • All IDCAMS processing requires a correct VSAM ‘back-pointer’ Catalog volume Volume with VSAM datasetforward pointerICF catalog VVDS SYSV.OMVS.ROOT.ZFS VVR back pointer VSAM dataset SYSV.OMVS.ROOT.ZFS

44. The distribution of VSAM datasets (1) On the installation system: • Define a new ICF usercatalog on one of the LS volumes • Define an alias for a new temporary HLQ with RELATE to the new usercat, i.e. SYSDEF • Copy the VSAM datasets to the LS with RENAMEUNCONDITIONAL, REFx  SYSDEF • Level-set is dumped to tape. On the end-user system: • Define a new ICF usercatalog on one of the IPLset volumes • Delete/define the SYSDEF alias • Restore VSAM clusters with the SYSDEF HLQ from tape (no rename) • IDCAMS ALTER all SYSDEF dsnames in the user catalog to the normal HLQ for VSAM

45. The distribution of VSAM datasets (2) • An alias with symbolic relate accomplishes the same ‘volume switching function’ as the symbolic volumes in the catalog do for non-vsam datasets: DEFINE USERCATALOG – (NAME(CATALOG.SYSV.IPCAA1) – IPL-set A ICFCATALOG – VOLUME(IPCAA2) – CYLINDERS(1 1)DEFINE USERCATALOG - (NAME(CATALOG.SYSV.IPCAB1) – IPL-set B ICFCATALOG – VOLUME(IPCAB2) – CYLINDERS(1 1) DEFINE ALIAS(NAME(SYSV) –SYMBOLICRELATE(CATALOG.SYSV.&SYSR1))

46. The distribution of VSAM datasets (3) • LISTCAT ENTRY(SYSV) ALIAS ALL ALIAS --------- SYSV IN-CAT --- CATALOG.PLXP.IPCAA1.MASTER HISTORY RELEASE----------------2 ASSOCIATIONS SYMBOLIC-CATALOG.SYSV.&SYSR1RESOLVED-CATALOG.SYSV.IPCAA1

47. Agenda • Terminology • The old situation • The new design • The Product Catalog in TDSL • Non-uniform software stacks • TDSL Product Selection & Generate (build) processes • The roll-out (restore) process • VSAM challenges • zFS on the installation system

48. z/FS file systems on the installation system (1) Risks of APPLY on statically mounted file systems: • If the wrong file system is mounted, an APPLY job could update USS components belonging to a different SMP/E zone than the OS libraries. • Potentially there could be data not hardened in the file system when a R/W mounted file system is copied with dfDSS.How automounted file systems can help: • USS path in dddef’s can contain a zFS qualifier that makes automount mount the file system corresponding to that zone. • Automount will unmount file systems that have not been accessed for some amount of time. This reduces the time window during which uncommitted data can exist.

49. z/FS file systems on the installation system (2) /etc/auto.master:/SERVICE/DB2/DB2BASE /etc/service_db2_db2base.map/etc/service_db2_db2base.mapname * type ZFS filesystem REFN.<uc_name>.DB2.DB2BASE.ZFSmode rdwr duration 60 delay 10 security Yes setuid no DDDEF SDSNABIN in target zone TD1010A contains USS path:'/SERVICE/DB2/DB2BASE/T10101A/bin/IBM/'

50. z/FS file systems on the installation system (3) The SMP/E APPLY job: //SMPCNTL DD * SET BDY(TD1010A) . APPLY SELECT(…) . When SMP/E opens a file in the SDSNABIN directory it will cause the automount daemon to mount REFN.T1010A.DB2.DB2BASE.ZFS onto mount point /SERVICE/DB2/DB2BASE/T10101AAutomount replaces <uc_name> in the map file with the path-qualifier from the SMP/E dddef path to obtain the zFS dsname:i.e. Filesystem REFN.<uc_name>.DB2.DB2BASE.ZFSbecomes REFN.T1010A.DB2.DB2BASE.ZFS