820 likes | 1.27k Views
Session Number: C49. CICS VSAM Transparency. Satish Tanna Satish_Tanna@uk.ibm.com. AGENDA What is CICS VT? Product highlights Components Migration process summary Other enhancements in CICS VT 1.2 Operational enhancements Performance enhancements Supported software levels.
E N D
Session Number: C49 CICS VSAM Transparency Satish Tanna Satish_Tanna@uk.ibm.com
AGENDA • What is CICS VT? • Product highlights • Components • Migration process summary • Other enhancements in CICS VT 1.2 • Operational enhancements • Performance enhancements • Supported software levels
WHAT EXACTLY IS CICS VT? • Tool to migrate VSAM files to DB2 without changing application programs • Legacy programs access DB2 data using driver modules CICS VT generates for each migrated data set • Migrated data can be accessed by SQL in new programs • Existing programs can be enhanced using SQL • Lowest risk migration strategy
WHAT EXACTLY IS CICS VT (continued)? • Before CICS VT WRITE, REWRITE Application Program VSAM File #1 VSAM File #2 READ
WHAT EXACTLY IS CICS VT (continued)? DB2 Table #1 • After CICS VT INSERT, UPDATE SELECT C I C S V T Application Program VSAM File #1 GET, READ VSAM File #2 WRITE, REWRITE Application programs unchanged
CICS VT HIGHLIGHTS • Completely separate from application program • Transparent access to data in DB2 • 100% static SQL • Migrate on a file by file basis • Single live copy of data • Dual mode available for test purposes • Data can be re-engineered • Same data returned to VSAM programs • Enhanced data available using SQL
CICS VT COMPONENTS • Mapping component • Establishes relationship between VSAM record layout and DB2 row • Data migration component • Utilities to migrate data to DB2 and re-engineer if required • Testing component • Enables application programs to be tested following data migration • Run time component • Intercepts application calls to VSAM data sets migrated to DB2
MAPPING COMPONENT “ The relationship between the record in VSAM and the row in DB2” • One time activity for each data set and alternate index • Manual and automated mapping methods • Automated method should handle at least 70% of all data sets • Run-time driver module generation • Performed by DB2 DBA and application programmer • Includes design of DB2 objects
MAPPING COMPONENT MAPPING DATA in DB2 ISPF VSAM RUN-TIME DRIVERS COPYBOOK or DSECT CREATE TABLE VSAM_TAB ( TAB_KEY DEC(5, 0) ,ORDER_NO CHAR(3) ,ORDER_TYPE INTEGER ,NEXT_FIELD DEC(3,0) PRIMARY KEY(TAB_KEY)) 01 VSAM-file. 02 file-key pic 9999 02 group-field. 03 order-number pic x(3) 03 order-type pic s9(8) comp 02 next-field pic s999 comp-3 ……… ……… Usually one to one, can be one to many
TYPICAL DB2 TABLE DESIGN - 1 • One VSAM file becomes one DB2 table • Each copybook field becomes a DB2 column • 01 ITEM-DET. • 02 ITEM-NUMBER PIC X(6). • 02 ITEM-NAME PIC X(12). • 02 ITEM-DESC. • 03 ITEM-COLOUR PIC X(6). • 03 ITEM-WEIGHT PIC X(4). • 03 ITEM-COST PIC S99999V99 COMP-3. • 02 ITEM-REORDER-QUANTITY PIC S999. • 02 ITEM-SUPPLIER-CODE PIC X(3). • 02 ITEM-DATE-FIRST-SHIP PIC S9(9) COMP-3. • 02 ITEM-SHELF-LIFE PIC X(2). • 02 ITEM-DESCRIPTION PIC X(35) .
TYPICAL DB2 TABLE DESIGN - 2 CREATE TABLE VID_ITEM (ITEM_NUMBER CHAR(6) ,ITEM_NAME CHAR(12) ,ITEM_COLOUR CHAR(6) ,ITEM_WEIGHT CHAR(4) ,ITEM_COST DEC(7,2) ,ITEM_REORDER_NO SMALLINT ,ITEM_SUPP_CODE CHAR(3) ,ITEM_DATE_FSHIPDATE ,ITEM_SHELF_LIFE CHAR(2) ,ITEM_DESCRIPTION CHAR(35) 02 ITEM-NUMBER PIC X(6). 02 ITEM-NAME PIC X(12). 03 ITEM-COLOUR PIC X(6). 03 ITEM-WEIGHT PIC X(4). 03 ITEM-COST PIC S99999V99 COMP-3. 02 ITEM-REORDER-QUANTITY PIC S999. 02 ITEM-SUPPLIER-CODE PIC X(3). 02 ITEM-DATE-FIRST-SHIP PIC S9(9) COMP-3. 02 ITEM-SHELF-LIFE PIC X(2). 02 ITEM-DESCRIPTION PIC X(35) . • Numeric fields re-engineered • Date field re-engineered
MANUAL VERSUS AUTOMATED MAPPING • Manual Mapping • User inserts information for each copybook field • User creates DDL • Additional mapping for alternate indexes • Automated Mapping • Mapping and DDL generated by CICS VT • Capability to edit column names and attributes • Automated alternate index mapping
AUTO-MAPPING REQUIREMENTS • There is a valid copybook or DSECT for a data set • Copybook is written in ASM, Cobol or PL/I and will assemble/compile • There are no COPY statements in the copybook • The data set will be migrated to a single table • There is a maximum of 5 levels of nested redefines • The maximum OCCURS clause is 99 • MACRO and MEND statements must be removed from DSECTS • User is responsible for copybook/DSECT “quality”
AUTO-MAPPING PROCESS • Step 1 - Batch utility gathers base cluster information • Step 2 - User specifies copybook or DSECT for file • Step 3 - Optionally edit generated columns • Names and attributes • REDEFINES • Step 4 - JCL generated and submitted • Step 5 - driver modules generated
MANUAL MAPPING • When data set will be mapped to multiple tables using exits • Multiple record types • Repeating groups of may fields • To introduce normalisation • DDL created by user • Data set mapped by user • Alternate indexes mapped by user • Batch mapping utility
DATA MIGRATION COMPONENT • Three steps in data migration process 1. Unload existing data from VSAM data set 2. Convert data to DB2 format 3. Load DB2 data (using DB2 LOAD utility) • CICS VT utilities provided for 1 and 2 • Optionally identifies invalid numeric fields that are mapped to numeric column types (which would cause data conversion errors)
STEP 1 - UNLOAD VSAM DATA SET • Supplied batch utility program VIDUNLOD //VIDUNLOD EXEC PGM=VIDUNLOD //STEPLIB DD DSN=VID.VIDLOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //FILEIN DD DSN=vsam.file,DISP=SHR //FILEOUT DD DSN=vsam.unload,UNIT=SYSDA, // DISP=(,CATLG,DELETE),RECFM=FB,LRECL=108, // SPACE=(CYL,(25,5),RLSE) • Creates sequential data set • Must be RECFM=FB • LRECL = longest record length plus 8 bytes for KSDS and 12 bytes for RRDS
STEP 2 - REFORMAT VSAM DATA - 1 • Supplied batch utility program VIDLOAD //VIDLOAD EXEC PGM=VIDLOAD,PARM=dimname //STEPLIB DD DSN=VID.DRIVERS.LOAD,DISP=SHR // DD DSN=VID,LOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //LOADIN DD DSN=vsam.unload,DISP=SHR //LOADOUT DD DSN=vsam.db2load,UNIT=SYSDA, // DISP=(,CATLG,DELETE),RECFM=FB,LRECL=100, // SPACE=(CYL,(25,5),RLSE) • Creates reformatted sequential data set ready for DB2 load • Must be RECFM=FB • LRECL=length of longest DB2 record • CICS VT invoked to perform data re-engineering
STEP 3 - LOAD DATA INTO DB2 • Uses standard DB2 LOAD utility (the most efficient way to load records into a DB2 table) • LOAD utility control cards generated by DB2 sample program DSNTIAUL • Will require WHEN statements if migrating one file to multiple DB2 tables
REFORMAT VSAM DATA - 2 • If file is mapped to multiple tables, DD name for output data is different. //VIDLOAD EXEC PGM=VIDLOAD,PARM=dimname //STEPLIB DD DSN=VID.DRIVERS.LOAD,DISP=SHR // DD DSN=VID,LOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //LOADIN DD DSN=vsam.unload,DISP=SHR //LOADOUTM DD DSN=vsam.db2load,UNIT=SYSDA, // DISP=(,CATLG,DELETE),RECFM=FB,LRECL=100, // SPACE=(CYL,(25,5),RLSE) • Relies on code in IRD exit to identify appropriate record type
REFORMAT VSAM DATA - 3 Example: FILE X has 8-byte key and contains 3 record types. Sub-field of key indicates record type. • Implemented as 4 tables in CICS VT • One FBE exit and one IRD exit to manage the data • Table 1 is mapped to CICS VT and contains the key columns • Table 2 contains type #1 records and is accessed by the exits • Table 3 contains type #2 records and is accessed by the exits • Table 4 contains type #3 records and is accessed by the exits • IRD exit must set table indicator in first 2 bytes of output record
REFORMAT VSAM DATA - 4 Data produced by VIDLOAD with LOADOUTM KEY 0001000013 0101000013DATA1000000000000041DATA1000000000000042DATA1000000000000043 0002000020 0202000020DATA20000000011DATA20000000012DATA10000000013DATA20000000014 0002000021 0202000021DATA20000000021DATA20000000022DATA10000000023DATA20000000024 0002000022 0202000022DATA20000000031DATA20000000032DATA10000000033DATA20000000034 0003000030 0303000030DATA30000011DATA30000012DATA30000013DATA30000014DATA30000015 0003000031 0303000031DATA30000021DATA30000022DATA30000023DATA30000024DATA30000025 LOAD DATA REPLACE INDDN SYSREC00 INTO TABLE TABLE_02 WHEN (1:2) = '02'
DATA MIGRATION - THE REALITY • Potentially old legacy files with high probability of inconsistent data • Numeric field values with BLANKS, binary zeroes • Inconsistent date values (how about a value of ASAP?) • VIDLOAD can be used to identify invalid numeric values • DB2 LOAD will fail if it encounters invalid column values • Potential to use exits for verification
IDENTIFY INCONSISTENT DATA • Additional DD statement added to VIDLOAD //VIDLOAD EXEC PGM=VIDLOAD,PARM=dimname //STEPLIB DD DSN=VID.DRIVERS.LOAD,DISP=SHR // DD DSN=VID,LOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //LOADIN DD DSN=vsam.unload,DISP=SHR //LOADOUT DD DUMMY //VIDS0C7 DD SYSOUT=* All invalid numeric fields identified VID DATA EXCEPTION REPORT • RECORD 0000000102 OFFSET 0029 DATASET VIDKSDS • FIELD NAME VIDF004 LENGTH 0004 VALUE xxxx • RECORD 0000000356 OFFSET 0063 DATASET VIDKSDS • FIELD NAME VIDF007 LENGTH 0003 VALUE yyy • FBE exit might be required for recurring fields
INCONSISTENT DATA - THE OPTIONS • Invalid data • Perform data cleansing exercise • Map inconsistent fields to CHAR columns, correct invalid data and remap to numeric columns and re-migrate • Perform data validation/correction using CICS VT exits • Correct data during migration • May have to identify and correct the application programs that are creating invalid data
TESTING APPLICATIONS USING DMF • Dual mode facility (DMF) - major new V1.2 feature • Application program calls to migrated data sets are processed in DB2 and in VSAM • VT automatically compares record areas and call status areas • Any difference is highlighted • Will significantly reduce application program test • Two main purposes: • Verify initial migration • Test user exits
TESTING APPLICATIONS USING DMF • Using DMF to verify initial data migration • The VT utility VIDUNLOD unloads a VSAM file to a sequential data set • The VT utility VIDLOAD converts sequential data set into a DB2 load file according to the mapping and user exits • The DB2 LOAD utility loads the DB2 table(s) • CICS VT V1.1 • User runs VIDUNLOD against the migrated data in DB2 • Compares unload data sets from VSAM and DB2 • CICS VT V1.2: • User runs VIDUNLOD enabled for DMF • CICS VT automatically compares data
TESTING APPLICATIONS USING DMF • VIDUNLOD JCL with and without DMF //VIDUNLOD EXEC PGM=VIDUNLOD //STEPLIB DD DSN=VID.VIDLOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //FILEIN DD SUBSYS=(ssi,db2id,dimname) //FILEOUT DD DSN=vt.unload,UNIT=SYSDA, // DISP=(,CATLG,DELETE),RECFM=FB,LRECL=108, // SPACE=(CYL,(25,5),RLSE) //VIDUNLOD EXEC PGM=VIDUNLOD //STEPLIB DD DSN=VID.VIDLOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //FILEIN DD SUBSYS=(ssi,db2id,dimname,VSAMDD) //FILEOUT DD DUMMY,LRECL=108,RECFM=FB //VSAMDD DD DSN=original-VSAM-file,DISP=SHR • Any DD name can be specified
TESTING APPLICATIONS USING DMF • Using DMF for batch application testing • Add 4th parameter to SUBSYS DD statement • Add DD statement for original VSAM data set //RTP050 EXEC PGM=appl-prog //STEPLIB DD DSN=VID.VIDLOAD,DISP=SHR //SYSPRINT DD SYSOUT=* //ORDERFL DD SUBSYS=(ssi,db2id,dimname,ORDERFLV) //ORDERFL@ DD DSN=hlq.ORDERFL.DUMMY,DISP=SHR //ORDERFLV DD DSN=hlq.ORDERFL,DISP=SHR • Any difference detected by DMF causes program abend • U3139 indicates a data difference • U3141 indicates a return code or reason code difference
TESTING APPLICATIONS USING DMF • DMF U3139 abend VIDSS227 DUALMODE ERROR FOR SUBSYS=VIDZ VSAM DDNAME=ENTITLV RECORD DATA WAS NOT THE SAME AT OFFSET X"0036" VIDSS219 DUMP OF DB2 RECORD: VIDSS220 +0016 F0F0F0F0 F0F0F0F4 F1E4F4F1 E5F4F100 *000000041U41V41.* VIDSS220 +0026 00004040 40404040 40404040 40404040 *.. * VIDSS220 +0036 40404040 40404040 40404040 40404040 * * VIDSS220 +0046 40404040 40404040 40404040 40404040 * * VIDSS219 DUMP OF VSAM RECORD: VIDSS220 +0016 F0F0F0F0 F0F0F0F4 F1E4F4F1 E5F4F100 *000000041U41V41.* VIDSS220 +0026 00004040 40404040 40404040 40404040 *.. * VIDSS220 +0036 C9C9C9D5 D5D5C6C6 C6404040 40404040 *IIINNNFFF * VIDSS220 +0046 40404040 40404040 40404040 40404040 * * • Hex offset of first byte identified in error message • X’20’ bytes before and after is displayed
TESTING APPLICATIONS USING DMF • DMF U3141 abend VIDSS228 DUALMODE ERROR FOR SUBSYS=VIDZ VSAM DDNAME=ENTITLV FEEDBACK DIFFERENT ON PUT DB2 RPLRTNCD=00 RPLERRCD=00 VSAM RPLRTNCD=08 RPLERRCD=08 ACB MACRF=(ADR,SEQ,DIR,IN) RPL OPTCD=(SEQ,KEY) KEYL=0025 • Call parameter list is displayed • Non-significant VSAM fields such as component code are ignored by DMF • Most errors will be a result of unsynchronised VSAM and DB2 data
TESTING APPLICATIONS USING DMF • Using DMF for CICS application testing • New file definition for VSAM data set must be added to CICS • File name is derived from original file name • Two new DMF parameters in the VT data set table (DST) • DMF is activated/deactivated with new VT transaction VTMA • 2 new transaction abend codes for DMF • 3139 for data differences • 3141 for RESP code differences • Errors written to new DD statement VIDCOMP
TESTING APPLICATIONS USING DMF • Enabling DMF in CICS • Add new CICS file definition for VSAM data set File name is derived from original file name • Add ‘@’ if original file name less than 7 characters • Replace 1st character with ‘@’ if file name is 8 characters • Two new DMF parameters in the VT data set table (DST) • ERRORACT=ABEND|REPORT • DUMP=YES|NO (only relevant if ERRORACT=ABEND) • Issue VTMA START +VIDCINIC - CICS VT DMF successfully initialised.
TESTING APPLICATIONS USING DMF • Using the VTMA transaction • VTMA STATUS • VTMA START • VTMA STOP • VTMA RESTART • Optionally add VIDCINIC program to PLT to activate DMF at CICS start up if required
TESTING APPLICATIONS USING DMF • CEMT I FI(*) STATUS: RESULTS - OVERTYPE TO MODIFY Fil(VIDKSDS ) Vsa Ope Ena Rea Upd Add Bro Del Sha Dsn( VID.VIDKSDS.DUMMY ) Fil(VIDKSDS@) Vsa Ope Ena Rea Upd Add Bro Del Sha Dsn( VID.VIDKSDS ) Fil(ORDERFIL) Vsa Ope Ena Rea Upd Add Bro Del Sha Dsn( VID.ORDERFIL.DUMMY ) Fil(@RDERFIL Vsa Clo Ena Rea Upd Add Bro Del Sha Dsn( VID.ORDERFIL ) Fil(VIDRRDS) Vsa Ope Ena Rea Upd Add Bro Del Sha Dsn( VID.VIDRRDS.DUMMY ) • Files VIDKSDS and ORDERFIL will be processed by DMF
TESTING APPLICATIONS USING DMF • VTMD transaction 03/21/05 ==> CICS VT File Definition List <== 10:32:57 Filename DIM name Org Status Operations Compare DFHCMACD DFHCMACD ENA INI REA BRO DEL REP N VIDKSDS VIDKSDS KSDS UNE REA UPD ADD BRO DEL REP N KSDS01 KSDS01 ENA INI REA UPD ADD BRO DEL REP N RRDS01 RRDS01 ENA INI REA UPD ADD BRO DEL ABEND N ORDERFIL ORDERFL ENA INI REA UPD ADD BRO DEL ABEND Y TVDP01 TVDP01 ENA INI REA UPD ADD BRO DEL ABEND Y End of File table
TESTING APPLICATIONS USING DMF • 3139 transaction abend – data is different ********************************************************************* * CICS VT COMPARE EXCEPTION * ********************************************************************* TRAN: K1P4 DIM: KSDS01 TERMINAL: 0019 TIME: 15.49.5300 REQ: 0000026 EXEC CICS READNEXT FILE(KSDS01 ) RIDFLD= 000 F0F0F0 RECORD AREA EXCEPTION @ OFFSET 00000017 -....5....0....5...+ CICS VT .ORIGINAL.DATA.FOR.K 40D6D9C9C7C9D5C1D340C4C1E3C140C6D6D940D2 VSAM .ORIGINAL.BAD.DATA.F 40D6D9C9C7C9D5C1D340C2C1C440C4C1E3C140C6 • Offset is hex value – 10 bytes of data either side is dumped
TESTING APPLICATIONS USING DMF • 3141 transaction abend – return codes are different ********************************************************************* * CICS VT COMPARE EXCEPTION * ********************************************************************* TRAN: K1P2 DIM: KSDS01 TERMINAL: 0019 TIME:15.31.3300 REQ:0000009 EXEC CICS READ FILE(KSDS01 ) RIDFLD= YYYYYY E8E8E8E8E8E8 EQUAL UPDATE RETURN CODE EXCEPTION RESP RESP2 RCODE CICS VT 0000000D 00000050 810000000000 VSAM 00000000 00000000 000000000000 Hex return code - NOTFND
TESTING APPLICATIONS USING DMF • DMF checking varies by call • Data areas, return and reason codes for READ/GET/BROWSE calls • Return and reason codes for WRITE/REWRITE/PUT calls • Return and reason codes for ERASE calls. Count of records deleted is also checked • Data MUST always be synchronised or results are unreliable • Especially important in batch programs – an abend will leave VSAM data partially updated, but updates will be automatically backed out in DB2
TESTING APPLICATIONS USING DMF • Can DMF be used in production? • If ERRORACT=ABEND then the transaction will fail • If ERRORACT=REPORT transaction won’t fail but data may be out of line, which will cause additional problems • What happens if both versions of the file (DB2 and VSAM) are not available? • There is a performance overhead • There is no scope to exploit 24x7 or online DB2 utilities • You can’t run mixed online and batch workloads
DATA MIGRATION PROCESS SUMMARY ANALYSIS • Data cleansing required • Exits for data validation/correction • IF SPACES test may dictate column attributes MAP MIGRATE TEST CUTOVER Data inconsistencies
RUN-TIME COMPONENT • Application programs continue to issue calls to VSAM files • CICS VT intercepts call • Using a global user exit (GLUE) in CICS • Using an MVS sub-system for batch programs • Translates call to equivalent SQL call and processes in DB2 • After call has been processed by CICS VT • Data and RESP codes return to CICS program • Data and return and reason codes passed back to batch program
RUN TIME CALL INTERCEPTION - CICS CICS VT TRUE DB2 C I C S V T CICS VT GLUE Application Program VSAM • GLUE performs DST look up
CICS VT DST - 1 • CICS table containing a list of migrated data sets • Macros based on CICS FCT VIDCDTAB VIDTAB TYPE=INITIAL • VIDTAB TYPE=ENTRY, • FILE=VIDKSDS, • DIM=VIDKSDS, • STATUS=ENA, • OPEN=S, • ADD=YES, • BRO=YES, • DEL=YES, • REA=YES, • UPD=YES, • RESET=YES, • RECORDF=FIX • VIDTAB TYPE=FINAL
CICS VT DST - 2 • To add an entry for a migrated data set • Define DIM and DDM drivers, plus user exits to CICS • Add data set information to table • Assemble new table to DFHRPL library • Restart the CICS VT interface VTMI RESTART • Data set available for processing by CICS VT • CICS VT installed and runs in AOR • Remote file requests no longer function-shipped to FOR
DEFINING DRIVERS TO CICS VT • DIM and DDM drivers must be defined to CICS • Drivers are assembler programs • Re-entrant in CICS VT 1.2 • Must be defined RELOAD NO • Can be defined RESIDENT if required • Can be auto-installed (assuming default is RELOAD NO) • DDM driver size reduced in CICS VT 1.2 • V1.1 drivers are not compatible with V1.2 and vice versa • Must be re-generated • CICS VT user exits RELOAD=NO
CICS VT - ADDITIONAL TRANSACTIONS • To control the CICS - CICS VT interface • VTMI with either STOP, START (or RESTART) STOP disables CICS VT GLUE and effectively CICS VT START enables the CICS VT GLUE and activates call interception • Useful control for testing MUST BE AUTHORISED IN PRODUCTION
RUN TIME CALL INTERCEPTION - BATCH 1 • MVS subsystem used for batch program call interception • SSI defined at CICS VT installation • Dynamic method - no IPL required • Permanent method - IPL required • Default CICS VT ssid = VIDS • JCL changes required to enable call interception
RUN TIME CALL INTERCEPTION - BATCH 2 • JCL changes Before: //ORDERFL DD DSN=ORDER.VSAM.DATASET,DISP=SHR //ORDERFL1 DD DSN=ORDER.VSAM.AIX,DISP=SHR After: //ORDERFL DD SUBSYS=(VIDS,ssid,DIM-name) //ORDERFL1 DD SUBSYS=(VIDS,ssid,DIM-name) • VIDS= CICS VT SSI, ssid= DB2 subsystem • DD statements for base cluster and AIX paths
RUN TIME CALL INTERCEPTION - BATCH 3 • Cobol and PL/1 programs use undocumented interface to SHOWCB macro during initialization • CICS VT uses dummy VSAM data set to achieve same • User must create dummy data set with same attributes as migrated data and add DDNAME to batch JCL • Before: • //ORDERFL DD DSN=ORDER.VSAM.DATASET,DISP=SHR • After: • //ORDERFL DD SUBSYS=(VIDS,ssid,DIM-name) • //ORDERFL@ DD DSN=ORDER.DUMMY.VSAM.DATASET,DISP=SHR