660 likes | 904 Views
MULTI-FETCH IN AND OUT OF NATURAL. Jim Poole. CONTENTS. WHAT’S NOT COVERED OVERALL SYNTAX HOW IT WORKS NATURAL vs ADAMLF UPDATING HOW MUCH TO MULTI-FETCH CONCERNS, ISSUES. WHAT’S NOT COVERED:. PREFETCH. ADAMLF Exclude: PREFXFIL/ PREFXCMD. HISTOGRAM L9’s. ADAMLF: NOT USING
E N D
MULTI-FETCHIN AND OUTOF NATURAL Jim Poole
CONTENTS • WHAT’S NOT COVERED • OVERALL • SYNTAX • HOW IT WORKS • NATURAL vs ADAMLF • UPDATING • HOW MUCH TO MULTI-FETCH • CONCERNS, ISSUES
WHAT’S NOT COVERED: PREFETCH ADAMLF Exclude: PREFXFIL/ PREFXCMD HISTOGRAM L9’s ADAMLF: NOT USING PREFNREC DIRECT CALLS PROGRAMS
PROGRAM PROGRAM ADABAS ADABAS REGULAR PROCESSING MULTI-FETCH PROCESSING READ FILE : : : END-READ READ FILE : : : END-READ CQ THREAD CQ THREAD *COUNTER=10,000 10,000 L3’S 10,000 CALLS TO ADABAS 10,000 CQ TO THREAD 10,000 RB’S FILLED 10,000 CALLS RETURNED *COUNTER=10,000 PREFNREC/MFF=100 100 L3’S 100 CALLS TO ADABAS 100 CQ TO THREAD 100 RB’S FILLED (WITH 100 RECORDS EACH) 100 CALLS RETURNED “?WHAT IS MULTI-FETCH ?” OVERALL RETURN MULTIPLE RECORDS ON ONE ADABAS CALL/CMD
2. BATCH/ONLINE NATURAL V4 READ/FIND MULTI-FETCH #MFF : : : ADA LINK “TWO APPROACHES TO MULTI-FETCH” OVERALL 1. BATCH PROGRAM ADAMLF: ADARUN PREF*= : : : ADALNK NATMLF ADAMLF NOTE: PGM MLF HAS PRECEDENCE
QUICK SYNTAX: ADAMLF JCL: //DDCARD (ALL PREFACED BY ADARUN) • PREFETCH=YES / NO • PREFSBL=130000 * MLF BUFFER SIZE/SEQUENCE • PREFTBL=n*PREFSBL * n=# MLF SEQUENCES) • PREFNREC= * MAX RECS TO MLF • * Code ICMD/IFIL pairs for cmd/fnr to MLF • * ICMD cmd’s separated by ‘/’ • * IFIL fnr’s separated by ‘,’ • PREFICMD=cmd/cmd/cmd … • PREFIFIL=fnr,fnr,fnr … !! ALWAYS CODE PREFNREC= !!
PREFETCH=YES,PREFSBL=130000,PREFTBL=390000 PREFNREC=100 PREFICMD=L2/L3/L3 PREFIFIL=22,18,16 QUICK “3 MLF SEQUENCES: L2/L5 FOR FNR 22 L3/L6 FOR FNR 18 L3/L6 FOR FNR 16” SYNTAX: ADAMLF EXAMPLES PREFETCH=YES,PREFSBL=130000,PREFTBL=130000 PREFNREC=20,PREFICMD=L2,PREFIFIL=241 “FNR 241, ALL L2/L5 WILL MLF UP TO 20 RECORDS PER L2/L5”
FOR EXAMPLE: 1 #MFF (I4)INIT <020> /*RECS/DB ACCESS FOR MLF : : : RD1. READ MULTI-FETCH #MFF DA-FILE PHYSICAL QUICK “FOR FILE DA-FILE, 20 RECORDS WILL BE RETURNED ON EACH DATABASE CALL” SYNTAX: NATMLF AVAILABLE WITH NATURAL 4 READ/FIND MULTI-FETCH factor view-name . . .
SYNTAX: NATMLF RULES 1. READ LOGICAL. WORKS ON BOTH ASCENDING, DESCENDING. 2. DOES NOT WORK IF DYNAMIC SEQUENCE CHANGES: “. . . IN DYNAMIC . . . SEQUENCE . . . “ 3. ONLY WORKS FOR BROWSE LOOPS (NO RECORD HOLDS), ‘NO UPDATES’. 3a. IF ‘UPDATE,DELETE’ THEN MLF TURNED OFF (no error) 4. MLF BUFFER RELEASED WITH REPOSITIONING OR LOOP-END 5. MLF BUFFER RELEASED AT END OF LOOP (RC CMD)
SYNTAX: NATMLF MFF REDUCED BY • READ/FIND (NNN) . . . • IF MFF > NNN THEN NNN USED • IF MFF < NNN THEN MFF USED (extra records may be read) B. RECORD BUFFER LENGTH (32K). More later C. ISNQ (*NUMBER) FOR FIND STATEMENTS. More later
QUICK SYNTAX: NATMLF. MULFETCH BUFFER NTDS MULFETCH,nn Or GLOBALS DS=(MULFETCH,nn) nn = 0 – 1024 KILOBYTES OF MEMORY = 0 MULTI-FETCH DISABLED /= 0 (NATURAL 4.1) SIZE AS NEEDED (UP TO MAX BUFFER LENGTH)
IB: ISN BUFFER 16 BYTE HEADERS: 4B – LENGTH OF THIS RECORD IN RB 4B – ADABAS RESPONSE CODE* 4B – ISN OF THIS RECORD 4B – FOR L9, ISNQ *IF RSP.CODE /= 0 THEN NO RECORD STORED IN RB ADABAS NUCLEUS HOW IT WORKS. DIRECT CALLS CB: CONTROL BLOCK CMD = L2/L5, L3/L6 L1/L4 where COP2=I or N ILL = MAX RECS TO RETURN COP1=M or O CID = NON-ZERO,NON-BLANK FB: FORMAT BUFFER RB: RECORD BUFFER DECOMPRESSED RECORDS NOTE: MAX BUFFER LENGTH = 32,768 (ADABAS 7) NOTE: V8 ACBX USES MLF ABD vs IB FOR HEADERS
ADABAS CB: CMD=L3 ILL=10 COP1=M CID=X’02000101’ FB: AC,020,A,AE,020,A,AD,020,A. FBL=27 RBL=10x60=600 RB: REC1 REC2 REC3 . . . REC 10 HDR1 HDR2 HDR3 . . . HDR 10 IB: IBL=10x16=160 HOW IT WORKS. DIRECT CALLS,eg. 1 EMP VIEW OF EMPLOYEES 2 FIRST-NAME 2 NAME 2 MIDDLE-NAME : : : READ MULTI-FETCH 10 EMP BY NAME
MAX AS DETERMINED BY RB: RBL=32768 / rec.size REC SIZE MAX FOR MLF 60 546 200 163 2000 16 ADABAS V7 HOW IT WORKS. MAX #MFF, PREFNREC MAX AS DETERMINED BY IB: IBL=32768 / 16 = 2048
QUICK ADABAS NUCLEUS HOW IT WORKS. ET/BT CB: CONTROL BLOCK CMD = ET, BT COP1 = P ILL = COUNT OF 6b IB ENTRIES IB: ISN BUFFER 6 BYTE ENTRIES: 2B – FILE NUMBER 4B – ISNs TO BE PUT BACK ON HOLD NOTE: ONLY AVAILABLE FOR DIRECT CALL PROGRAMMING: COP1=M RELEASE ONLY ISN’S IN IB IB: 4b COUNT OF ELEMENTS n8b ELEMENTS (4b FNR, 4b ISN)
QUICK HOW IT WORKS. CA-APAS CB ISNLL = MAX MFF RECORDS REQUESTED CA-APAS ISN-LOWER-LIMIT ( ISNLL ) equals 0 ISNLL-REPORTED shows MFF requested ISN-QUANTITY ( ISNQ ) shows #RECS in RB Note: ISNQ may not equal ISNLL-REPORTED
PROGRAM CPU MFF= 0 10 25 50 100 1000 ADABAS CPU MFF= 0 10 25 50 100 1000 CPU COSTS OFMULTI-FETCH 5 RUNS EACH: MFF = 0 MFF = 10 MFF = 25 MFF = 50 MFF = 100 MFF = 1000 AVERAGES SHOWN NO ADABAS DELAY: NO IO, NO OTHER USERS NO PGM DELAY.
1ST MLF CALL ADABAS FILLS BUFFER 2nd MLF CALL ADABAS FILLS BUFFER 3rd MLF CALL ADABAS FILLS BUFFER AND SO ON . . . “NORMAL CALLS TO ADABAS” NATMLF PROCESSES ADABAS NUCLEUS PROGRAM ADALNK PROGRAM READS BUFFER SAVE ON PGM-TO-ADA TIME CQ-THREAD TIME SOME PGM TIME SOME ADA TIME PROGRAM READS BUFFER
1ST MLF CALL ADABAS FILLS BUFFER 2nd MLF CALL PROGRAM READS BUFFER ADABAS FILLS BUFFER 3rd MLF CALL ADABAS FILLS BUFFER PROGRAM READS BUFFER AND SO ON . . . “ASYNC CALLS TO ADABAS” ADAMLF PROCESSES ADABAS NUCLEUS PROGRAM ADAMLF ADALNK ½ PREFSBL ½ WHILE PROGRAM PROCESS BUFFER, ADABAS FILLS NEXT BUFFER
IDEAL PREFSBL FOR EXAMPLE: PREFSBL = 130,000 1/2 = 65K EACH IB > 32K EACH RB > 32K ! USE PREFNREC ! PREFSBL = 64K 1/2 = 32K EACH IB = 8K EACH RB = 24K ADAMLF PREFSBL USAGE PREFSBL 1 / 2 1 / 2 1 / 4 IB IB 3 / 4 RB RB
NOT USED IN NATMLF ADAMLF “WARM UP” DETERMINE RBL = ( PREFSBL / 2 ) X (3/4) MAX RECORDS RETURNED ON EACH MLF ADAMLF CALL 1ST CALL - ONE RECORD RETURNED (NO MLF) 2ND CALL - NUMBER OF RECORDS THAT CAN FIT IN 1/8 RBL 3RD CALL - NUMBER OF RECORDS THAT CAN FIT IN 1/8 RBL 4TH CALL - NUMBER OF RECORDS THAT CAN FIT IN 1/2 RBL 5TH CALL - NUMBER OF RECORDS THAT CAN FIT IN RBL : : : 5+n CALL - NUMBER OF RECORDS THAT CAN FIT IN RBL
CALL# IBL RBL ISNQ* 1 8,190 100 1 2 8,190 3,071 30 3 8,190 3,071 30 4 8,190 12,286 122 5 8,190 24,573 200 6 8,190 24,573 200 7 8,190 24,573 200 8 8,190 24,573 200 9 8,190 24,573 200 10 8,190 24,573 200 11 8,190 24,573 200 12 8,190 24,573 200 13 8,190 24,573 200 14 8,190 24,573 17 1 REC, NOTE RBL=100 1/8 RBL 1/8 RBL 1/2 RBL MAX REACHED EOF ADAMLF “WARM UP” Example PREFSBL = 65K, RBL = ( 65K / 2 ) X (3/4) = 24,573 PREFNREC = 200 *NOTE: CA-APAS ISNQ = RECS MLF’D
ADAMLF vs NATURAL MLF (overall) ADAMLFNATURAL/PROGRAM MLF 1. ASYNC PROCESSING SYNC PROCESSING 2. BATCH ONLY ONLINE OR BATCH 3. NO PROGRAM CHANGES CHANGED WITHIN PROGRAM 4. PREFNREC: FOR ALL SEQs VARY FOR @ ACCESS LOOP 5. “WARMS UP” WHAT YOU CODE, YOU GET 6. CAN DO UPDATES ! NO ! UPDATING 7. CONTROL VIA //DDCARD CONTROL WITHIN PROGRAM (PROCS, PDS MBRs, ETC.)
ADABAS RESPONSE (CPU,IO,OVERLAP) PGM PROCESS ADABAS CALLS * ( #RECS / MFF ) PATHLENGTH TO/FROM PGM-ADABAS PGM DELAY (CPU,IO,ETC.) + * ( #RECS ) ADAMLF vs NATURAL MLF (Performance) PERFORMANCE (PGM ELAPSED TIME) = • MULTI-FETCH SAVES ON: • PATHLENGTH • SOME ADABAS RESPONSE • SOME PGM TIME TO PROCESS CALLS ADAMLF IMPROVES AS PGM DELAY INCREASES
ADAMLF vs NATURAL MLF (Performance) • IN THE LAB: NATMLF vs ADAMLF • MFF=100 • 800,000 RECORDS • READ PHYSICAL • MEASURE: PGM ELAPSED TIME • PGM CPU TIME • ADABAS RESPONSE TIME (DURATION) • ADABAS RELATIVE CPU TIME (CMDTIME) • SCENARIOS: • NO DELAY • ADABAS: NO IO, NO OTHER USERS • NATURAL: READ LOOP ONLY • B. ADABAS DELAY (IO) • C. ADABAS/PGM DELAY (LOTSA MOVE STATEMENTS) • FOR EACH SCENARIO, MULTIPLE RUNS NATMLF,ADAMLF RESULTS
KEY: ELAPSED (NAT 12% Better) CPU (3% Diff) ADAMLF NATMLF ? CQ-TRHD DELAY CPU (NAT 56% Better) DURATION (NAT 39% Better) PROGRAM ADABAS A. NO DELAY ADAMLF vs NATURAL MLF (Performance)
KEY: ELAPSED (ADA 8% Better) CPU (ADA 5% Better ?) ADAMLF NATMLF CPU (NAT 54% Better) DURATION (No Diff) PROGRAM ADABAS B. ADABAS DELAY ADAMLF vs NATURAL MLF (Performance)
KEY: ELAPSED (ADA 22% Better) CPU (2% Diff) ADAMLF NATMLF CPU (2% Diff) DURATION (No Diff) PROGRAM ADABAS C. DELAY PGM/ADA ADAMLF vs NATURAL MLF (Performance)
ADAMLF. UPDATES. ET FOR EXAMPLE: REVIEW RD1. READ TEST PHYSICAL IF AM-FIELD = 'AA' THEN AM-FIELD := 'BB' ELSE AM-FIELD := 'AA' END-IF UPDATE (RD1.) ETCNT := ETCNT + 1 IF ETCNT = ETLIMIT END TRANSACTION ETCNT := 0 END-IF END-READ /*(RD1.) END TRANSACTION Lx CB COP1=‘O’ ET CB COP1=‘P’ IB = 6b fnr/isn to put back on HQ PREFNREC=100 PREFIFIL=241,PREFICMD=L2 RESULTS
ETLIMIT = 10 ADAMLF. UPDATES. ET CM COP1 ILL IBL L5 O 1 16,247 A1 0 0 L5 O 100 16,247 L5 O 100 16,247 9 A1'S ET P 191 1,146 10 A1'S ET P 181 1,086 10 A1'S : : : ET P 101 606 10 A1'S L5 O 100 16,247 : : : RC S 0 0 RI 0 16,247 RI 0 16,247 : : : 1ST CALL UPDATE 1ST RECORD 1ST MLF CALL, 100 RECs 2nd ASYNC MLF CALL, 100 RECs 9 MORE UPDATES 191 RECS BACK ON HOLD 10 MORE UPDATES 181 RECS BACK ON HOLD 3rd ASYNC MLF CALL, 100 RECs RECS ON HOLD AT LOOP END
ETLIMIT = PREFNREC = 100 ADAMLF. UPDATES. ET (NOTE A1s NOT ON REPORT) CM COP1 ILL IBL L5 O 1 16,247 L5 O 100 16,247 L5 O 100 16,247 ET P 0 606 L5 O 100 16,247 ET P 0 606 L5 O 100 16,247 ET P 0 606 L5 O 100 16,247 ET P 0 606 : : RC S 0 0 • EVERY ET PUTS 101 RECS BACK ON HOLD !! CANNOT SYNC ET’S WITH PREFNREC !! • EOF, NO RI’s MIN ADARUN NISNHQ=2 X PREFNREC
ET EVERY 1 A1 ET EVERY 10 A1 ET EVERY 100 A1 ADAMLF. UPDATES. ET COSTS ADABAS AVG CPU TIME / ET NREC = 0 25 50 100 200
ET EVERY 1 A1, 200K ET’S ET EVERY 10 A1, 20K ET’S ET EVERY 100 A1, 2K ET’S ADAMLF. UPDATES. ET COSTS ADABAS SUM CPU TIME ET’s NREC = 0 25 50 100 200 ET COSTS INCREASE AS PREFNREC INCREASES
ADAMLF. UPDATES. UPD/DELETE A1/E1 RECORD BUFFER: CURRENT RECORD RECORD +1 RECORD +2 RECORD +3 RECORD +4 RECORD +5 : : : FOR EVERY UPDATE PREFSBL BUFFER SCANNED TO SEE IF ISN IN “YET-TO-BE-READ” RECORDS ( NOT CURRENT RECORD ) INCREASE IN PROGRAM CPU/ELAPSED TIME NO INCREASE IN ADABAS TIMES (NEITHER CPU NOR DURATION) ? MEASURE CPU COSTS ? TEST PROGRAM
QUICK ADAMLF. UPDATES. A1/E1 COSTS DEFINE DATA LOCAL 1 #CPU-PGM-ALL (I4) INIT <0000> /* TIME IN 10 MS UNITS 1 #CPU-UPD-ALL (I4) INIT <0000> /* TIME IN 10 MS FOR UPDS 1 #CPU-UPD (I4) /* CPU TIME COUNTER 1 TEST VIEW OF SOME-FILE : : : END-DEFINE ST. SETTIME ASSIGN #CPU-PGM-ALL = *CPU-TIME RD1. READ TEST PHYSICAL : : : ASSIGN #CPU-UPD = *CPU-TIME /* UPDATE (RD1.) ASSIGN #CPU-UPD = *CPU-TIME - #CPU-UPD ASSIGN #CPU-UPD-ALL = #CPU-UPD-ALL + #CPU-UPD : : : END-READ /*(RD1.) ASSIGN #CPU-PGM-ALL = *CPU-TIME - #CPU-PGM-ALL WRITE 'DA-TIME' *TIME *TIMD(ST.) (EM=99:99:99'.'9) WRITE 'DA-PGM-CPU' #CPU-PGM-ALL WRITE 'DA-UPD-CPU' #CPU-UPD-ALL VARY PREFNREC= RESULTS
ADAMLF. UPDATES. A1/E1 COSTS SUM PROGRAM CPU TIMEs TOTAL TOTAL TOTAL TOTAL A1 A1 A1 A1 ET ET ET ET NREC: ----- 0 ----- ---- 100 ---- ---- 400 ---- ---- 800 ---- UPDATE COSTS INCREASE AS PREFNREC INCREASES
ADAMLF. UPDATES. A1/E1 VIA REREAD FOR EVERY READ, REREAD (L4) THEN UPDATE NO INCREASE IN ET TIMES NO INCREASE IN ADABAS TIMES MAJOR INCREASE IN PROGRAM TIMES MEASURE CPU COSTS RESULTS
ADAMLF. UPDATES. A1/E1 VIA REREAD SUM PROGRAM CPU TIMEs TOTAL TOTAL TOTAL TOTAL A1 A1 A1 A1 NREC: ----- 0 ----- ---- 100 ---- ---- 400 ---- ---- 800 ---- !! DON’T REREAD RECORD AND UPDATE !!
FNR CM MFF COUNT --- -- ---- ------ 0 CL 0 1 0 ET 0 200 241 A1 0 19,980 241 L1 0 84 241 L3 800 28 241 L4 0 19,980 NOTE ADAMLF. UPDATES. OUTSIDE OF LOOP RB: CURRENT RECORD FOR EXAMPLE: 1. MLF READ 2. UPDATE/DELETE YET-TO-BE-READ-RECORD 3. IF THAT RECORD IS IN PREFSBL, IT WILL BE REREAD BY ADAMLF (L1/L4) RECORD +1 RECORD +2 RECORD +3 : : : FOR EXAMPLE: READ MLF LOGICAL GET isn-x/UPDATE/REC (isn-x not yet read ) !! CAN CAUSE OUT-OF-SEQUENCE FOR READ LOGICAL !!
SIZE MFF/PREFNREC. UPPER LIMIT 1. 2048 (IBL=32K/16 ADABAS V7) 2. 32K / (RECORD.SIZE) (ADABAS V7) 3. ISN-QUANTITY FOR FIND (Sx) STATEMENTS 4. RECORDS PER LOOP (*COUNTER)
1. DATABASE USE: SHARED ( ) OR DEDICATED ( )? 2. ADAMLF. PROGRAM DELAY: MUCH ( ) OR LITTLE ( )? 3. ADAMLF. ARE UPDATES BEING DONE ( )? 4. IS SEQUENCE READ PHYSICAL ( ) OR LOGICAL ( ) ? SIZE MFF/PREFNREC. VARY VALUE IF:
SIZE MFF/PREFNREC. 1. (PERSONAL PHILOSOPHY): GOAL OF 7 TO 10 IO’S PER COMMAND. READ PHYSICAL, USE ADAREP, RECS/BLOCK X 10 READ LOGICAL, DETERMINE AVERAGE IO/CMD 2. = J / D (ROUND UP TO INTEGER) J = ELAPSED TIME OF JOB (IDEALLY READ SEQUENCE) D = SUM OF ADABAS DURATION FOR SEQUENCE 3. START SMALL (5,10,20, . . . ) MONITOR CPU, ELAPSED TIMES. ! DO NOT INCREASE IF ELAPSED TIME DOES NOT INCREASE ! ANALYSIS REPORT for #1, #2
QUICK CA-APAS SIZE MFF/PREFNREC. MCIMLF: SUMMARIZE(SUBTOT=OFF) MIN(TIME) MAX(TIME) COUNT (PF=N7) SUM(TOTAL-IO) (PF=N7) SUM(DURATION) (PF=N4.2) SUM(ENQ-TIME) (PF=N4.2) SUM(CMD-TIME) (PF=N4.2) MEAN(RBL) (PF=N3.1) WHERE FILE = 241 AND CMD = 'L2‘ * AND CID = X'02000101' AND NAT-PROGRAM = 'TPGM' * AND JOBNAME = 'THISJOB‘ PRINT TO ADHOC1 INSIGHT-LINES=600; ANALYSIS REPORT for #1, #2 TO ENSURE ONLY 1 SEQUENCE IS MEASURED QUALIFY SEQUENCE FOR EXAMPLE
EXAMPLE 1 (L2 SEQ): EXAMPLE 2 (L3 SEQ): 1. IO / CMD = 180K / 1103K = .16 10 / .16 = 60 (MFF) 2. ELAPSED TIME = 2780 TIME / DUR = 2780 / 531 = 6 (MFF) 1. IO / CMD = 6604 / 4460 = 1.5 10 / 1.5 = 7 (MFF) 2. ELAPSED TIME = 555 TIME / DUR = 555 / 61 = 10 (MFF) SIZE MFF/PREFNREC. Eg. Method #1,#2 SUM SUM SUM SUM MIN MAX CMD TOT DURAT ENQTIM CMD MEAN TIME TIME COUNT IO SECS SECS TIME RBL EXAMPLE 1: 154931 1635511,103,425179,865530.76 939.96 70.14 184.0 EXAMPLE 2: 1609261618414,4606,60461.07 0.02 1.11 204.0
SIZE MFF/PREFNREC. AGAIN #3 3. START SMALL (5,10,20, . . . ) MONITOR CPU, ELAPSED TIMES. ! DO NOT INCREASE IF ELAPSED TIME DOES NOT INCREASE ! PERFORMANCE (PGM ELAPSED TIME) = ADABAS RESPONSE (CPU,IO,OVERLAP) PGM PROCESS ADABAS CALLS PATHLENGTH TO/FROM PGM-ADABAS PGM DELAY (CPU,IO,ETC.) EXAMPLES
PATHLENGTH PROGRAM TIME ADABAS TIME MINIMAL PGM DELAY MINIMAL PATHLENGTH SIZE MFF/PREFNREC. PGM ELAPSED TIME STOP NREC: - 0 - - 25 - - 50 - - 100 - 200 - - 400 -
PGM DELAY: Lotsa MOVE stmts MINIMAL PATHLENGTH SIZE MFF/PREFNREC. PGM ELAPSED TIME PATHLENGTH PROGRAM TIME ADABAS TIME STOP NREC: 0 - 5 - - 10 - - 20 - - 30 - - 60 - - 400 -