1 / 48

Disk I/O Performance focusing on Caching

Disk I/O Performance focusing on Caching. Guy Peleg President Maklee Engineering guy.peleg@maklee.com. Why should I care?. Computers became very fast over the last decade I/O performance remained (almost) the same Disk I/O is an extremely! slow operation. Why should I care?.

rad
Download Presentation

Disk I/O Performance focusing on Caching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Disk I/O Performancefocusing on Caching Guy Peleg President Maklee Engineering guy.peleg@maklee.com

  2. Why should I care? • Computers became very fast over the last decade • I/O performance remained (almost) the same • Disk I/O is an extremely! slow operation

  3. Why should I care? • 1.6Ghz rx7640 single cell • CPU cycle is 0.5 ns • Memory latency is ~ 250 ns • EVA5000 read latency 5 – 10 ms • Now let’s change the scale and assume 0.5 ns = 1 sec

  4. Why should I care? • 1.6Ghz rx7640 single cell • CPU cycle is 1 second • Memory latency is 8.3 minutes • EVA5000 read latency 23 days !!!!!

  5. Why should I care? • Test program written in C • Creates a 1GB container file • Opens the newly created file • Positions at EOF • Writes a small record • Closes file • 1.6Ghz rx2600 • OpenVMS V8.3-1H1

  6. Why should I care? 88% difference Seconds to complete the test program less is better

  7. I/O Best Practices • “The fastest I/Os are those that are avoided” OpenVMS info desk Oct. 2004 • Cache as much as you can • Caching was invented to attempt and reduce the need for physical I/Os. • This session is all about providing techniques for avoiding/reducing I/O

  8. High Water Marking • The file highwater mark attribute guarantees that a user can not read data that was not written by the user. • The init command turns this feature on by default • Tradeoff between security and performacne • INIT/NOHIGH • SET VOLUME/NOHIGH

  9. The impact of highwater mark Seconds to create a container file less is better

  10. Data caching • Data could be cached by: • The Application • RMS (or Database cache) • OpenVMS file caching (XFC) • Hardware (disk controller) No one knows your application better than you !! Try to satisfies I/O requests as close to the application as possible

  11. XFC • eXtended File Cache • Introduced with OpenVMS V7.3 • Replaces VIOC • 64 bit • No closed file limit • Will cache I/Os larger than 35 blocks

  12. XFC • Supports readahead • (currently) does not support writeback • May grow up to 50% of physical memory • Grows and shrinks according to free physical memory. • May be forced to return memory instantly • No caching for files opened for write cluster wide

  13. XFC Reserved Memory Using reserved memory for XFC is slightly faster Reserved Memory for Permanent memory SYSMAN> RESERVED_MEMORY ADD VCC$MIN_CACHE_SIZE/SIZE=1500 /ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL VCC_MAX_CACHE must be greater than the reserved memory size.

  14. XFC SYSGEN paraeters $ mc sysgen sh vcc Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- VCC_FLAGS 2 2 0 -1 Bitmask VCC_MAXSIZE 6400 6400 0 3700000 Blocks VCC_MAX_CACHE -1 -1 0 -1 Mbytes D VCC_MAX_IO_SIZE 256 127 0 -1 Blocks D VCC_MAX_LOCKS -1 -1 50 -1 Locks D VCC_READAHEAD 1 1 0 1 Boolean D VCC_WRITEBEHIND 1 1 0 1 Boolean D VCC_WRITE_DELAY 30 30 0 -1 Seconds D VCC_PAGESIZE 0 0 0 -1 D VCC_RSVD 0 0 0 -1 D $

  15. $show memory /cache/full $ sh mem/cache/full System Memory Resources on 13-MAY-2008 12:44:19.80 Extended File Cache (Time of last reset: 8-MAY-2008 22:57:33.05) Allocated (GBytes) 1.35 Maximum size (GBytes) 4.00 Free (GBytes) 0.21 Minimum size (GBytes) 0.00 In use (GBytes) 1.13 Percentage Read I/Os 26% Read hit rate 95% Write hit rate 0% Read I/O count 2901784 Write I/O count 8118032 Read hit count 2781426 Write hit count 0 Reads bypassing cache 180 Writes bypassing cache 6860600 Files cached open 848 Files cached closed 993 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 3 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0 I/O Statistics - Distributions (MAX_IO_SIZE: 127) ------------------------------------------------- Transfer Size: Reads Read Hits Writes 1 Block IO: 193447 191602 435529 2 Block IO: 77685 76719 2148301 3 Block IO: 37242 37063 322191 4 Block IO: 108729 108557 238616 5 Block IO: 11719 11592 139933 6 Block IO: 77626 77453 92434 7 Block IO: 45780 45618 141327 8 Block IO: 191750 191351 136810 9 Block IO: 5359 5280 207786 10 Block IO: 17835 17787 140148 11 Block IO: 5343 5285 60112

  16. Show memory /cache/full … 27 Block IO: 4670 4655 14815 28 Block IO: 5657 5625 15013 29 Block IO: 5218 5192 14984 30 Block IO: 4717 4696 14841 31 Block IO: 4708 4679 14344 32 Block IO: 527966 467655 996501 33 Block IO: 48 33 2269 … 63 Block IO: 466 432 6366 64 Block IO: 1075132 1065615 22811 65 Block IO: 2 0 1144 124 Block IO: 200562 190862 193467 125 Block IO: 1 0 493 126 Block IO: 1 0 504 127 Block IO: 0 0 512 >127<256 Block IO: 21 263932 >255 Block IO: 49 176683 ------------------------------------------------- Totals: 2901784 2781426 8118032

  17. Forcing XFC to shrink $ show mem/cache System Memory Resources on 13-MAY-2008 12:42:54.59 Extended File Cache (Time of last reset: 13-MAY-2008 10:13:01.11) Allocated (MBytes) 222.32 Maximum size (MBytes) 4096.00 Free (MBytes) 0.19 Minimum size (MBytes) 3.12 In use (MBytes) 222.12 Percentage Read I/Os 87% Read hit rate 29% Write hit rate 0% Read I/O count 90334 Write I/O count 12864 Read hit count 26537 Write hit count 0 Reads bypassing cache 589 Writes bypassing cache 9969 Files cached open 383 Files cached closed 631 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 2 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0 SYSBCK> mc sysgen SYSGEN> SET VCC_MAX_CACHE 100 SYSGEN> W A SYSGEN> EXIT $ set cache/reset

  18. Forcing XFC to shrink $ sh mem/cache System Memory Resources on 13-MAY-2008 12:43:22.10 Extended File Cache (Time of last reset: 13-MAY-2008 10:13:01.11) Allocated (MBytes) 95.63 Maximum size (MBytes) 100.00 Free (MBytes) 0.02 Minimum size (MBytes) 3.12 In use (MBytes) 95.60 Percentage Read I/Os 87% Read hit rate 29% Write hit rate 0% Read I/O count 90391 Write I/O count 12894 Read hit count 26584 Write hit count 0 Reads bypassing cache 589 Writes bypassing cache 9996 Files cached open 383 Files cached closed 181 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 2 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

  19. Cache pollution • The Cache is a limited resource • Avoid polluting the cache • OpenVMS attempts to reduce pulltion • VMS BACKUP • Caching disabled for all backup IO • Some utilities are not “cache safe” • COPY • CONVERT • Consider disabling caching temporarily • vcc_max_io_size = 0 • mount/nocache • set file/cache=no • OpenVMS provides DCL commands to view the contents of the cache

  20. Free Hot File Tracking Utility $ sh mem/cache=(volume=*,topqio) System Memory Resources on 26-APR-2008 01:39:15.03 Extended File Cache Top QIO File Statistics: _$1$DGA242: (DISK$SEC6), Caching mode is VIOC Compatible _$1$DGA242:[VMS$COMMON.SYSEXE]RIGHTSLIST.DAT;1 (open) Caching is enabled, active caching mode is Write Through Allocated pages 9 Total QIOs 107 Read hits 92 Virtual reads 107 Virtual writes 0 Hit rate 85 % Read aheads 0 Read throughs 107 Write throughs 0 Read arounds 0 Write arounds 0 _$1$DGA242:[VMS$COMMON.SYSEXE]VMS$OBJECTS.DAT;2 (open) Caching is enabled, active caching mode is Write Through Allocated pages 0 Total QIOs 9 Read hits 0 Virtual reads 9 Virtual writes 0 Hit rate 0 % Read aheads 0 Read throughs 9 Write throughs 0 Read arounds 0 Write arounds 0 _$1$DGA242:[VMS$COMMON.SYSEXE]VMS$AUDIT_SERVER.DAT;1 (open) Caching is enabled, active caching mode is Write Through Allocated pages 1 Total QIOs 4 Read hits 0 Virtual reads 4 Virtual writes 0 Hit rate 0 % Read aheads 0 Read throughs 4 Write throughs 0 Read arounds 0 Write arounds 0 Total of 3 files for this volume

  21. Free Hot File Tracking Utility _$1$DGA142: (DISK$MVMS), Caching mode is VIOC Compatible _$1$DGA142:[VMS$COMMON.SYSLIB]DECC$SHR.EXE;1 (open) Caching is enabled, active caching mode is Write Through Allocated pages 303 Total QIOs 1646 Read hits 1561 Virtual reads 1646 Virtual writes 0 Hit rate 94 % Read aheads 0 Read throughs 1642 Write throughs 0 Read arounds 4 Write arounds 0 _$1$DGA142:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1 (open) Caching is enabled, active caching mode is Write Through Allocated pages 143 Total QIOs 1165 Read hits 1123 Virtual reads 1165 Virtual writes 0 Hit rate 96 % Read aheads 0 Read throughs 1164 Write throughs 0 Read arounds 1 Write arounds 0 _$1$DGA142:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.EXE;1 (open) Caching is enabled, active caching mode is Write Through Allocated pages 12 Total QIOs 720 Read hits 711 Virtual reads 720 Virtual writes 0 Hit rate 98 % Read aheads 0 Read throughs 720 Write throughs 0 Read arounds 0 Write arounds 0 Avoid caching files that pollute the cache

  22. I/O response time SDA> xfc show volume/brief Summary of XFC Cached Volumes (CVBs)------------------------------------Volume Name       CVB                Open   Closed      Total       Read       Read      Write      ... Response (Milliseconds)...                                    Files    Files       I/Os       Hits      Count      Count          Hits       disk    AverageDISK$FOOBAR       FFFFFFFEE01895E0      0        0          0          0          0          0       (N/A)      (N/A)      (N/A)DISK$GP           FFFFFFFEE0189380      0        0          0          0          0          0       (N/A)      (N/A)      (N/A)DISK$ORACLE       FFFFFFFEE0189120     26        3    1872255          0          0    1872255       (N/A)       0.0000     0.0000DISK$ORA2        FFFFFFFEE0188EC0     73      177   22015701   14108183   21116834     898891        0.0232     0.5811     0.2236DISK$SYS      FFFFFFFEE0188C60      0        0          0          0          0          0       (N/A)      (N/A)      (N/A)DISK$ORAIDX      FFFFFFFEE0188A00      0        0          1          0          1          0       (N/A)      (N/A)      (N/A)DISK$SYSBCK    FFFFFFFEE01887A0      2        0          0          0          0          0       (N/A)      (N/A)      (N/A)DISK$LEEHE        FFFFFFFEE0188540      4        3   27676052   27667501   27674665       1387        0.0118     0.4007     0.0120DISK$MIKA     FFFFFFFEE01882E0      0        0          0          0          0          0       (N/A)      (N/A)      (N/A)DISK$SYS831H1     FFFFFFFEE0188080    313      183    2736618    2668894    2713025      23594        0.0179     0.5425     0.0308 SDA>XFC SHOW VOLUME/BRIEF

  23. The XFC “overhead” Elapsed time to copy 150MB file, rx2600, HSG80, OpenVMS V8.3

  24. The XFC “overhead” Seconds to create a container file less is better

  25. Shadowed RAM disk • Applications that mainly read data would benefit from using shadowed RAM disk. • The Shadow server will read from memory and will write to both devices. • Forces data to remain resident in memory. • Significantly boosts performance when files are opened cluster wide by multiple users. • XFC will not help

  26. Physical Disk Vs. RAM disk • C application that processes records read from sequential file • Each I/O 124 Blocks • RX2600, OpenVMS V8.3, HSG80 Elapsed time to read 250MB file (less is better)

  27. RMS • RMS holds great potential for reducing I/Os & improving performance • The C RTL uses RMS • Most C applications would benefit from RMS tuning

  28. RMS • RMS parameters related to performance: • FAB/RAB parameters (should you have access to the code): • ASY, RAH, WBH, DFW • ALQ & DEQ • MBC & MBF • NOSHR, NQL, NLK • SET RMS … • /SYSTEM | /PROCESS • /BUFFER_COUNT=n • /BLOCK_COUNT=n • SYSGEN> SET RMS_SEQFILE_WBH 1 • Global Buffers

  29. gZIP & RMS • gZIP is written in C – I/Os eventually reach RMS • 1.6 Ghz rx2600, MSA30, OpenVMS V8.3 • Test 1 • Compress 5.67 GB saveset • Decompress 2.74 gZIP archive • Default O/S & RMS settings • Test 2 • Compress 5.67 GB saveset • Decompress 2.74 gZIP archive • SET RMS/BLOCK=127/EXTEN=60000/BUFFER=8, RMS_SEQFILE_WBH=1

  30. gZIP & RMS Elapsed Time in Minutes (less is better)

  31. FTP Performance

  32. Impact of RMS tuning Seconds to create a container file less is better

  33. I/O Consolidation - PC Sampling – “Before” PC Sampling Information (CPU_CYCLES):-------------------------------------Start VA          End VA             Image                                             Count   Percent----------------- -----------------  ----------------------------------------    -----------  --------FFFFF802.89400000 FFFFF802.89415FFF  Kernel Promote VA                                     2     0.78%FFFFFFFF.80000100 FFFFFFFF.800111FF  SYS$BASE_IMAGE                                       10     3.89%FFFFFFFF.80011200 FFFFFFFF.80068FFF  SYS$PLATFORM_SUPPORT                                  1     0.39%FFFFFFFF.800A0000 FFFFFFFF.801D3CFF  SYSTEM_PRIMITIVES_MIN                                86    33.46%FFFFFFFF.801D3D00 FFFFFFFF.802537FF  SYSTEM_SYNCHRONIZATION_MIN                           21     8.17%FFFFFFFF.80365600 FFFFFFFF.80451FFF  EXCEPTION                                             1     0.39%FFFFFFFF.80452000 FFFFFFFF.80543EFF  IO_ROUTINES                                           6     2.33%FFFFFFFF.80553200 FFFFFFFF.80699DFF  PROCESS_MANAGEMENT                                   43    16.73%FFFFFFFF.80699E00 FFFFFFFF.807A25FF  SYS$VM                                                2     0.78%FFFFFFFF.807A2600 FFFFFFFF.807F05FF  LOCKING                                              14     5.45%FFFFFFFF.80847600 FFFFFFFF.80927CFF  F11BXQP                                              22     8.56%FFFFFFFF.80927D00 FFFFFFFF.80B40CFF  RMS                                                  10     3.89%FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF  SECURITY                                              4     1.56%FFFFFFFF.80CB0A00 FFFFFFFF.80D2C9FF  SYS$XFCACHE                                          11     4.28%FFFFFFFF.810C1B00 FFFFFFFF.811080FF  SYS$DKDRIVER                                          1     0.39%FFFFFFFF.8406E000 FFFFFFFF.8425FFFF  LIBRTL                                                3     1.17%FFFFFFFF.84260000 FFFFFFFF.84285FFF  LIBOTS                                                2     0.78%FFFFFFFF.844FA000 FFFFFFFF.845ABFFF  PTHREAD$RTL                                          18     7.00%                                     Not Found                                             4

  34. I/O Consolidation - PC Sampling – “After” PC Sampling Information (CPU_CYCLES):-------------------------------------Start VA          End VA             Image                                             Count   Percent----------------- -----------------  ----------------------------------------    -----------  --------00000000.00000000 00000000.7ADCBFFF  Process Space                                     28936    34.50%FFFFF802.89400000 FFFFF802.89415FFF  Kernel Promote VA                                   236     0.28%FFFFFFFF.80000000 FFFFFFFF.800000FF  SYS$PUBLIC_VECTORS                                   13     0.02%FFFFFFFF.80000100 FFFFFFFF.800111FF  SYS$BASE_IMAGE                                     3272     3.90%FFFFFFFF.80011200 FFFFFFFF.80068FFF  SYS$PLATFORM_SUPPORT                                 56     0.07%FFFFFFFF.800A0000 FFFFFFFF.801D3CFF  SYSTEM_PRIMITIVES_MIN                              7848     9.36%FFFFFFFF.801D3D00 FFFFFFFF.802537FF  SYSTEM_SYNCHRONIZATION_MIN                         2349     2.80%FFFFFFFF.8027C700 FFFFFFFF.8029CAFF  SYS$EI1000.EXE                                        2     0.00%FFFFFFFF.80365600 FFFFFFFF.80451FFF  EXCEPTION                                           153     0.18%FFFFFFFF.80452000 FFFFFFFF.80543EFF  IO_ROUTINES                                        3664     4.37%FFFFFFFF.80543F00 FFFFFFFF.805531FF  SYSDEVICE                                           838     1.00%FFFFFFFF.80553200 FFFFFFFF.80699DFF  PROCESS_MANAGEMENT                                 5334     6.36%FFFFFFFF.807A2600 FFFFFFFF.807F05FF  LOCKING                                               1     0.00%FFFFFFFF.80927D00 FFFFFFFF.80B40CFF  RMS                                                   2     0.00%FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF  SECURITY                                            479     0.57%FFFFFFFF.80CB0A00 FFFFFFFF.80D2C9FF  SYS$XFCACHE                                           3     0.00%FFFFFFFF.80D58E00 FFFFFFFF.80E1C6FF  SYS$CLUSTER                                           1     0.00%FFFFFFFF.81049500 FFFFFFFF.810C1AFF  SYS$PKMDRIVER                                         4     0.00%FFFFFFFF.810C1B00 FFFFFFFF.811080FF  SYS$DKDRIVER                                          1     0.00%FFFFFFFF.8110BF00 FFFFFFFF.8113D3FF  SYS$LAN_CSMACD.EXE                                    1     0.00%FFFFFFFF.8113D400 FFFFFFFF.811B03FF  SYS$PEDRIVER.EXE                                      6     0.01%FFFFFFFF.81356900 FFFFFFFF.813BCAFF  SYS$LOADABLE_IMAGES:SYS$LTDRIVER.EXE                  2     0.00%FFFFFFFF.813E3E00 FFFFFFFF.816028FF  TCPIP$INTERNET_SERVICES                               6     0.01%FFFFFFFF.81640300 FFFFFFFF.817141FF  TCPIP$NFS_SERVICES                                    2     0.00%FFFFFFFF.8406E000 FFFFFFFF.8425FFFF  LIBRTL                                            18395    21.93%FFFFFFFF.84260000 FFFFFFFF.84285FFF  LIBOTS                                             2251     2.68%FFFFFFFF.844FA000 FFFFFFFF.845ABFFF  PTHREAD$RTL                                        9452    11.27%FFFFFFFF.845AC000 FFFFFFFF.850BDFFF  DECC$SHR                                            555     0.66%                                     Not Found                                           633

  35. RMS Global Buffers • RMS Global Buffers provide another layer of caching between RMS & XFC • Does it make sense to use both? • YES !

  36. RMS Global Buffers and/or XFC? • RMS Global buffers WILL cache files open for writeshare in a cluster. • RMS GB require knowledge of application & I/O to apply them effectively. • $ SET FILE/GLOBAL_BUFFER=n • XFC is S2 Space based and has no effective limit to its size other than physical memory available • Prior to V8.3 RMS GB are limited to a max value of 32767 buffers per file.

  37. RMS Global Buffers AND XFC make a good pair • Remember slide 13? • RMS is closer to the application. • Use global buffers for files open for shared write in a cluster – RIGHTSLIST.DAT is a good example with a common system disk. • XFC will cache as much as it can once RMS passes I/O to it

  38. The Impact of RMS Global Buffers

  39. The Impact of RMS Global Buffers

  40. File Expansions • The volume extension size specifies the number of blocks to be added when RMS automatically extends a file. • Larger values result in fewer file extensions. • SET VOLUME/EXTENSION=n • FAB$W_DEQ • Maximum extension size – 65,535 blocks

  41. File Expansions $ sh dev dka100/full Disk $1$DKA100: (STRIKE), device type DEC RZZ28B, is online, allocated, deallocate on dismount, mounted, file-oriented device, shareable, available to cluster, error logging is enabled. Error count 0 Operations completed 57 Owner process "SYSTEM" Owner UIC [SYSTEM] Owner process ID 00000222 Dev Prot S:RWPL,O:RWPL,G:R,W Reference count 2 Default buffer size 512 Total size 14.05GB Sectors per track 32 Total cylinders 28786 Tracks per cylinder 32 Logical Volume Size 14.05GB Expansion Size Limit 16.96GB Allocation class 1 Volume label "XBC3_RESD$" Relative volume number 0 Cluster size 4 Transaction count 1 Free space 106.93MB Maximum files allowed 2947659 Extend quantity 5 Mount count 1 Mount status Process Cache name "_$1$DKA0:XQPCACHE" Extent cache size 64 Maximum blocks in extent cache 21900 File ID cache size 64 Blocks in extent cache 0 Quota cache size 0 Maximum buffers in FCP cache 1724 Volume owner UIC [SYSTEM] Vol Prot S:RWCD,O:RWCD,G:RWCD,W:RWCD Volume access control list: (IDENTIFIER=%X80010033,ACCESS=READ+WRITE+CREATE+DELETE)

  42. Oracle 10g Database Export 20% Improvement Minutes to complete database export less is better

  43. Larger Block Size with RMS • Going beyond 127 blocks I/O requires using RAB64 • Implemented in VMS COPY • /BLOCK qualifier • The Port drivers currently support a maximum of 256 blocks per I/O • Larger I/Os are segmented

  44. Impact of larger block size Seconds to copy 5GB file from disk to disk less is better

  45. Evolution of the test program Seconds to create a container file less is better

  46. FastPath • Spread I/Os across all CPUs $ show fast Fast Path preferred CPUs on MAKLEE 13-MAY-2008 21:20:41.82 HP rx3600 (1.59GHz/9.0MB) with 4 active CPUs Device: Fastpath CPU: EIA0 1 EIB0 0 FGC0 1 FGA0 0 FGB0 3 FGD0 3 PKA0 2 OpenVMS TCP/IP is currently running on CPU 1

  47. Summary • Disk I/O is an extremely slow operation • Don’t stall the application • Use Asynchronous I/Os whenever possible • Consolidate your I/Os • Perform larger and fewer I/Os • Cache information as close to the application as possible

  48. Questions? • See us at www.maklee.com for: • Performance improvements • Oracle Tuning • Platform Migration • Custom Engineering solutions • Custom Training

More Related