1 / 51

juliandyke.com

Automatic Storage Management. Julian Dyke Independent Consultant. Web Version - December 2008. juliandyke.com . Objectives. Understand how Oracle database files are stored in ASM Calculate how long ASM rebalance operations will take. Agenda. ASM Instances ASM Disk Groups Metadata

denver
Download Presentation

juliandyke.com

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automatic StorageManagement Julian Dyke Independent Consultant Web Version - December 2008 juliandyke.com

  2. Objectives • Understand how Oracle database files are stored in ASM • Calculate how long ASM rebalance operations will take

  3. Agenda • ASM Instances • ASM Disk Groups • Metadata • Extent Distribution • Rebalancing • Redundancy

  4. ASMInstances

  5. OracleClusterware ASMInstance RDBMSInstance Server ASM Single Instance Architecture OCSSDDaemonOnly DedicatedStorage

  6. MMAN DIAG X000 GMON RBAL PSP0 DIA0 DBW0 LGWR CKPT SMON PMON VKTM ASM Single Instance Background Processes Oracle 11.1 Fixed Area Variable Area ASM Cache

  7. Oracle Clusterware Oracle Clusterware Oracle Clusterware Oracle Clusterware ASMInstance ASMInstance ASMInstance ASMInstance RDBMSInstance RDBMSInstance RDBMSInstance RDBMSInstance Node 1 Node 2 Node 4 Node 3 ASM RAC Architecture PublicNetwork Private Network Storage Network SharedStorage

  8. ASM RAC Architecture Clusterware CLUSTERWARE CLUSTERWARE ASM Instances +ASM1 +ASM2 RDBMS Instances PROD1 TEST1 PROD2 TEST2 PROD TEST Database Files

  9. DIAG RBAL PMON SMON CKPT LGWR DBW0 VKTM MMAN KATE PSP0 X000 MARK DIA0 LMON LMS0 LMD0 LCK0 DIAG GMON ASM RAC Instance Background Processes Oracle 11.1 Fixed Area Variable Area ASM Cache

  10. ASMDisk Groups

  11. Disk 1 Disk 4 Disk 2 Disk 6 Disk 5 Disk 7 Disk 3 ASM Disk Groups and Disks Disk Group 1 Disk Group 2 Disk Group 3

  12. ASM Disk Groups and Disks Disk Group 1 Disk Group 2 Disk Group 3 Disk 1 Disk 4 Disk 2 Disk 5 Disk 6 Disk 7 Disk 3

  13. ASM Disk Groups, Disks and Database Files Disk Group 1 Disk Group 2 Disk Group 3 File 2 File 3 File 1 File 5 File 4 File 6 File 5 Disk 3 Disk 1 Disk 4 Disk 2 Disk 5 Disk 6 Disk 7

  14. File Extents versus Allocation Units • File Extent • Logical unit of ASM file • Map to allocation units • One to many mapping • Allocation Unit • Physical unit of ASM disk • Oracle 10.2 and below • Always 1MB • Can be increased using _asm_ausize • Oracle 11.1 and above • Variable size • 1MB, 2MB ,4MB, 8MB, 16MB, 32MB, 64MB

  15. X$KFFXP • Maps file extents to allocation units • Only populated in ASM instance • Columns include

  16. ASM Metadata • Metadata is stored in first 256 files in ASM disk group • Space is initially allocated when disk group is created • Can be subsequently extended • Metadata allocation units are divided into blocks • Each block is 4096 bytes • Block size specifed using _asm_blksize • Metadata files include

  17. ASM Metadata Metadata Header Disk Header Partner Status Table Free Space Table File Directory Disk Directory Allocation Table Active ChangeDirectory Continuing OperationsDirectory Template Directory Alias Directory Metadata Header

  18. ASM Metadata • Initial Allocation (Single Instance) • Active Change Directory • Records changes to metadata • Used during recovery of instance or operation failures • Continuing Operations Directory • Maintains state of active operations

  19. ASM Metadata Block Types

  20. KFED Utility • In Oracle 10.2 and above the kfed utility can be used to inspect and edit the contents of ASM blocks [oracle@server3 ~]$ $ORACLE_HOME/bin/kfed -has/mlib ASM Library [asmlib='lib']aun/um AU number to examine or update [AUNUM=number]aus/z Allocation Unit size in bytes [AUSZ=number]blkn/um Block number to examine or update [BLKNUM=number]blks/z Metadata block size in bytes [BLKSZ=number]ch/ksum Update checksum before each write [CHKSUM=YES/NO]cn/t Count of AUs to process [CNT=number]d/ev ASM device to examine or update [DEV=string]o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT]p/rovnm Name for provisioning purposes [PROVNM=string]s/eek AU number to seek to [SEEK=number]te/xt File name for translated block text [TEXT=string]ty/pe ASM metadata block type number [TYPE=number] • This utility should only be used under the guidance of Oracle Support

  21. KFED Utility • For example to dump blocks in aliases directory in DISKGROUP1 • Find group number SELECT group_number FROM v$asm_diskgroupWHERE name = 'DISKGROUP1'; • Alias directory is stored in file number 6 SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1AND number_kffxp = 6AND lxn_kffxp = 0; Disk Allocation Unit 0 49 • Find disk name SELECT path FROM v$asm_diskWHERE group_number = 1AND disk_number = 0; Path/dev/oracleasm/disks/VOL1

  22. KFED Utility • Example (continued) • Allocation unit is 1MB • Block size is 4096 • Therefore there are 256 blocks per allocation unit • Starting block offset = 256 * 49 = 12544 for (( f = 12544 ; f < 12544 + 256 ; f++ ))do kfed op=read blkn=$f dev='/dev/oracleasm/disks/VOL1' > blk${f}done

  23. Extent Distribution

  24. Extent Distribution • Creating a disk group: CREATE DISKGROUP diskgroup1 EXTERNAL REDUNDANCYDISK '/dev/oracleasm/disks/VOL1'; • Dropping a disk group: DROP DISKGROUP diskgroup1 INCLUDING CONTENTS

  25. 0 1 2 3 4 5 6 7 Extent Distribution1 disk Metadata Disk 0 Metadata Data

  26. 12 7 5 3 1 14 10 8 9 4 2 0 11 13 15 6 Extent Distribution2 disks Metadata Disk 0 Disk 1 Metadata Data

  27. 22 7 3 31 27 23 19 15 0 4 8 12 16 24 28 2 20 10 29 25 21 6 13 9 5 17 30 26 11 18 14 1 Extent Distribution4 disks Metadata Disk 0 Disk 1 Disk 2 Disk 3 Metadata Data

  28. 6 10 9 7 4 3 2 0 5 8 11 1 Extent Distribution1 large disk - 1 small disk Metadata Disk 0 Disk 1 Metadata Data

  29. 13 7 2 19 14 1 6 16 9 0 11 5 3 17 18 15 12 10 8 4 Extent Distribution1 large disk - 3 small disks Metadata Disk 0 Disk 1 Disk 2 Disk 4 Metadata Data

  30. 20 11 5 22 19 3 9 15 21 0 2 6 16 12 10 8 4 1 17 7 13 18 14 23 Extent Distribution2 large disks - 2 small disks Metadata Disk 0 Disk 1 Disk 2 Disk 4 Metadata Data

  31. Rebalancing

  32. Extent Distribution • Adding a disk: ALTER DISKGROUP diskgroup1 ADD DISK '/dev/oracleasm/disks/VOL2' REBALANCE POWER 0; • Dropping a disk: ALTER DISKGROUP diskgroup1 DROP DISK 'DISKGROUP1_0002' REBALANCE POWER 0; • Rebalancing a disk group: ALTER DISKGROUP diskgroup1 REBALANCE POWER 1;

  33. 1 7 5 0 3 1 2 3 4 5 6 7 1 5 0 1 2 3 7 5 6 7 3 4 Disk 1 STOP RebalancingAdding disks - 1 disks to 2 disks Disk 0 Metadata Data

  34. 0 6 2 5 0 4 1 1 2 3 4 5 6 7 0 1 0 1 2 3 4 2 6 7 6 5 4 5 STOP RebalancingAdding disks - 1 disks to 4 disks Disk 0 Disk 1 Disk 2 Disk 3 Metadata Data

  35. 4 0 0 3 1 1 2 3 4 5 9 7 5 3 1 14 16 17 6 11 17 13 2 0 10 11 15 5 2 8 12 8 4 14 RebalancingAdding disks - 2 disks to 3 disks Disk 0 Disk 1 Disk 2 Metadata Data

  36. 0 4 6 2 5 0 1 1 2 3 4 5 6 7 5 13 8 6 4 2 0 15 11 14 9 7 10 3 12 5 4 0 13 8 1 12 1 9 STOP RebalancingAdding disks - 2 disks to 4 disks Disk 0 Disk 1 Disk 2 Disk 3 Metadata Data

  37. 4 1 5 2 0 3 1 2 4 5 7 8 2 1 0 3 6 4 2 5 7 4 5 8 1 STOP RebalancingDropping disks - 3 disks to 1 disk Disk 0 Disk 1 Disk 2 Metadata Data

  38. 6 2 4 0 0 1 2 3 4 5 6 7 14 10 8 6 4 15 13 11 9 7 5 3 12 0 6 8 1 2 2 4 10 12 14 0 STOP RebalancingMoving disks - 2 disks to 2 disks Disk 0 Disk 1 Disk 2 Metadata Data

  39. RebalancingV$ASM_OPERATION • Contains details of ongoing rebalance operations Estimate of remaining time

  40. RebalancingPower Limit • Power limit can be 0 to 11 • 0 disables rebalance operation • 1 to 11 specifies number of ARBn background processes used for rebalance • In Oracle 10.2 • RBAL manages rebalance operation • Each ARBn background process is allocated a range of 128 allocation units to rebalance • When complete another range is requested • AD lock is taken while an allocation unit is being rebalanced • Rebalance operations take much longer than theoretically necessary. Possible reasons include: • Locking • GES updates with other ASM instances • Updates to RDBMS instance

  41. RebalancingSummary • EST_MINUTES column of V$ASM_OPERATION is reasonably accurate • Allow a few minutes for SAN cache to stabilize • Check regularly for changes to estimate • ASM rebalance operations do not affect workload • Locks are only taken briefly • Lock mechanism has changed in Oracle 11.1 • SAN cache and I/O performance will be affected • In Oracle 10.2 rebalancing is fastest if • Other ASM instances are shutdown • RDBMS instance is shutdown • Estimated completion time will be affected by: • Use of SAN cache and I/O by rest of workload • Rate of change by applications to blocks in ASM files being rebalanced

  42. Redundancy

  43. Redundancy • ASM Supports three levels of redundancy • External Redundancy • Implemented externally using storage layer • Most common configuration in production • Normal Redundancy • Two copies of each extent maintained in separate failure groups • Used with extended clusters • Used occasionally in production e.g. CERN • Increases CPU overhead on servers • High Redundancy • Three copies of each extent maintained in separate failure groups • Very rare in production

  44. Disk 1 Disk 2 Disk 3 ASM Failure Groups - External Redundancy Disk Group

  45. Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 ASM Failure Groups - Normal Redundancy Disk Group Failure Group 1 FailureGroup 2

  46. Disk 1 Disk 2 Disk 1 Disk 2 Disk 1 Disk 2 ASM Failure Groups - High Redundancy Disk Group Failure Group 1 Failure Group 2 Failure Group 3

  47. Failure Group 2 Failure Group 1 0 0 1 2 2 3 3 1 10 8 3 0 7 11 2 5 6 9 4 1 3 7 0 6 11 8 5 9 4 2 1 10 Normal Redundancy1 Disk Per Failure Group Disk 0 Disk 1 Metadata Primary Secondary Data Primary Secondary

  48. 5 7 4 6 4 2 3 1 0 0 1 5 7 3 2 6 17 12 5 0 9 16 1 13 8 11 21 20 7 6 22 2 10 19 15 14 18 23 4 3 17 19 2 11 21 14 15 13 1 5 3 9 20 16 23 10 22 6 8 4 0 12 18 7 Normal Redundancy2 Disks per Failure Group Failure Group 2 Failure Group 1 Disk 0 Disk 1 Disk 2 Disk 3 Metadata Primary Secondary Data Primary Secondary

  49. Failure Group 3 Failure Group 2 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 9 8 2 0 1 6 5 4 3 7 8 1 2 3 0 6 9 5 4 7 0 1 9 6 3 2 5 4 7 8 High Redundancy1 Disk per Failure Group Failure Group 1 Disk 0 Disk 1 Disk 2 Metadata Tertiary Primary Secondary Data Tertiary Primary Secondary

  50. References • Oracle Automatic Storage Management (Oracle Press) • Nitin Vengurlekar • Murali Vallath • Rich Long • What ASM and ZFS Can Do For You • Jason Arneil - Nominet • A Closer Look Inside Oracle ASM • Luca Canali - CERN • Implementing ASM Without HW Raid • Luca Canali - CERN

More Related