Agenda

1. Agenda • ExadataMatchine Overview • Exadata Hardware Architecture • Exadata Software Architecture • Oracle Exadata Terms • Exadata Storage Server Overview • Exadata Database Machine Overview • Infiniband Overview • Key Capabilities of Exadata Database Machine • Patching • Backup and Recovery for Exadata Database Machines • Migrating databases to Exadata database machin • Basic Commands

2. Exadata Machine Overview Oracle Exadata Machine is engineered system which is specially designed for Oracle Database on cluster architecture supported by Oracle RAC. There is no generic configuration. All component s of Exadata are fully tuned as per exadata requirement. It is combination of hardware and software which provide esteemed performance, scalability and availability. It is introduced in September 2008. In exadata, Private Interconnect is not used. It is accomplished by infiniband . Oracle Provide different version of Exdata Machine V2,X2,X3,X4,X5,X6 and X7 Advantage of Exadata • Exadata is well-suited for consolidating multiple databases onto single Exadata system. • It is used for Data warehouse( where large and complex queries are used ) and OLTP environment( supporting large transaction with quick and consistent response time) • It is delivered with complete package of Software,Servers,Network and Storage • There is single point of accountability, meaning that issues will never remain unresolved Since there is no need to wait for other vendors

3. Key Components include : • Powerful and intelligent shared storage provide by Exadata Storage server • High bandwidth and low latency cluster interconnect and storage networking provided using InfiniBand • Powerful and well-balanced database servers joined together in cluster Some of other hardware and software features are: Hardware Components 1. Cisco network switch 2. KVM 3. PDU 4. Rack 5. Flash Cache Software Components 1. Storage Software 2. Operating System (Linux/Solaris) 3. Oracle Database Software (11gr2,12c)

4. Exadata Hardware Architecture

6. Database Server – The Exadata Database Machine runs Oracle Database 11g/12c RAC .The cluster and the database run on the servers known as database server or compute server . A full rack has 8 Database server running Oracle Linux or Oracle Solaris. • Storage Server - The disks are not attached to the database server , as is normally the case with the direct attached storage, but rather to a different server known as the storage server (or just “cell”; there are 14 storage in a full rack). The Oracle Exadata Server Software runs in these cells on top of the OS. • Disks – Each cell has 12 disks. Depending on the configuration, these disks are either 600GB high performance or 2TB high capacity (GB here means 1 billion bytes, not 1024MB). You have a choice in the disk type while making the purchase. Flash disks – Each cell has 3.2 tb t of flash disks in High Capacity Storage server and Extreme Flash Storage server. These disks can be presented to the compute nodes as storage (to be used by the database) or used a secondary cache for the database cluster (called smart cache).

7. Infiniband Network – the cells and nodes are connected through infiniband for high speed and low latency. There are 3 infiniband switches for redundancy and throughput. Note: there are no fiber switches since there is no fiber component. • Ethernet switch – the outside world can communicate via infiniband, or by Ethernet. There is a set of Ethernet switches with ports open to the outside. The clients may connect to the nodes using Ethernet. DMAs and others connect to the nodes and cells using Ethernet as well. Backups are preferably via infiniband but they can be done through network as well. • KVM switch – there is a keyboard, video, and mouse switch to get direct access to the nodes and cells physically. This is used initially while setting up and when the network to the system is not available. In a normal environment you will not need to go near the rack and access this KVM, not even for powering on and off the cells and nodes. Why not? You’ll learn why in the next installment. (Not all models have a KVM switch.)

8. Types of Exadata Machine The Exadata Machine comes as a Full Rack, Half Rack or Quarter Rack , Eighth Rack. Full Rack Database Machine- It has 8 Database Servers and 14 Exadata storage servers . Half Rack Database Machine- It has exactly half the capacity. It has 4 Database Servers and seven Exadata storage servers . It has 256 GB of memory per database server node. Quarter Rack Database Machine -It consists of several virtual machines configured to provide two database servers and three Exadata Storage Servers and two infiniband switches Eighth Rack Database Machine – Just like a Quarter Rack, It contain two database servers and three Exadata Storage Servers and two infiniband switches. However ,in an Eighth Rac ,half of database server CPU cores are disabled

9. Types of Exadata Machine

10. Exadata Software Architecture

11. Restart server (RS) and Management Server (MS) run on both Database and storage server The Exadata Database Machine software architecture include • Exadata Database Servers use Oracle Linux x86_64 as operating system • Exadata Database Servers run 11g_r2 and 12c r1. • ASM is required and provides a file system and volume manager. • Oracle Database communicate with Exadata cells by using a special library called LIBCELL. It will communicate with Exadata cells by using iDB. • The Database Resource Manager (DBRM) is integrated with Exadata Storage server I/O resource Manager(IORM). DBRM and IORM work together to ensure that I/O resources are allocated based on administrator-defined priority. • Diskmon checks the storage network interface state and cell liveness. • Management Server (MS) on Database server provides set of management and configuration functions. It work in cooperation with the DBMCLI command-line interface . • Restart Server (RS) - It is used to start up and shut down the MS on Database server and monitor MS to automatically restart it if require.

12. The Exadata Storage Machine software architecture include • Exadata Storage Servers use Oracle Linux x86_64 as operating system • Restart Server (RS) - It is used to start up and shut down the Cell Server (CELLSRV) on storage server To check ,run below command on storage server [celladmin@qr01celadm01 ~]$ ps -ef | grep cellrs • Management Server (MS). It also monitors these services to check if they need to be restarted. MS provides Exadata cell management and configuration. It works in cooperation with the Exadata cell command-line interface (CellCLI). In addition, MS is responsible for sending alerts and collects some statistics in addition to those collected by CELLSRV. To check ,run below command on storage server [celladmin@qr01celadm01 ~]$ ps -ef | grep msServer

13. CELLSRV is the primary Exadata software component and provides the majority of Exadatan storage services. CELLSRV is a multithreaded server. Primarily, CELLSRV communicates with Oracle Database to serve simple block requests, such as database buffer cache reads, and Smart Scan requests, such as table scans with projections and filters. CELLSRV also implements I/O Resource Management (IORM) and collects numerous statistics relating to its operations. To check ,run below command on storage server [celladmin@qr01celadm01 ~]$ ps -ef | grep "/cellsrv “ • Cell and Grid Disk are a logical component of the physical Exadata storage. • A cell or Exadata Storage server cell is a combination of Disk Drives put together to store user data.

14. Each Cell Disk corresponds to a LUN (Logical Unit) which has been formatted by the Exadata Storage Server Software. Typically, each cell has 12 disk drives mapped to it. • Grid Disks are created on top of Cell Disks and are presented to Oracle ASM as ASM disks. Space is allocated in chunks from the outer tracks of the Cell disk and moving inwards. One can have multiple Grid Disks per Cell disk. In Exadata, a LUN (Logical Unit) is a logical abstraction of a storage device. LUNs are based on hard disks and flash devices. LUNs are automatically created when Exadata is initially configured. Each Exadata cell contains 12 hard disk-based LUNs along with 4 flashbased LUNs. List the LUNs on your primary Exadata cell/Storage Server. CellCLI> list lun

15. A cell disk is a higher level storage abstraction. Each cell disk is based on a LUN and contains additional attributes and metadata. Examine the attributes for the cell disk-based on the LUN. CellCLI> list celldisk CD_09_qr01celadm01 detail • A grid disk defines an area of storage on a cell disk. Grid disks are consumed by ASM and are used as the storage for ASM disk groups. • Each cell disk can contain a number of grid disks. Examine the grid disks associated with the cell disk step. Note the names and sizes of the grid disks. CellCLI> list griddisk where celldisk=CD_09_qr01celadm01 detail • By default, Exadata Smart Flash Cache is configured across all the flash-based cell disks.Use the LIST FLASHCACHE DETAIL command to confirm that Exadata Smart Flash Cache is configured on your flash-based cell disks. • CellCLI> list flashcache detail

16. In addition to Exadata Smart Flash Cache, Exadata Smart Flash Log provides a • mechanism for improving the latency of database redo log write operations. • Exadata Smart Flash Log uses a small portion of high-performance flash memory as temporary storage to • facilitate low latency redo log writes. • By default, Exadata Smart Flash Log uses a total of 512 MB on each Exadata Storage Server; • Use the LIST FLASHLOG DETAIL command to examine the Exadata Smart Flash Log area on this cell. • CellCLI> list flashlog detail

17. Oracle Exadata Terms • EHCC - Exadata Hybrid Columnar Compression • CU - Compression Unit • PDU - Power Distribution Unit • IB - InfiniBand • ILOM - Integrated Lights Out Manager • IPMI - Intelligent Platform Management Interface • LUN - Logical Unit Number • HARD - Hardware Assisted Resilient Data • RDS - Reliable Data Socket protocol • iDB - Intelligent Database protocol • UDMs - User Defined Metrics • IOPS - IOs Per Second • OEL - Oracle Enterprise Linux • ULN- Unbreakable Linux Network

18. SNMP - Simple Network Management Protocol • MS - Management Server • RS - Restart Server • EM - Enterprise Manager • EMGC - Enterprise Manager Grid Control • HCA - Host Controller Adapter • HBA - Host Bus Adapter • SM - Subnet Manager • DBRM - Database Resource Manager • SATA - Serial Advanced Technology Attachment • SAS - Serial Attached SCSI • OFED –Open Fabrics Enterprise Distribution • PSU – Patch Set update • CPU – Critical patch update • BP – Bundle Patch

19. Exadata Storage Server • Exadata Storage Server is highly optimized storage for Oracle Database. • It delivers outstanding I/O and SQL processing for database. • A Single Storage server is also called a Cell. A Cell is building block for a Storage grid. • Each cell has OS(linux x86_64), CPUs,Memory,a bus,disks and network adapters. • There is two of exadata storage server 1) High Capacity Storage server-The maximum SQL bandwidth for a full Rack database machine(14 cells) is 25GB/s 2) Extreme Flash Storage server-The maximum SQL bandwidth for a full Rack database machine(14 cells) is 263GB/s

20. Advantage of Storage Server • Database can offload some database processing on storage server which is called Smart Scan. • It is highly optimized for fast processing of large queries • It is intelligent to use high performance flash memory to boost performance • It use InfiniBand network for higher throughput • It support Hybrid Columnar compression which provide high level of data compression • It manage I/O resource through IORM • It uses ASM to evenly distribute the storage load for every database • We can assign dedicated storage to a single database . Shared storage is not a perfect solution. Running multiple types of workloads and databases on shared storage often leads to performance problems. large parallel queries on one database can impact the performance of critical queries on another database. Also, a data load on an analytics database can impact the performance of critical queries running on it.

23. Disk Layout-The disk layout needs some additional explanation because that’s where most of the activities occur. The disks are attached to the storage cells and presented as logical units (LUNs), on which physical volumes are built.  Each cell has 12 physical disks. In a high capacity configuration they are about 8TB and in a high performance configuration, they are about 3.2GB each. The disks are used for the database storage. Two of the 12 disks are also used for the home directory and other Linux operating system files. These two disks are divided into different partitions as shown in Figure 2 below.

25. The physical disks are divided into multiple partitions. Each partition is then presented as a LUN to the cell. Some LUNs are used to create a filesystem for the OS. The others are presented as storage to the cell. These are called cell disks. The cell disks are further divided as grid disks, ostensibly referencing the grid infrastructure the disks are used inside. These grid disks are used to build ASM Diskgroups, so they are used as ASM disks. An ASM diskgroup is made up of several ASM disks from multiple storage cells. If the diskgroup is built with normal or high redundancy (which is the usual case), the failure groups are placed in different cells. As a result, if one cell fails, the data is still available on other cells. Finally the database is built on these diskgroups.

26. Exadata utility • Most administration functions are performed using the Exadata cell command-line interface (CellCLI). • CellCLI can be used only from within a cell to manage that cell.CellCLI works in conjunction with the Exadata Storage Server Management Server (MS). • CellCLI provides the command interface while MS performs the administrative functions, such as creating and dropping grid disks. • [celladmin@exa1cel01 ~]$ cellcli CellCLI: Release 12.1.2.1.0 - Production ... CellCLI> list cell exa1cel01 online There is three default administrative user account on exadata storage server 1) root 2) celladmin 3) cellmonitor

27. ExaCLI is same as CellCLI; however, the main difference is that ExaCLI manages cells from a remote host , typically an Exadata database server, whereas CellCLI runs directly on a cell. ExaCLI enables users to perform most management functions without the requirement to establish an SSH connection to the cell. It communicate through https. • All communication between ExaCLI and Management Server (MS) running on the cell is over https. • The exadcli utility allows you to simultaneously execute a command on multiple Exadata Storage Servers: Command types: ExaCLI commands, ExaCLI scripts • The dcli utility allows you to simultaneously execute a command on multiple Exadata servers: Command types: Operating system commands CellCLI commands Operating system scripts CellCLI scripts

28. Configuring Hosts to Access Exadata Cells Configuration files on each database server enable access to Exadata storage. • cellinit.ora identifies the storage network interfaces on the database server. • cellip.ora identifies the Exadata cells that are accessible to the database server. • The cellinit.ora file contains the database server IP address that connects to the storage network. This file is host specific, and contains the IP addresses of the InfiniBand storage network interfaces for that database server. The IP addresses are specified in Classless Inter-Domain Routing (CIDR) format. • The cellip.ora file contains the IP addresses for the InfiniBand storage network interfaces of the Exadata Storage Servers that are accessible to the database server. • To ensure that ASM discovers Exadata grid disks, set the ASM_DISKSTRING initialization parameter. A search string with the following form is used to discover Exadata grid disks: • o/<cell IP address>/<grid disk name> • Wildcards may be used to expand the search string. For example, to explicitly discover all the available Exadata grid disks set ASM_DISKSTRING='o/*/*'. To discover a subset of available grid disks having names that begin with data, set ASM_DISKSTRING='o/*/data*'.

29. Bear in mind the following general considerations when reconfiguring Exadata storage: • Reconfiguring an existing disk group requires the ability to drop disks from the disk group, reconfigure them and then add them back into the disk group. If the amount of free space in the disk group is greater than the REQUIRED_MIRROR_FREE_MB value reported in V$ASM_DISKGROUP, then you can use methods which reconfigure the diskgroup one cell at a time. If the free space is less than REQUIRED_MIRROR_FREE_MB,then you may need to reorganize your storage to create more free space. It may also be possible, though not recommended, to reconfigure the storage one disk at a time. • Best practices recommend that all disks in an ASM disk group should be of equal size and have equal performance characteristics. For Exadata this means that all the grid disks allocated to a disk group should be the same size and occupy the same region on each disk. There should not be a mixture of interleaved and non-interleaved grid disks, likewise there should not be a mixture of disks from high-capacity cells and high- performance cells. Finally, the grid disks should all occupy the same location on each disk.

30. If you try to drop a grid disk without the FORCE option the command will not be processed and an error will be displayed if the grid disk is being used by an ASM disk group. If you remove a disk from an ASM disk group ensure that the resulting rebalance operation completes before attempting to drop the associated grid disk.If you need to use the DROP GRIDDISK command with the FORCE option, use extreme caution since incorrectly dropping an active grid disk could result in data loss.If you try to drop a cell disk without the FORCE option the command will not be processed and an error will be displayed if the cell disk contains any grid disks. It is possible to use the DROP CELLDISK command with the FORCE option to drop a cell disk and all the associated grid disks. Use the FORCE option with extreme caution since incorrectly dropping an active grid disk could result in data loss. • Clusterware files (cluster registry and voting disks) are stored by default in a special ASM disk group named DBFS_DG. Resizing the DBFS_DG disk group is generally not recommended since the grid disks associated with it are sized specially to match the size of the system areas on the first two disk in each cell. If there is a requirement to alter this disk group, or the underlying grid disks or cell disks, special care must be taken

31. to preserve the clusterware files. Reconfiguring Exadata storage on an active system without any downtime is possible,however doing so can be a time-consuming process involving many ASM rebalancing operations. The time required depends on the number of storage cells, the existing disk usage and the load on the system.

32. Exadata Database Machine

33. Full Rack Database Machine Here X6-2 mean X6 is version and 2 is model and 2 means two socket Eg:

34. InfiniBand Network • It is reliable and high speed storage network. It is also used for Interconnect (There is no need of private IP ) • It is also used for Backup server connectivity ,ETL server and Middleware server • From X4 server , Active Bonding in infiniBand is also possible • Its speed is 10GB/second

35. InfiniBand Network

37. Status • To get the status of the Infiniband services. First to check the status of the infiniband devices, use the ibstatus command.# ibstatusInfiniband device 'mlx4_0' port 1 status:        default gid:     fe80:0000:0000:0000:0021:2800:01a0:fd45        base lid:        0x1a        sm lid:          0xc        state:           4: ACTIVE        phys state:      5: LinkUp        rate:            40 Gb/sec (4X QDR) • Infiniband device 'mlx4_0' port 2 status:        default gid:     fe80:0000:0000:0000:0021:2800:01a0:fd46        base lid:        0x1c        sm lid:          0xc        state:           4: ACTIVE        phys state:      5: LinkUp        rate:            40 Gb/sec (4X QDR)… output truncated …

38. How to manually reboot an Infiniband Switch • Do NOT reboot both Exadata Switches at the same time -- you'll get into lots of trouble • Each Exadata component has a dedicated ILOM that can be accessed by a dedicated IP address. but not the IB Swicthes; an IB Switch ILOM is embedded within the Switch itself and has to be accessed using the ILOM shell with the "spsh" command and then use the "reset /SP" command to reboot the Switch as shown below

39. Key Capabilities of Exadata Database Machine • Smart Scan • Smart Flash Cache • Smart Flash log • Storage index • I/O Resource Management • Hybrid Columnar Compression

40. Exadata Smart Scan • It offloads I/O operations to storage servers .It filters rows and columns at storage server and return relevant rows to user and It reduces storage interconnect traffic by eliminating unneeded rows and columns, improving query response times while reducing total CPU and memory requirements on the database server.Smart Scan optimization is a runtime decision. it is not integrated with oracle optimizer. however it is influenced by the results of query optimization. Smart Scan includes: • Full Table and Fast Full Index Scans: Scans are performed inside Exadata Storage Server, rather than transporting all the data to the database server. • Predicate filtering: Only the requested rows are returned to the database server, rather than all the rows in a table. • Column filtering: Only the requested columns are returned to the database server, rather than all the table columns. • Join filtering: Join processing using Bloom filters are offloaded to Exadata Storage Server. • Exadata Smart Flash Cache provides a caching mechanism for frequently accessed data on each Exadata cell. • By default, Exadata Smart Flash Cache operates in writethrough mode; • however, Exadata Smart Flash Cache can also be configured to operate in write-back mode.

42. In addition to the query-specific requirements, the following general requirements must also be met to enable Smart Scan: • Smart Scan must be enabled within the database. The CELL_OFFLOAD_PROCESSING initialization parameter controls Smart Scan. The default value of the parameter is TRUE, meaning that Smart Scan is enabled by default.Each segment being scanned must be on a disk group that is completely stored on Exadata cells. The disk group must also have the following disk group attribute settings: 'compatible.rdbms' = '11.2.0.0.0' (or later) 'compatible.asm' = '11.2.0.0.0' (or later) 'cell.smart_scan_capable' = 'TRUE‘ • V$SQL • SQL_TEXT • PHYSICAL_READ_BYTES • PHYSICAL_WRITE_BYTES • IO_INTERCONNECT_BYTES • IO_CELL_OFFLOAD_ELIGIBLE_BYTES • IO_CELL_UNCOMPRESSED_BYTES • IO_CELL_OFFLOAD_RETURNED_BYTES • OPTIMIZED_PHY_READ_REQUESTS

43. The following are common wait event SELECT s.name, m.value/1024/1024 MB FROM V$SYSSTAT s, V$MYSTAT m WHERE s.statistic# = m.statistic# AND(s.name LIKE 'physical%total bytes' OR s.name LIKE 'cell phys%‘ OR s.name LIKE 'cell IO%'); • cell interconnect retransmit during physical read- Database wait during retransmission for an I/O of a single-block or multiblock read • cell list of blocks physical read - Cell equivalent of db file parallel read • cell single block physical read - Cell equivalent of db file sequential read • cell multiblock physical read - Cell equivalent of db file scattered read • cell smart table scan - Database wait for table scan to complete • cell smart index scan - Database wait for file creation operation • cell smart incremental backup - Database wait for incremental backup operation • cell smart restore from backup - Database wait during file initialization for restore

44. Hybrid Columnar Compression • Hybrid Columnar Compression reduces disk space usage and typically yields faster query times as fewer blocks are required to satisfy the same operation. • Hybrid Columnar Compression is possible at table,partition and tablespace level. • Data is organized into sets of rows called compression unit. within compression unit, Data is organized by column and then compressed .The column organization of data brings similar value close together , enhancing compression ratios. Each row is self –contained within a compress unit. • Candidate big Table with Hybrid Columnar Compression: • Data must be loaded via direct path read • Compressed tables should expect very little DML

45. Hybrid Columnar Compression

46. Smart Flash Cache • Smart Flash Cache reduces physical disk I/O by caching data in PCI flash, which offers lower I/O latency than physical disks. • Exadata Storage server provides Exadata Smart Flash • Exadata Smart Flash Cache provides a caching mechanism for frequently accessed data(data and index block) on each Exadata cell. • By default, Exadata Smart Flash Cache operates in writethrough mode,however, Exadata Smart Flash Cache can also be configured to operate in write-back mode. • writethrough mode is best suited to random repeated reads commonly found in OLTP application. • write-back mode is best suited to write intensive applications. • It uses internal statistics and other measures to determine whether or not an object (table or index) should be cached .it takes into account the size of the object and the frequency of access to the object.

47. Smart Flash Cache

48. Exadata Smart Flash Log • Exadata Smart Flash Log provides a mechanism for improving the latency of database redo log write operations. Exadata Smart Flash Log uses a small portion of the high-performance flash memory on Exadata Storage Server as temporary storage to provide low latency redo log writes. By default, Exadata Smart Flash Log uses a total of 512 MB on each Exadata Storage Server. The default Exadata Smart Flash Log can be dropped and recreated using a different size. However, the size of the default Exadata Smart Flash Log is sufficient for most uses. Exadata Smart Flash Log is managed automatically by Exadata Storage Server.

49. Exadata Smart Flash Log

50. Storage Indexes • A storage index is a memory-based structure that reduces the amount of physical I/O in a the exadata cell when accessing data via Exadata Smart Scan. A storage index keeps track the minimum and maximum column values and information is used to avoid useless I/Os. • By default, storage indexes are created and maintained automatically by Exadata's cell services software • Storage indexes is a unique whose primary goal is to reduce the amount of I/O required to service I/O requests for Exadata Smart Scan. • The storage space inside each cell disk is logically divided into 1MB chunks called storage regions