Scalability and Reliability using Oracle Real Application Clusters

1. Scalability and Reliability using Oracle Real Application Clusters Suresh Chaganti Syed Asharaf

2. Agenda • Introductions • Real Application Clusters – Defined • Architecture of Real Application Cluster • Memory Structures • Technology Stack Components • Storage Considerations • Pre-Installation Tasks • Installing Real Application Clusters • Case Study I – Benchmark Statistics with Oracle 10g • Case Study II – Migrating to RAC on 11i • Closing Thoughts

3. About Cozent Cozent is Oracle services firm based out of Naperville, IL . Cozent specializes in implementation of Oracle E-Business suite, design of High Availability Solutions using Oracle RAC and support services for Oracle Databases and E-Business Suite.

4. Real Application Clusters • A cluster comprises multiple interconnected computers or servers that appear as if they are one server to end users and applications. • Single-instance • one-to-one relationship between the Oracle database and the instance. • RAC Environments • one-to-many relationship between the database and instances • combined processing power of the multiple servers can provide greater throughput ,scalability and availability • Oracle Clusterware is a portable cluster management solution that is integrated with the Oracle database.

5. Real Application Clusters - Architecture • Two or more database instances that each contain memory structures and background processes • A RAC database is a logically or physically shared everything database • All datafiles, control files, PFILEs, and redo log files in RAC environments must reside on cluster-aware shared disks • At least one additional thread of redo for each instance • An instance-specific undo tablespace • Must have private interconnect between cluster nodes

6. The Real Application Clusters Memory Structures • Each instance has a buffer cache in its System Global Area (SGA). • Using Cache Fusion, RAC environments logically combine each instance's buffer cache to enable the instances to process data as if the data resided on a logically combined, single cache. • The SGA size requirements for RAC are greater than the SGA requirements for single-instance Oracle databases due to Cache Fusion

7. RAC High Availability Components • Voting Disk • Manages cluster membership by way of a health check • RAC uses the voting disk to determine which instances are members of a cluster • must reside on shared disk • Oracle Cluster Registry (OCR) • Maintains cluster configuration information as well as configuration information about any cluster database within the cluster • must reside on shared disk

8. Installing Oracle Clusterware and Real Application Clusters Pre-Installation • user equivalence • network connectivity • directory and file permissions • Use Cluster verification Utility to check requirements Two-phase installation Oracle Clusterware Installation • use Oracle Universal Installer (OUI) to install Oracle Clusterware • Installed in separate home ORA_CRS_HOME Real Application Clusters Installation and Database Creation • Use Oracle Universal Installer (OUI) to install RDBMS • Install RDBMS in separate ORACLE_HOME • Use DBCA to Create and configure Databases

9. Storage considerations • Types of Files to consider • Voting Disk • Oracle Cluster Registry • Shared Database files • Oracle Software • Recovery • Can use any combination of File Systems shown on the right • For each filesystem choice, the disks/devices need to be partitioned and configured. Please see manual for full details

10. Certification Matrix • Leading Edge technology, Pay good attention to Certification Matrix • OCFS2 is not certified yet for Linux AS 4 -64 bit, Itanium, Power and s390x ( at the time of writing this) • Oracle may not have tested ALL Hardware configurations / Vendors

11. Pre-Installation Tasks • Use CVU to confirm that all pre-installation requirements are met • /dev/dvdrom/crs/Disk1/cluvfy/runcluvfy.sh stage -pre crsinst -n node1,node2 • cluvfy -help ( to display help and command info) • Xserver availability ( to run OUI in a GUI mode) • Ensure required Operating groups (dba,oinstall) and users ( oracle,nobody) • Ensure oracle user can ssh to local node and remote nodes without prompting for password. This is the single most important thing to consider • Ensure that date is set on the cluster nodes using NTP or some common mechanism. If the date settings on the nodes vary, the installation may fail during copy of binaries to the remote nodes

12. Pre-Installation Tasks - Continued • Ensure enough RAM , SWAP and TMP space • Installation specific. Refer to installation manual. • 2 Network adapters one each for public connection and Private interconnect between the nodes • Configure /etc/hosts to ensure entries for private, public and vip hostnames for ALL nodes are available in each of the nodes. • Ensure that all rpms required are installed. • Check latest release notes for errata

13. Pre-Installation Tasks - Continued • Configure Kernel Parameters as per installation guide • Create ORACLE_BASE, ORACLE_HOME directories • Configure oracle user’s environment

14. Final Checks Run Cluster Verification utility after completing the pre-installation tasks to ensure that following aspects are covered • Node Reachability: All of the specified nodes are reachable from the local node. • User Equivalence: Required user equivalence exists on all of the specified nodes. • Node Connectivity: Connectivity exists between all the specified nodes through the public and private network interconnections, and at least one subnet exists that connects each node and contains public network interfaces that are suitable for use as virtual IPs (VIPs). • Administrative Privileges: The oracle user has proper administrative privileges to install Oracle Clusterware on the specified nodes. • Shared Storage Accessibility: The OCR device and voting disk are shared across all the specified nodes. • System Requirements: All system requirements are met for installing Oracle Clusterware software, including kernel version, kernel parameters, memory, swap directory space, temporary directory space, and required users and groups. • Kernel Packages: All required operating system software packages are installed. • Node Applications: The virtual IP (VIP), Oracle Notification Service (ONS) and Global Service Daemon (GSD) node applications are functioning on each node.

15. Installing Clusterware and RDBMS • Very straightforward with Oracle Universal Installer if all the pre-installation requirements are met • Need to run in 2 phases, first to install Clusterware and second to install Oracle RAC RDBMS software • Run DBCA to configure ASM instance ( if the storage option chosen is ASM) • Run DBCA to create Database

16. Case Study – I Benchmark Statistics for 10g R2 on RAC Background Client is in healthcare industry analyzing high volume of claims, eligibility information along with transaction processing application for disease management.

17. Before and After RAC Environment before RAC • Custom application running on Single node Windows based Oracle 9i RDBMS • Reporting and Datawarehouse requirements met from same Database • Performance and Scalability issues Environment after RAC • 3 Node 10g R2 RAC system on Linux AS 3 running on 2 way AMD 64 bit processor machines • 9 GB main memory on all 3 nodes • 4.5 GB SGA configured • Instances separated for Production and Datawarehouse • Configured ASM for database Files

18. Benchmark Statistics on Stress Testing • Used TPC-C compliant set of data • Simulates a real workload by allowing a large number of concurrent connections performing user-defined transactions simultaneously. • Used to gauge the server performance under a CPU and memory intensive Oracle database workload • middle 80% of transactions are used for the benchmark readings

19. Test Configuration • executed at several iterations of 5, 10, 25, 50, 100, 150, 200 and 250 sessions • Each iteration executed 100,000 transactions. • Initially executed on a single node at a time • The test was repeated and sessions connected to the three available nodes depending on load.

20. Results • the server handled various database loads optimally and scaled very well. • The three node RAC had a steady database throughput and was able to handle user load up to 800 concurrent users ( Configuration Limit). • Load larger than 800 users resulted in error because of database configuration limit of 800 users. • The database could be altered if needed to increase the user limit.

21. Case Study - IIMigrating to 11i RAC on 9i Database Background Client is in discreet manufacturing industry with plants all over North America. The requirements were for a 24 X 7 environment running on an Oracle 11i (11.5.8) E-Business Suite application

22. Environment and Configuration • Oracle 11.5.8 E-Business Suite application • Oracle database version 9.2.0.7 • Oracle Clustered File System V1 • 4 node environment • 2 Database nodes • 2 Application Server nodes • Concurrent Manager tier on 2 database nodes • Reports and Forms tier on 2 Application Server nodes • Parallel Concurrent Processing configured • Virtual host name configured

23. Key Steps in Implementing RAC on 11i • Configure SAN storage with shared disks • Install and configure OCFS • Install Oracle Cluster Manager • Install Oracle 9i (9.2.0.4) and upgrade database to latest release • Cloned production instance to new servers • Apply all pre-request’s patches to the Cloned Production instance for converting to RAC. ( ex: ATG_PF.H , AD.I etc ). • Converted database to RAC • Configured application tier for RAC • Created database instance 2 on Node 2. • Created application tier 2 on Node 2. • Configured virtual host and configured application to use the virtual host name. • Configured PCP ( Parallel Concurrent Processing )

24. Critical Step – Enabling Autoconfig • Copy the appsutil, appsoui and oui22 directories from the OLD_ORACLE_HOME to the NEW_ ORACLE_HOME. • Set environment variables ORACLE_HOME, LD_LIBRARY_PATH and TNS_ADMIN to point to NEW_ ORACLE_HOME.Set ORACLE_SID variable to point to instance name running on this database node. • Shutdown the instance and database listener. • Start the instance by using parameter file as init<sid.ora>. Start the database listener. • Generate instance specific xml file using NEW_ORACLE_HOME/appsutil/bin adbldxml.sh tier=db appsuser=<APPSuser> appspasswd=<APPSpwd> • Execute the AutoConfig utility (adconfig.sh) on database tier from NEW_ORACLE_HOME/appsutil/bin. Verify the log file located at NEW_ORACLE_HOME>/appsutil/log/<context_name>/<MMDDhhmm

25. Critical Step – Converting database to RAC • Execute AutoConfig utility on the application tier. Verify the AutoConfig log file located at $APPL_TOP/admin/<context_name>/log/<MMDDhhmm>. • Execute $AD_TOP/bin/admkappsutil.pl to generate appsutil.zip for the database tier. • Transfer this appsutil.zip to database tier in the NEW_ORACLE_HOME. • Unzip this file to create appsutil directory in the NEW_ORACLE_HOME. • Execute the AutoConfig on database tier from NEW_ORACLE_HOME/appsutil/<context_name>/scripts by using adautocfg.sh • Verify the AutoConfig log file located in the NEW_ORACLE_HOME NEW_ORACLE_HOME>/appsutil/log/<context_name>/<MMDDhhmm. • Execute the following command to accumulate all the information about the instance NEW_ORACLE_HOME/appsutil/scripts/<context_name>/perl adpreclone.pl database • Shutdown the instance • Ensure that listener process on database tier is also stopped. • Execute the following from the NEW_ORACLE_HOME/appsutil/clone/bin.perl adcfgclone.pl database • Answer the prompted questions

26. Critical Step – Converting Database to RAC ( Contd ) The process will • Create instance specific context file • Create instance specific environment file. • Create RAC parameter specific init.ora file. • Recreate the control files. • Create redo log threads for other instances in the cluster. • Create undo tablespaces for other instances in the cluster. • Execute AutoConfig on the Database tier. • Start the instance and database listener on the local host.

27. Critical Step - Configure Applications Environment for RAC • Execute the AutoConfig by using $AD_TOP/bin/adconfig.sh contextfile=$APPL_TOP/admin/<context_file>. • Verify the AutoConfig log located at $APPL_TOP/admin/<context_name>/log/<MMDDhhmm>for errors. Source the environment by using the latest environment file generated. • Verify the tnsnames.ora, listener.ora files located in the 8.0.6 ORACLE_HOME at $ORACLE_HOME/network/admin and $IAS_ORACLE_HOME/network/admin. Ensure that the correct tns aliases are generated for load balance and fail over. • Verify the dbc file located at $FND_SECURE. Ensure that the parameter APPS_JDBC_URL is configured with all instances in the environment and load_balance is set to ON.

28. Critical Step – Load Balancing Oracle Apps Environment • Run the Context Editor through Oracle Applications Manager interface to set the value of "Tools OH TWO_TASK","iAS OH TWO_TASK" and "Apps JDBC Connect Alias" • To load balance the forms based applications database connections, set the value of "Tools OH TWO_TASK" to point to the <database_name>_806_balance alias generated in the tnsnames.ora file. • To load balance the self-service applications database connections, set the value of iAS OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in the tnsnames.ora file. • Execute AutoConfig by using $AD_TOP/bin/adconfig.sh contextfile=$APPL_TOP/admin/<context_file> • Restart the applications processes by using the latest scripts generated after AutoConfig execution. • Set profile option "Application Database Id" to dbc file name generated at $FND_TOP/secure. • Update session_cookie_name in table ICX_PARAMETERS to <service_name>.

29. Critical Step – Configure Parallel Concurrent Processing Prerequisites for setting up PCP • Configure the Application to use GSM (Generic Service Management), The GSM profile options should be YES Setup PCP • Ensure that all pre-requisite patches are applied. • Edit the applications context file through Oracle Applications Manager interface and set the value of APPLDCP=ON and "Concurrent Manager TWO_TASK" value to instance alias. • Execute AutoConfig by using $COMMON_TOP/admin/scripts/<context_name>/adautocfg.sh on all concurrent nodes. • Source the application environment by using $APPL_TOP/APPSORA.env • Check the configuration files tnsnames.ora and listener.ora located under 8.0.6 ORACLE_HOME at $ORACLE_HOME /network/admin/<context>. Ensure that you have information of all the other concurrent nodes for FNDSM and FNDFS entries.

30. Critical Step – Configure Parallel Concurrent Processing ( Contd ) Setup PCP • Restart the application listener processes on each application node. • Logon to Oracle E-Business Suite 11i Applications using SYSADMIN in login and System Administrator Responsibility. Navigate to Install > Nodes screen and ensure that each node in the cluster is registered. • Navigate to Concurrent > Manager > Define screen, and set up the primary and secondary node names for all the concurrent managers according to the desired configuration for each nodes workload. The Internal Concurrent Manager should be defined on the primary PCP node only. • Set the $APPLCSF environment variable on all the CP nodes pointing to a log directory on a shared file system

31. Closing Thoughts • Lot of choices to be made. Analyze and decide what best fits your environment • Pay close attention to certification matrix, documentation and release notes • Expect some issues with any configuration

32. Thank You ! For any questions/ Comments please contact Suresh.chaganti@cozent.com Syed.asharaf@cozent.com