Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA

1. Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA.com

2. Who Are You? • DBA Experience? • RAC Experience? • ASM Experience? • Linux Server Experience? • Firewire Experience? • Disk Subsystem IO Tuning Experience? • Production v. Dev/Test

3. What Is This Guy Talking About? • RAC • Done with Firewire on Linux • Something you can do without a huge budget • Why Linux? • Firewire Share Locking only works on Linux • Why Not Linux? • Disk IO. More to come…

4. What is RAC? • Multiple Instances (memory) accessing one Database (disk) • Fault-tolerance and Transparency • Performance and Scalability • Cheaper to grow out instead of up • Google Success Story

5. RAC Done Cheap • Low-end RAC deployments cost around $20,000 for the hardware alone • You can build your own test system for 1/10th the cost, using Linux and Firewire • This is not a production-ready solution, but it is good for proof-of-concept and testing

6. RAC Shared Storage • Fiber is the best • Costs more, but you get what you pay for • Fiber speed is up to 4 Gbps now • Supports traditional SCSI protocol • Supports up to 127 nodes

7. RAC Shared Storage (continued) • SCSI is about ½ the cost of Fiber • Less nodes, but likely enough • Less performance, may not be good enough • IPSCSI • Danger, Danger! Who comes up with this? • Better distance, poor performance and reliability • Stay away from NAS/NFS • Reliability and Performance Standards • Who says this is OK?

8. RAC Shared Storage (continued) • Firewire • 1/30th the cost of SCSI • Not supported, not ready for production • Works well for testing and proof-of-concept • Only works with Linux • Special share locking programmed by Wim Coekaerts “Oracle’s Mr. Linux” and his team.

9. RAC Shared Storage (continued) • Transfer Speeds (Max spec in MBps) • You’ll never get Max. You’re lucky to get half. Expect 1/3rd. More information in another presentation. • Serial = 0.014 • IDE = 16.7, Ultra IDE = 33 • USB1 = 1.5, USB2 = 60 • Firewire = 50 • ATA/100 = 12.5, SATA = 150, Ultra SATA = 187.5 • FW SCSI = 20, Ultra SCSI = 40, Ultra3 SCSI = 80, Ultra 160 SCSI = 160, Ultra 320 SCSI = 320 • Gb Fiber = 120, 2Gb Fiber = 240, 4 Gb Fiber = 480

10. Hardware Requirements • Two Intel-based PCs • AMD is OK if you have it, Intel is cheaper • Cheap, old, used, whatever. • Pre-Pentium is fine, but will be slower • VMware is OK. Performance is not as good • Ask about running Linux and Oracle on VMware

11. Hardware Requirements (continued) • 512 MB RAM each. Try hard for 1 GB. • 10 GB internal Hard Drive each • Two network cards each • CD-ROM, monitor, keyboard

12. Hardware Requirements (continued) • Firewire card each • Dual-port Firewire external HDD • Do not connect the firewire drive until the firewire modules are installed and configured correctly! • Cabling for all

13. Install Linux • Oracle is tightly intertwined with RedHat • Jeffrey Hunter did this with WBEL • SuSE and some others work well • Typical/default setup for the most part • Manual Partitioning • /boot = 100 MB • Swap = 150% RAM or 200% if < 1 GB RAM • / gets the rest • Flexibility here, but /usr should not be a mount point

14. Install Linux (continued) • Typical/default setup for the most part (continued) • Separate networks for the two NICs • Interconnect cannot be the same network as the public • Don’t use DHCP • Time sync

15. Install Linux (continued) • Typical/default setup for the most part (continued) • Go ahead, install all packages! • If you don’t use/need it you can always disable it • up2date, up2date, up2date!!! • Oracle has specific package requirements • Don’t do this the hard way

16. Verify Packages • Advanced Server doesn’t install everything • You may have chosen not to install everything • Find your requirements • There are minimum (and in some cases exact) package version requirements for RAC. Be sure to follow carefully • See the links to RAC documentation on the slide titled “More Information”

17. Firewire Modules • http://oss.oracle.com/projects/firewire • Choose the correct module for your system • Download and install it (rpm –ivh) • Add to /etc/modprobe.conf options sbp2 exclusive_login=0 alias scsi_hostadapter sbp2 • Once this is done on both machines you can power them down, connect the firewire device and restart them

18. Firewire Modules (continued) • lspci |grep –i firewire should return one line • lsmod |egrep “ohci1394”|sbp2|ieee1394|sd_mod|scsi_mod” should return one line each • fdisk • /CRS = 300 MB • /oradata (usually like to have multiples) • /orabackup

19. Hangcheck-timer Module • http://oss.oracle.com/projects/hangcheck-timer/ • Hangcheck tick setting • Time between system health checks • Default is 60 (seconds) • Oracle recommends 30 • Maybe 10 or 15 (overhead v. discovery)

20. Hangcheck-timer Module (continued) • Hangcheck margin setting • Time of tolerated delay before node reset • Default and Oracle recommendation is 180 (seconds) • Maybe 60 (quick reset v. delayed reset) • Add to /etc/modprobe.conf options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

21. More Kernel Settings • Set via sysctl (sysctl –w net.core.) • rmem_default, rmem_max, wmem_default and wmem_max = 262144 • sysctl –w kernel. • shmmax=2147483648 (2 GB) • sem=“250 32000 100 128” • sysrq = 0 • core_uses_pid = 1

22. More Kernel Settings (continued) • sysctl –w fs. • file-max = 65536 • Sysctl –w net.ipv4 • ip_forward = 0 • Sysctl –w net.ipv4.conf.default • rp_filter = 1

23. Oracle Users and Groups • Create the oracle user and the dba and oinstall groups • UIDs and GIDs must be the same on both machines • Create a software install directory on both machines’ internal drives (e.g. /opt/oracle)

24. Oracle Users and Groups (continued) • Profile • ORACLE_BASE and ORACLE_HOME • ORACLE_CRS_HOME • ORACLE_SID (each node is unique) • PATH=$ORACLE_HOME/bin:$ORACLE_BASE/common/oracle/bin:$PATH:$/usr/bin:/bin:/usr/X11/bin:/usr/local/bin • LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib • CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib • THREAD_FLAG=native (use OS Java version)

25. Introducing Cache Fusion • Oracle Parallel Server Disk Writes • RAC High Speed Interconnect • Performance Gains • Outward Scalability • Performance Losses • Overhead for Cache Fusion

26. The Network • Hard-set, do not use auto-negotiate, auto-detect, auto-sense, auto-*, auto-bad • Gigabit Hub for the interconnect • We use crossover in the test. • Don’t mess with Cache Fusion! • Not supported

27. The Network (continued) • Set up a virtual IP address in hosts and the name server • Only need name/IP • Must be the same network as the public interface • Remove the linux name alias on the loopback line and put it on the right line

28. Firewall Configuration • Allow incoming SSH • Key-based access only? • Don’t allow root/oracle remotely? • Allow incoming TNS • Allow all outgoing • Allow traffic related to established sessions

29. Firewall Configuration (continued) • Log • All dropped? • All allowed? • Port specific? • SE Linux • Start with warn mode to see what you want to do • More specific than the firewall, but the firewall is usually good enough

30. Configuring SSH • Used for • Remote administration • RAC software deployment • Create a key pair for root and oracle on each node • ssh-keygen –t dsa –N “” • Accept the defaults • cd ~/.ssh • scp id_dsa.pub <other_node>:./.ssh/<this_node>.pubkey • NOTE: Password is required for this scp because the key is not in place yet

31. Configure SSH (continued) • On the other node • cd ~/.ssh • cat <other_node>.pubkey >> authorized_keys • Make the initial connection to accept the key familiarity question • Software installation will error if it is presented with unexpected text

32. Introducing OCFS • Oracle Cluster File System • Used for RAC on Linux and Windows • Eliminates the need for third-party clusters • Can store all RAC-related and Database files

33. Install OCFS • http://oss.oracle.com/projects/ocfs-tools • http://oss.oracle.com/projects/ocfs • OCFS v. ASM • ASM can store datafiles, OCFS can store everything • OCFS uses DBA-friendly, filesystem-style datafiles • Just download and install the RPMs • ocfs2console (this is the ocfs2 version) • OCFS2/RHEL4 not supported, but it works! • Select cluster > configure nodes • Add the name/IP of the public interfaces

34. Install OCFS (continued)

35. Configuring OCFS for Autostart • Fix the chkconfig bug (OCFS2) • Remove the lines between (and including) ‘### Begin Init Info’ and ‘### End Init Info’ in /etc/init.d/o2cb • Reregister • chkconfig --del o2cb • chkconfig --add o2cb • Reload • /etc/init.d/o2cb offline ocfs2 • /etc/init.d/o2cb unload • /etc/init.d/o2cb configure

36. Configure OCFS File Systems • Back to ocfs2console • Tasks > format • 4K Block Size. Get it right now, it can’t be changed. • Cluster Size Setting • 32K or 64K for OCR. • 128K or 256K for OLTP Data Files. • 512K or 1M for Data Warehouse Data Files. • 1M for backups • Reboot both nodes • OCFS FS should be mounted on both

37. Configure OCFS File Systems (continued)

38. Get the RAC Software • Oracle Database Server • Oracle Companion Software • Oracle Cluster (version dependent) • Oracle Clusterware • Oracle Cluster Ready Services

39. Install Oracle Cluster Software • Unset environment variables • ORA_CRS_HOME • ORACLE_HOME • runInstaller • Use the ORA_CRS_HOME • Use a cluster name that doesn’t match either SID • Mirror the OCR and voting Disks! (10gR2)

40. Install Oracle Cluster Software (continued) • Update CSS miscount • Timeout before evicting a node • Can cause installs to hang • Set it higher for slower disks • Recommend 60 for most disks, 480 for firewire • Edit root.sh before running it on each node

41. Install Oracle Cluster Software (continued) • Run root.sh • Run on each node, one at a time • Takes a while to complete • Get several warnings. Ignore them. • May get a critical alert regarding virtual IP addresses at the end on the last node • Run the vipca GUI and set the interface information • Verify • $ORA_CRS_HOME/bin/olsnodes –n • Should see all nodes in the cluster

42. Install the DB Server • Unset environment variables • ORA_CRS_HOME • ORACLE_HOME • runInstaller • Use ORACLE_HOME • After specifying the ORACLE_HOME the next screen should be a clustered install screen • Do not proceed if this is not the case. Something is wrong with the cluster. Fix it first!!!

43. Install the DB Server(continued) • runInstaller (continued) • Select all nodes • Choose install Software only. Use dbca later, not as part of the initial installation • Run root.sh on all nodes, one at a time

44. Create the Listener Service • Run this from only one node in the cluster • Netca • Choose cluster configuration • Select all nodes • Use different names for each node’s service

45. Install Companion Software • NCOMP is required or you will hit a bug (10.1.0.3 – 10.1.0.4) • runInstaller • Use ORACLE_HOME • Specify Hardware Cluster Installation

46. Transparent Application Failover (TAF) • Available in 10g. Use it!!! • Automatically migrate sessions from a failed instance to a working instace • Session continues, uninterrupted to the user • Requires a simple modification to your configuration • Should be selected during DB creation

47. Create the DB • Run dbca on ONE node • Should be presented with a RAC welcome screen • Stop! If you don’t get the RAC welcome your cluster is not running or configured properly. Fix the cluster before you create the DB • Install on all nodes • Choose a “custom” installation unless you have already created a RAC template that uses TAF • Add a TAF Service

48. Configure TAF • Just update the tnsnames.ora files ORCL1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl2)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcltest.idevelopment.info) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) )

49. Connecting to the DB • You may connect to • An instance on either node • Transaction aborted when that instance dies • The DB service • Oracle chooses an instance to connect you to • Transaction aborted when that instance dies • The TAF DB service • Oracle chooses an instance to connect you to • Instance failure is transparent

50. Test TAF • Connect to the TAF Service with SQL • Select instance_name from v$instance; • Shut that instance down • Your session is still connected • Select instance_name from v$instance;