Red Rock Consulting Ten RAC lessons You Can’t Do Without Presented By John-Paul Tocker

1. Red Rock ConsultingTen RAC lessons You Can’t Do Without Presented By John-Paul Tocker

2. Agenda • Introduction • Overview of Oracle RAC • The Cost of RAC • The Difference between RAC and Dataguard • The RAC Stack • Interconnect, bigger is better. • Questions you should ask yourself. • Top 10 lessons learnt • Questions

3. Who Is Red Rock Consulting • Oracle Partner and Re-seller • DBA / Middleware Consulting and Out-Task • Support Oracle, SQL Server, and Sybase • Also Java and Oracle Application Express custom development • Also Oracle e-Business Suite, Peoplesoft, JDE • 50 people in Wellington and Auckland • Overall, 350 in Australasia

4. Introduction • The typical New Zealand business is not on scale with a World Wide enterprise that would that would deploy a large scale geographically separated RAC environment. • Most RAC projects overrun budget and timeframe. • Lessons learnt and what best practises can help you with your implementation.

5. The Sales Pitch Could a 1976 Lada Tow your new Yacht to Auckland?

6. To RAC or Not to RAC • Did you identify and research your requirements before you went shopping? • How did you specify your requirements? A solution just to get the job done or a robust system that will allow for growth and change. • What about a proof of concept? Or real working builds? • Engage or find out who is your local Oracle sales representative, they have access to the tools and documentation to help you through this process. • RAC is not Bullet Proof!

7. Get Involved In The Project • Get Involved Early on. Identify the misconceptions “management” often have of Oracle RAC. • RAC is an Enterprise Decision and must involve people from all corners of your technology shop. • Storage Solutions – ASM or Third Party Clustered file system making the right choice. • Network – Identify the requirements of the Interconnect and extra requirements for RAC such as VIP’s and Public IP’s. • Infrastructure – HP OTB Intel Based Blade Servers or Large Sun boxes for example. • Identify Requirements for the Database – Goes with out saying. • Application – Has it been written for RAC? Is the application certified for RAC

8. The Butterfly Effect … Not The Movie • The Butterfly Effect. Decisions made early on without consideration to the Database design will have large ramifications later on in the project. • Your Database Solution should reflect the following considerations • Hardware • Storage • Availability • Redundancy • Network • Support

9. The RAC Puzzle • The build process can take more than a week • Your disk partitions must be presented before the build process. • ASM, Veritas, GPRS or OCFS2 perhaps • The interconnect • Public and Private IP’s configured • ssh configured on all nodes • What about the voting disk? Not to mention the OCR, where and how are we storing those?? • Satisfy CLUVFY • RAC specific failover and performance testing.

10. Dataguard Or RAC ? • RAC is scalable, Dataguard isn’t. • RAC is a complete solution where Dataguard can be dumped anywhere on any hardware • Cost factor $$$$ • Dataguard is a redundant copy, But now can be used for reporting with 11g active Dataguard. • RAC by itself is not a DR solution.

11. Node 1 Node 2 Instance 1 Instance 2 Interconnect LocalDisk Shared Storage LocalDisk Scaling RAC Node 3 Instance 3 Add node here Add node here

12. What The Documentation Doesn’t Say • Transparent Application Failover (TAF) is semi transparent. Applications must handle the error message. • Rolling Patches aren't all ways rolling. Any changes to the catalogue mean you will require an outage! • Your Database still needs outages for Maintenance. • RAC is not a guaranteed performance increase over a single instance database. • RAC is not a DR solution.

13. The Real Cost of RAC • Hardware • Interconnect • Licensing • Support • Development costs

14. The Real Cost of RAC • Does your Database meet the required business SLAs • If you don't go down the Oracle RAC path and more processing power is required down the track re-platforming maybe be more expensive in the long run...... • The majority of RAC incidents and outages are caused by the operator or DBA. • With complexity comes risk. • Big factor to consider here….. • Does the high cost of Oracle RAC offset the costs of failing to meet SLA’s or down time?

15. What About Standard Edition • Database for servers with up to four sockets. • It is also upwardly compatible with Enterprise edition and can easily grow with you, protecting your initial investment. • Database files must use ASM • Must use Oracle Clusterware • Shared RAW partitions must be used to store the Oracle Clusterware configuration files, CRS and Voting Disk. • Third party volume managers and file systems are not supported for this purpose. • Could save as much as $200,000! And 30k per year in support. • Metalink note 271886.1 and 465465.1

16. K.I.S.S. • Keep it Simple Stupid  • Why add more pieces to the puzzle? • Increase support requirements and troubleshooting managing several vendors to congregate which nodes are being ejected = Headaches! • Follow the Crowd!!! Choose what is common. • ASM, CRS on a well supported Platform!!! • OTB even!

17. Choosing Your RAC Stack • Choosing your RAC configuration…… • ASM • VERITAS • QCFS • GPRFS • Sun Cluster • QFS • Should you go with Oracle for the full stack? I don’t see why not? • Remember the more pieces you add to the puzzle the more grey areas you will have to manage.

18. Sun Clustering • You may however need to install Sun Cluster in the following scenarios: • If there is application running with in the cluster along with Oracle RAC database that you want to configure for HA and Sun Cluster provide the cluster resourced (easy to use) to manage and monitor the application. THIS can be achieved with Oracle Clusterware but you will have to write your own cluster resource for that. • If you want to install cluster file system such as QFS then you will need to install the Sun Cluster. If this cluster is only running the Oracle RAC database then you can rely on Oracle technologies such as ASM, raw devices without installing Sun Cluster. • Any certification conflicts. • If your application requires a shared filesystem.

19. RAC Certification • Certify your Hardware Stack with Oracle and your Vendors. • Certify your application. • Ensure your hardware has a life time to see you well into the future.

20. The Interconnect • Crossover cable not accepted. • Make your network admin RAC aware. • Know what traffic is on your Interconnect. • Gigabit should be a minimum. • Try and get a base line for performance on the Interconnect • Fault tolerant network

21. What You Need To Ask Yourself • Is your application written for RAC, Does it require a re-write to support the add RAC functionality such as TAF and load balancing? • Is your application RAC certified? • Does your project support the monetary costs of RAC and the extra costs of supporting RAC? • Have you identified single points of failure outside the database solution, ie single points of failure in the network, no UPS single HBA cards and NIC’s.

22. What You Need To Ask Yourself • Can you sustain a 20min outage? • Do you require a DR function to allow your system to continue after fatal disaster? • Do you have the DBA skills to build and support your RAC? • Can your company leverage off an existing RAC solution? • Will you have to setup Dataguard to another geographically separate RAC site.

23. Shared Oracle_Home’s • It provides another level of redundancy. • Allows you to mange patching and upgrades, in some cases rolling patches will allow your RAC to operate while one home is being patched. • Reduced downtime. • In a large RAC where you are managing a possible 15 nodes then it’s not likely that you will be able to manage each home individually. This is where you will consider to choose Shared against local. Although you do have the option of using opatch and ssh and replicating the patch across all nodes.

24. Solution Best Practices • Use local ORACLE_HOME and CRS_HOMES. • Standardize your RAC OS build and paint it Gold. • Clearly identify requirements and identify either RAC or Dataguard. • RAC is not a DR solution • TAF is not seamless. • Select the right Technology to support your RAC. • Identify if your application is written for RAC or certified for RAC. • Cluster file systems complicate the solution. • One vendor one phone call for support. • Don’t compromise on the Interconnect.

25. Questions John-Paul Tocker Red RockConsulting  email: tockerj@redrock.net.nz phone: 04 495 3350 MSN: jptocker@gmail.com virtual: www.redrock.net.nz