Deploying E-Business Suit 11i on Oracle RAC 11g on AIX Oracle OpenWorld September 2010

1. Deploying E-Business Suit 11i on Oracle RAC 11g on AIXOracle OpenWorldSeptember 2010

2. Introduction

3. Agenda • Bunnings at a glance • Our Business Benefits of RAC • Infrastructure Capacity Management • Initial Build and Deployment Experience • Our Build and Design • Preparation and pre-requisites • Patching and Installation • RAC conversion • RDBMS • Required Mid-tier changes • OEM Configuration • Issues and Resolution • Useful References • Next steps • Q&A

8. WIDEST RANGE LOWEST PRICES BEST SERVICE

9. Sales = IT Challenges Sales ($m)

10. Our Business Benefits of RAC • Ensuring business core applications remain and have the needed Flexibility and Agility • High Availability 24x7 Australia and New Zealand Operations • Deliver on Performance • Provide a level of Workload Segregation “Our Mission critical systems must be agile to support our business by improving operational effectiveness as well as supporting decision making which is based on timely, accurate and flexible information”

11. User Uniqueness • PDT – ordering • Reporting – Discoverer • Store Support • Forms • Self-service • Custom Forms and Self-service • Stores • MC-75 PDA/PDT running self-service • Touch screen and pen input • WIFI and 3G Wireless • Live reporting via XML Publisher and Discoverer 10g

12. Our Mobile Procurement Solution

13. Transactions Processed No. Transactions Per Month Infrastructure Capacity Management ** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth

14. Metalink article 554539.1 (Using Database Partitioning with Oracle E-Business Suite ) Has all the information needed to implement partitioning in EBS Infrastructure Capacity Management • Information Lifecycle Management is part of the solution • EBS partitioning needs to be done early otherwise special issues come up if regular purging cannot be undertaken due to business requirements • Inventory, GL, AP, AR, PO can grow quickly depending on use • Moving data in live systems is tricky

15. Infrastructure Capacity Management • RAC - scales very well… but some caveats! • OLTP/Warehouse type queries, Global Cache Service (GCS) contention • Concurrent gatherstats across nodes can cause lock ups • Any contention in a single node environment magnified in multi-node RAC

16. Initial Build and Deployment Experience

17. Initial Build and Deployment Experience

18. Initial Build and Deployment Experience

19. Our Build and Design

20. Our Build and Design • Our design and build phase experiences • Don’t try to do it all at once • Implement in multiple steps and give time to bed in changes and gauge their full impact before moving on • Follow stream deployment and testing strategy • Start in your Sandpit/Test environments to work out the specific requirements of your environment • Most systems are integrated, CIT and SIT testing is next.. Beware of TNSNAME changes during RAC conversion for DB Links

21. Our Build and Design • Our design and build phase experiences • Test thoroughly • Don’t rush testing and thoroughly regression and load test the new architecture, some issues only come out under load with custom code

22. Testing Approach EBS Backend Load Tests • Swingbench Architecture • Custom class extensions to enable more control • Junit integration with Swingbench to allow OA Framework testing

23. Testing Approach • Infrastructure testing using SwingBench bots deployed across our varied network topology to calculate latency and load. • Packeteer was installed at end points to enable granular monitoring of network utilisation.

24. Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover ) ID 294652.1] and Metalink note 244366.1. Our Build and Design • Our design and build phase experiences • First transformation was to move to ASM 11g/CRS11g 10g database • Requires a patch to solve locking issue when changes happen in ASM • Second transformation was to upgrade 10g RDBMS to 11g RDBMS • Use best practices, prepare for changes in execution plans, baseline 10g system and compare, make sure you are fully patched up on 11g

25. Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover ) ID 294652.1] and Metalink note 244366.1. Our Build and Design • Our design and build phase experiences • Third transformation was to go to RAC with 3 nodes • More reliability and less impact if loss of one node • Better ability to distribute workload • Mixed workload OLTP/Reporting • Discoverer and other reporting tools can be segregated to specific nodes

26. Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover ) ID 294652.1] and Metalink note 244366.1. Our Build and Design • Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction) • We plan Active/asymmetric service descriptions, 2 nodes per service for HA • TNSNAMES and JDBC are used for external access such as reporting tools (Discoverer, Cognos, Portal) to isolate workloads

27. Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover ) ID 294652.1] and Metalink note 244366.1. Our Build and Design • Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction) • Resource Manager Consumer Groups and Mapping are used to manage resource utilisation which are then mapped to services • Can go to application level via application profiles, dbc and session_cookie_name settings which can be changed to match RAC services

28. Metalink document : Using Oracle Real Application Clusters and Automatic Management with Oracle e-Business Suite Release 11i and Oracle 11g. Metalink note 455398.1 Minipack 11i AD.1.5, Metalink note 233044.1 Preparation and pre-requisites • The minimum software levels for various EBS modules can be found on Metalink. Key Patches • Minipack11i.AD.I.5 (233044.1) • 11GR1 APPS Interoperability Patch (6328363) • 11i.ATG_PF.H RUP6 – (5903765) • 11.5.10:SFM Unable to Process Orders in RAC Config (4022732) ( this can be applied before setting up Oracle RAC)

29. Review the two key Metalink notes, 950896.1 and 244040.1 for known performance patches/issues with EBS 11i Patching and Installation • Follow best practices for RAC and 11g • Separate homes for CRS, ASM, DB • Patch up each component to latest patchset available on 11g starting with Clusterware • Check key Metalink documents for known performance patches/issues with EBS 11i • Shared file systems for directory objects (output directories), OCFS2, NFS, IBM GPFS • RAC memory overhead over single instance needs to be considered (about 10% more for buffer cache and 15% for shared pool)

30. EBS to RAC Conversion, Metalink note 455398.1 and 455398.1 RAC Conversion RDMBS • CRS installation on each node • ASM conversion if required • RAC conversion (follow Metalink) • Edit ConvertToRAC.xml • Use RConfig utility and plan a few dry runs • NetCA listener reconfiguration for cluster (RDMBS level) • UNDO and TEMP tablespaces • FLASH area may need adjusting • Enable AutoConfig on all database nodes

31. Cloning Considerations and implications , refer to details on Metalink note 760637.1 RAC Conversion RDMBS • Standby • Archive logs generated from each instance • Archive logs name change • RMAN scripts may need to be modified • Cloning implications • RAC to RAC cloning, multiple to single node RAC or multiple to multiple adds complexity and will be dependant on site • Rapid Clone, Oracle Application Management Pack, manual • IBM SVC Metro Mirror and Flash-Copy, block level copying (high speed cloning)

32. Known Issue, Service Manager Processes Are Not Starting After A Restart - usdsop cannot redirect standard output, Metalink note 733901.1 RAC Conversion Mid-Tier changes • Auto configuration, get ready to run it a few times on all nodes! • Implement Load Balancing using context editor by pointing to correct balance listeners • Tools OH TWO_TASK (s_tools_twotask) • iAS OH TWO_TASK (s_weboh_twotask) • Apps JDBC Connect Alias (s_apps_jdbc_connect_alias) • Parallel Concurrent Processing configuration • In multiple application tier only • Transaction managers

33. OEM configuration • Oracle Enterprise Manager is an essential part of our administration, control and support structure • Moving the instance from non-RAC to RAC had some impacts • Need to re-register the instance as RAC • User Defined Metrics need to be moved from the instance level to Cluster level, we have a lot of these and process was manual • Challenges in doing live performance management with OEM in RAC context • need to hone in with SQL scripts looking at session information combined with various GV$views • drill down in OEM goes from cluster to instance to session view which makes it difficult to pinpoint cross instance locking => identifying library cache pin locks is a good example

34. Issues and Resolution • Initial issues identified in testing addressed with patches • Unique environment challenges • Testing stream alignment (UAT/SIT/CIT/SP) • extra infrastructure required to reproduce • 1 node RAC can be used for development and preliminary test environments • Right sizing test data and environments • UAT must be sized to an adequate ratio to production to outline issues in contention and scaling

35. Next Steps • Extending EBS Environment Externally • OLM • Procure to Pay • Credit Card (own - closed loop) • Further Growing Reporting Requirements • Improve Performance • Partitioning • Continue with archiving

36. Oracle E-Business Suite Release 11i High Availability Documentation Roadmap – Metalink note 1072807.1 Oracle Applications Release 11i with Oracle 11g Release 1 (11.1.0) –Metalink note 452783.1 - Interoperability Notes Using Oracle Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 11i and Oracle Database 11g – Metalink note 455398.1 Useful Resources

37. John Olszewski • jolszewski@bunnings.com.au

38. THANK YOU