490 likes | 872 Views
PeopleSoft High Performance and MAA Guidelines on Oracle Exalogic and Oracle Exadata ' [CON7985]. Latha Krishnaswamy | Director, Product Management | Exalogic Dennis McCormick, Systems Manager at City and County of San Francisco Aaron DeLosreyes , Director, Oracle Technology, Cognizant
E N D
PeopleSoft High Performance and MAA Guidelines on Oracle Exalogic and Oracle Exadata' [CON7985] LathaKrishnaswamy| Director, Product Management | Exalogic Dennis McCormick, Systems Manager at City and County of San Francisco Aaron DeLosreyes, Director, Oracle Technology, Cognizant Oct 2nd, 2014 Oracle Confidential – Restricted
Program Agenda 0 1 PeopleSoft on Exalogic - Anti-pattern traps to avoid…. What is Exalogic Elastic Cloud Software? Exalogic Elastic Cloud Software – PeopleSoft Optimizations PeopleSoft Reference Architecture on Exalogic MAA Best practices PeopleSoft on Exalogic Case Studies – City of San Francisco, Cognizant Demo Summary 2 3 4 5 6 7 Oracle Confidential – Restricted
PeopleSoft on Exalogic: Anti-Patterns Observed… • Multiple PeopleSoft Environments • Port conflict, session state • Hardware (F5) & Software Load Balancers (Apache) vs OTD on Exalogic • Non-separation of JVMs • PIAs for FS, TE, HR – traffic crisscrossing • Integration Broker on same JVM as PIA • Scheduler logs • For recovery & DR, log file share locations and continuous replications • Choice of protocol • Not TCP over JDBC, but rather SDPoIB • JOLT over RDMA - PT 8.53+ • PeopleTools mid-tier • Shared storage use by different VMs • Patching/upgrading multiple VMs • COBOL is chatty. Scheduler on Exalogic. • PeopleSoft Application database • PUM, SYS DB, SES DB, migrate to Exadata Let’s begin our journey to embark on best ways to onboard PeopleSoft workload on Exalogic & Exadata……. Oracle Confidential – Restricted
Program Agenda What is Exalogic Elastic Cloud Software? 1 Exalogic Elastic Cloud Software – PeopleSoft Optimizations PeopleSoft Reference Architecture on Exalogic MAA Best practices PeopleSoft on Exalogic Case Studies – City of San Francisco, Cognizant Demo Summary 2 3 4 5 6 7 Oracle Confidential – Restricted
Exalogic Elastic Cloud System Software Enterprise Manager Middleware and Business Applications Coherence WebLogic Tuxedo Traffic Director Exalogic Control Exabus Integration Exabus Integration Exabus Integration Exabus Integration Physical Oracle Linux/Solaris Oracle Linux Guest OS Oracle VM 3 for Exalogic Exabus Exalogic Elastic Cloud Hardware = Exalogic Elastic Cloud Software
PeopleSoft on Exalogic – Building Blocks Enterprise Manager PeopleSoft Mid-Tier Deployment Application Server Tuxedo PIA WebLogic Scheduler/ Integration Broker Traffic Director Exalogic Control Exabus Integration Exabus Integration Exabus Integration Exabus Integration Physical Oracle Linux/Solaris Oracle Linux Guest OS Oracle VM 3 for Exalogic Exabus Exalogic Elastic Cloud Hardware = Exalogic Elastic Cloud Software
Exalogic Elastic Cloud Software Optimizations for PeopleSoft Workloads Oracle Confidential – Restricted
Tuxedo on Exalogic Increased Performance from Exabus StandardHardware • Eliminates BRIDGE process as bottleneck • Enables greater scalability of Tuxedo domains • 7x throughput • No application changes needed Exalogic 7X 95,595 BRIDGE Single TCP Connection Over Ethernet EECS 1.02.0X 13,680 Exabus BRIDGE dge Tuxedo Applications
Sockets Direct Protocol (SDP) SupportBypasses TCP/IP Stack Tuxedo Domain 2 Tuxedo Domain 1 Node 1 WebLogic Server Server 1 Server 2 /WS Clients Node 1 Server 3 Node 2 Server 4 Server 1 Jolt Clients DB Server 2
Exalogic Optimizations in WebLogic Server • JDBC & Data Source Optimizations • Active GridLink for RAC, Socket Direct Protocol (SDP) for JDBC • I/O & Work Manager Optimizations • Scattered reads / gathered writes, optimized self-tuning thread pool • Cluster State Replication Optimizations • Lazy deserialization, multiple channels for replication • SDP Optimizations • Network channels, JDBC connectivity • Web Application Scalability and Performance Optimizations • Oracle HTTP client connection management • Faster Cluster Recovery • Rapid death detection and failover • Documentation: • http://docs.oracle.com/middleware/1212/core/FMWLC/products2.htm#CHDGCFDC Oracle Confidential – Restricted
Exalogic Optimizations in Coherence and Tuxedo • Coherence Optimizations • Coherence*Web caching performance optimizations • Use of optimized point-to-point communications in Coherence clusters • IMB transport option on Linux • Documentation: • http://docs.oracle.com/middleware/1212/coherence/COHAG/deploy_checklist.htm#COHAG5300 • Tuxedo Optimizations • Efficient algorithm for Self-Tuning Lock Mechanism (SPINCOUNT) • Optimized performance of intra-node communication (shared memory instead of IPC queue, reduced buffer copies) • Documentation: • http://docs.oracle.com/cd/E53645_01/tuxedo/docs12cr2/exalogic/exalogic.html#wp1092546 Oracle Confidential – Restricted
Oracle Traffic Director – Overview Built-in Application Delivery Controller for Load Balancing over InfiniBand High Availability Quality of Service • Built-in HA on engineered systems • Dynamic reconfiguration • Request, content-based routing • Request rate acceleration • Request rate & connection limiting • Quality of service tuning High Performance Security • Built-in HTTP, TCP load balancing • Native Exabus integration • SSL & TLS offloading • Content caching & HTTP Compression • Throughput • HTTP Reverse proxy • Support for SSL 3.0 and TLS 1.0 • Web-application firewall Oracle Confidential – Internal
High Performance – PeopleSoft on Exalogic & Exadata Performance Test Results + Anecdotes Commodity http://moafshar.blogspot.com/2013/10/middleware-optimizations-for-exalogic.html Blog Entry URL: https://blogs.oracle.com/wernerwatschke/ Oracle Confidential – Restricted
Program Agenda 1 What is Exalogic Elastic Cloud Software? Exalogic Elastic Cloud Software – PeopleSoft Optimizations PeopleSoft Reference Architecture on Exalogic MAA Best practices PeopleSoft on Exalogic Case Studies – City of San Francisco, Cognizant Demo Summary 2 3 4 5 6 7 Oracle Confidential – Restricted
PeopleSoft on Exalogic (Virtual) & ExadataReference Architecture • vServers on Exalogic • Infiniband connection • Shared Storage (ZFSSA) • Common PeopleTools • VM Templates • Multiple PIA - WLS • Multiple AppServers • Multiple Schedulers • Multiple DB (App, SES) • Multiple Workloads • Application Isolation • Better consolidation • Patch/Upgrade faster • Database RAC nodes • …… Oracle Confidential – Restricted
PeopleSoft F5, PIA, OTD configuration on Exalogic virtual • SSL Configured on F5 and OTD • PeopleSoft PIA’s use F5 virtual address for SSL config • PIA, Tuxedo, OTD Admin Node servers deployed • on Exalogic Infiniband channel • OTD Failover groups configured on EoIB Channel • OTD used in Active – Passive configuration Oracle Confidential – Restricted
PeopleSoft OVM Template for Exalogic Simplified Provisioning, Easy Patching and High Performance • Supports PeopleTools8.53, 8.54 • Rapidly test drive new PeopleTools release features: Fluid UI, Activity Guides • Includes PIA and App Server – Weblogic 11g and Tuxedo 10gR3 • Application agnostic – can support any PeopleSoft App Tier • New template for every patch-set simplifying patching going forward • High Performance – parity with bare metal • Complete Instance Management • Reconfigure and save Gold images
Apps Life Cycle on Exalogic - Simplified • Patch apps using OVM Template Apps Patch sets - save 100s of hours and ensure consistency across App, Middleware and Hardware • Manage Apps, Middleware, DB, Servers and Storage from single EM console • Scale Up and Scale down app instances on Exalogic using EM and Apps OVM Templates • Platinum Services resolve issues faster and reduce risk Application Operations EM for Apps Drill Through Views Data Center Operations - EM for Exalogic
Program Agenda 1 What is Exalogic Elastic Cloud Software? Exalogic Elastic Cloud Software – PeopleSoft Optimizations Reference Architecture MAA Best practices PeopleSoft on Exalogic Case Studies – City of San Francisco, Cognizant Demo Summary 2 3 4 5 6 7 Oracle Confidential – Restricted
VMs with shared storage on Exalogic virtualSave Time, Eliminate Mistakes, Follow Best Practices • Shared PS_HOME, PS_APP_HOME on ZFSSA • Facilitates patching & upgrading • Can scale up VMs depending on load • Typical Issues during Load Test • Port locking • Crisscrossing of OTD failover groups for PIAs • PeopleSoft MAA Best practices • Clear demarcation of VMs for each PIA • PIA, AppServer per env i.e. FS, TE, HR • Integration Broker on their own VMs • OTD Active Passive setup • Storage Replication • ZFS storage shares scheduled replication • Scheduler logs continuous replication • PeopleSoft MAA OTN White Paper Oracle Confidential – Restricted
Push Button Apps Deployment with ExceleratorsSave Time, Eliminate Mistakes, Follow Best Practices App Topology Design Provision VMs Configure App • Available Now • PeopleSoft • Commerce • EBS (H2FY15) • Automated App Install • From Days to hours • Create Test Dev Cloud • Apply MAA Best Practices • Reduce manual tasks and errors Push-Button Application Infrastructure Service Application Infrastructure Install & Config Directives Exalogic Application Install & Config Directives Excelerator Excelerator Production Ready Oracle App per MAA VMs Storage Networks Exadata Test-Dev App Cloud Delivered as Fixed Price ACS Service
Program Agenda 1 What is Exalogic Elastic Cloud Software? Exalogic Elastic Cloud Software – PeopleSoft Optimizations Reference Architecture MAA Best practices PeopleSoft on Exalogic Case Studies – City of San Francisco, Cognizant Demo Summary 2 3 4 5 6 7 Oracle Confidential – Restricted
San Francisco’s Exa Story • San Francisco Overview • Exa Upgrade Objectives and Criteria • Exa Upgrade Implementation Timeline • Then and Now • Performance • Maximum Availability Architecture • Looking Back: Challenges and Lessons Learned • Final Thoughts
City of San Francisco • PeopleSoft 9.0 (Ben, HR, TL, Absence, Payroll,eSuites) • Original 9.0 Implementation: Aug 2012 • 25 Environments on HCM 9.0 and HCM 9.2 (Upgrade) • Production= 211 GB (compressed) 6 GB growth biweekly • 30k employees, 100,000k covered lives, $100m payroll • Controller’s eMerge (PeopleSoft) Division = 40 FTE • Responsible for application (4 FTE), OS (2 FTE) and hardware • Department of Technology: =200 FTE • Responsible for database, storage, network and datacenters
Upgrade Project Objectives • Refresh Hardware • Move to new Production Data Center • Move to new Disaster Recovery/Test/Dev Data Center • Simplify Architecture
San Francisco’s Exa Implementation Timeline Upgrade Exadatas to 1/4 rack Exalogic new 1/8rack Inception Excite Process Kickoff May 9th Install at New Production Data Center Dec 12th -20th Install at New Test/DEV/DR Data Center Feb 10-21st Go-Live June 6th Vendor Selected November 5th RFQ Issued October 25th Datacenter Lease Signed July 10th Apply July Patch Aug. 15th-28th 1G Connection Established Jan 17th Apply April Patch May 19-22nd • Operating/Database System • Upgraded from PT 8.5.0 to 8.5.3 • Implemented RAC • Implemented Active Dataguard • Expanded OEM usage • Implemented GoldenGate • Infrastructure • Moved to 2 new data centers • Built Test/DEV/DR from floor-up • Established connectivity • Hardware • 2 1/8 rack Exadatas • 2 1/8 rack Exalogics • UCS for Windows
Reference Architecture • Simplify the visio!
Hardware Refresh: Key Criteria • Create a stable, scalable base for future work • Increase performance • Implement MAA/DR • Reduce Total Cost of Ownership: How? • Simplify architecture across data centers: reduce training time • Increased automation via OEM: more time for value added work • One vendor: less time troubleshooting • Increased performance: less overtime, more time for Q/A • Other options considered: new blades, Vblock, hybrid model
MAA: Highlighted Examples • Rolling patching: no overtime and no outages for end users • Failover: DR test was planned and implemented in a few hours, SunGard test took months of planning time • Replaced parts in production chassis during workweek • Exachecks: proactive monitoring, great Oracle Support • Use of ZFS Exalogic Shares via Vservers to manage images of PeopleSoft HOME & Oracle HOME: makes everything easy
Performance • More than just run time improvements…. • Trouble shooting is much faster and smoother • End User Experience: “the pages jump at you” • Templates: spun up PeopleSoft 9.2 Environments in ½ a day, previous lifecycle would have been months or weeks • Increased from 1,000 end users in June to 3,00o today, with plans to grow to 30,000 in early 2015
Payroll Processing Improvements: how we used the 7 hours • Return to more normal work life balance • More time for quality assurance and data entry tasks • Will return slack time to end user departments: goal is 1:00 PM on Thursday
Looking Back…. • Recommendations… • Plan for growth from day one • Training should be “just in time” hard to relate to scale of change • Don't deviate from best practice; listen don’t do it the old way • Develop documentation and procedures that work for your organization there is no one size fits all • Oracle to do… • Improve documentation for first timers • 800 page user manuals are too dense: need quick reference guides • Improve Platinum Gateway experience • Issues with 8.53.10 template during initial move and build
Final Thoughts • Are we confident we made the right decision? YES • Why? • Total Cost of Ownership • Scalability: ability to scale up and scale down is a game changer • One vendor is really, really nice • What’s next? • Purchased two new Exadata 1/8 racks and new Exalogic 1/8 rack for Oracle Identity Manager and other applications
Maximum Availability Architecture at Cognizant PeopleSoft Systems • Aaron De Los Reyes • Sr. Director, Oracle Technology • Cognizant, Inc Oct 2, 2014
Cognizant’s PeopleSoft ESA, Fin, HR on Exalogic MAA Global Large Critical • And, got problematic • Peak time slowness • More and more servers • Lack of ability to scale • Longer maintenance window
Oracle Exalogic Solution? Exalogic upgrade on web layer Lintel / Weblogic Benefits 50% improved downstream integration processing Server consolidation – 5:1 Maintenance window reduced to few hours Support effort down 50% Better CPU utilization – 3:1
Didn’t quite stop there… Oracle Exadata • Performance woes • 10-15 slow incidents/month • Batch failures / over runs • Slow online pages due to DB2 table locking • Long running SQLs • Inability to scale for 2015+ volume DB2/AIX 3 Options, 1 Pick DB Layer upgrade to Exadata
Implementation, Outcomes & Benefits Strict timelines Conversion of ~6,000 SQLs Constantly changing production system Only <3 days available for cut over Full DR 60 hours cut over DR upgraded same time as production No major post production issues • Financials close batch jobs complete 56% faster • Payment batch jobs run 93% faster • Customer billing batch jobs run 55% faster • RTO for DR down rom hours to near-zero • Transactions with <1s response 75% from 30% • Batch window reduced to ~4 hours from 10+ hours • Huge floor, and power saving • 24x7 availability
Cognizant’s Application Landscape PeopleSoft Suite Cognizant’s PeopleSoft Implementation for Corporate functions include - • PeopleSoft Financials/Enterprise Services Automation - GL, AP, AR, AM, CM, PO , T&E, CA, BI, RM, PC • PeopleSoft HCM – Core HR, Absence Management, Time and Labor, Payroll, ePerformance, Self-Service, Workforce Administration, Recruitment, eCompensation, Profile Management • PeopleSoft ELM for Cognizant Academy • User base – 180K , World wide • Database size : ESA – 2 .5 TB, HCM – 1 TB, ELM – 800 GB • Peak user load: ESA – 15K bi-weekly, HCM – 40K semi-annually, ELM – 10K (irregular peaks)
Architecture Technology Components Deployment for 3 Applications (Financials, HCM, ELM) Primary: App Tier: ½ Rack Exalogic DB Tier: ¼ Rack Exadata (High Performance) Stand by: App Tier: 1/8th Exalogic DB Tier: ½ Rack Exadata (High capacity) Applications: PS HCM 9.1, PS Financials 9.1, PS ELM 9.1 Environment Management Oracle EM for PeopleSoft Applications Monitoring for Health & Perf: Oracle EM for PeopleSoft and Oracle Real User Experience (RUEI)
Maximum Availability Architecture: PeopleSoft Deployment at Cognizant SLA: 99.9% uptime
Summary • Exalogic is engineered systems platform ideal for PeopleSoft workloads • Software optimizations and shared storage help scale & serve multiple envs • Virtual servers house various components of PeopleSoft – PIA, Tuxedo, integration broker, process scheduler… • Multiple VMs ensure High Availability & Workload Isolation • PeopleSoft MAA best practices ensure lower Recovery times (RTO) • PeopleSoft Exalogic Virtual Templates available for PeopleTools release • PeopleSoft Excelerator for rapid provisioning using automation Oracle Confidential – Restricted