Pacific Gas and Electric Company

Pacific Gas and Electric Company Ishaq Mohammed / Anil Saka Online and Middleware Application Operations

Agenda • Overview • Challenges • Solution • Middleware Provisioning Use Cases • PG&E’s Provisioning Portal • Benefits Gained • Future Plans • Summary • Q/A

Company Profile • Headquarters in San Francisco, California • Provides energy to approximately 15 million people • 70,000 square mile service territory • Total Assets: $50 billion • 2012 Net Income: $816 million • 2012 Capital Expenditures: ~$4.5 billion PG&E SERVICE AREAIN CALIFORNIA

IT Overview - Infrastructure • 280+ WebLogic Domains (versions 9.2 through 12.1.1) • SOA Suite & Oracle Service Bus (OSB) • WebLogic Portal & Integration • 970+ WebLogic Servers • 205 Databases (versions 10g through 11.2.0.3) • 500+ host servers running Linux and AIX • Microsoft SQL Server, Apache HTTP Server 2.2 and Sun One. • 2,000 IT employees – Team of 8 in middleware management

IT Overview – MyEnergy Business Application • MyEnergy application most critical from business perspective • Account management (e.g. customers can monitor /manage energy use, pay bills online) • Promotion of energy-savings programs • Scheduled outages of PG&E services • 2.2 million active users • 6,000 users logged in hourly during peak times

IT Overview – MyEnergy Architecture Diagram

Challenges • High cost in manually building 30+ middleware environments each year (~$2.5 million) • 20% increase in WebLogic landscape past 2 years • What worked in pre-production sometimes failed to work in production

Solution: Oracle Enterprise Manager to Provision Middleware Environments • Modernizes the build process • Guarantees speed to market delivery • Eliminates errors • Reduces labor cost and time spent • Provides out-of-box monitoring

Oracle Enterprise Manager Cloud Control 12c Deployment • Four environments for Cloud Control 12c – dev, test, stage and production • Production environment for Cloud Control 12c: • Management Service 12.1.0.3 Release 3on Redhat Linux with Active-Standby high availability configured • Management Repository in two node RAC DB 11.2.0.3 on Redhat Linux • Management Agents 12.1.0.3 Release 3 on Redhat Linux and AIX • Fusion Middleware Plug-in 12.1.0.5

Middleware Use Cases WebLogic Server 12c (WLS) version 12.1.1 Oracle Service Bus 11g (OSB) version 11.1.1.6 WebLogic Portal 10g (WLP) version 10.3.4 Small, Medium Or Large Deployment

Middleware Provisioning • Discover reference Oracle Home/domain • Create provisioning profile in software library • Create directives in software library • Edit predefined deployment procedure • Test deployment procedure from console to capture input parameters • Create wrapper script which calls emcli “submit_procedure” • Enable end users to make requests using PG&E’s Provisioning Portal

Prerequisites • Configure Software Library on OEM Server • Enable passwordless (ssh keys) connection between Oracle Management Service and new VM • Update the /etc/sudoerswith: weblogic ALL=(root)/u01/app/Oracle/product/agent12c/sbin/nmosudo • Setup named credentials and privilege delegation. • Install required packages on the OS, create OS user and group(webLogic) • Install and start Oracle Management Agent on new server

Sample: Provisioning Profiles

Directives • Provisioning Profiles Capture “Standard” Files and Artifacts of WebLogic Domain[Gold Image]. • Additional Files Specific to WL Domain must be moved to Target Host(s) with File Transfer Step or Directive. • With Directives you upload required Files to Software Library along with script to move and/or Manipulate Files, called Main file • Be sure to check “Use Most Recent Version” check box, else your procedure keeps calling older version

Sample Directive

Customized Deployment Procedure Oracle-provided deployment procedure named "Provision Middleware" customized to accommodate PG&E requirements • Custom Directives (Described Earlier), or • Shell Script that Execute Shell Commands and/or call WLST • We Generally Added One Directive and 3 to 4 Shell Scripts to Customized Procedures • Shell Script Cannot Exceed 4096 Characters. • Can Use Runtime Variables in Scripts, e.g. ${target.DOMAIN_HOME_DEST_ADMIN_HOST}

Sample Deployment Procedure

Options to Provision • Provision using EMCLI: • Capture once and execute many Provision using console: • Several clicks and repetitive process

EMCLI Wrapper Script • Generated Properties Files Made Generic by Substituting Actual Values with Specific Keywords: <Hostname> -> MyHostName <Domain Name> -> MyDomainName <DBSid> -> MyDBSid • Wrapper Scripts Creates Copy of Properties Files and Substitutes New Values Passed on Command Line • Removes Copy When Completed • Provision New WebLogic Portal Domain Example: sh./emcliWrapperDB.sh "DB_DATA" "S" “database-host“ "Linux" "WLPDBS" sh./emcliWrapperWL.sh WLP S EIWLP wlp-mw-host database-host WLPDBS

PG&E Provisioning Portal • Operating Systems • SAP Environments • Oracle Databases • Oracle Fusion Middleware Environments • Capable of Provisioning:

PG&E Provisioning Portal Processes • PG&E has built a portal for request intake and complete orchestration of the provisioning process • Provides central interface to end users • Makes REST calls to Pureflex System for OS provisioning • Installs Management Agent • Invokes EMCLI scripts to provision middleware environments

PG&E Provisioning Portal Features • Authentication against PG&E Active Directory / SSO • Ability to notify Requestor/Approver via email • System provisioning clients pay for the resources • System provisioning clients are not responsible for the maintenance of any part of the shared infrastructure • Ability to monitor performance of their workloads to ensure SLAs are met • Provides at least three deployment options for each artifact in terms of sizing: small, medium, large

One Click! • Provision Operating System • Install Middleware • Configure Domain • Discover Domain to obtain out-of-box monitoring

Achieved Results and Improved Business • Standardized OS, middleware and database images • Reduced overall provisioning time by 75% • Saved ~$1 million since rollout

Future Plans • Extend provisioning to more use cases: • Oracle SOA Suite • Oracle Business Process Management Suite • Web Servers

Summary • PG&E uses Oracle Enterprise Manager Cloud Control 12c deployment procedures with custom steps and scripts to provision all Oracle WebLogic Server, Portal and Service Bus environments • PG&E has built an end-user portal to provision middleware infrastructure with a single click • Provisioning brings PG&E a 75% reduction in time to market • Many lines of businesses are enjoying the benefits of lower costs, faster delivery, and standardized deployments

Thank you for your participation! • Q/A

Pacific Gas and Electric Company