Lessons Learned At Entergy Enterprise Applications Integration Project

1. Lessons Learned At Entergy Enterprise Applications Integration Project Date: 12/9/02: Author: Rick Pedings - Consultant TVA Business Sensitive

2. Entergy EAI Development Program • Vendor • WebMethods (initially started as Active Works. WebMethods acquired A/W while project was underway) • Product Description • Transactional Ring and Spoke processing • Guaranteed Delivery (debatable issues) • Publish and Subscribe method of communications • Some Batch Processing was used through a FTP Adapter • Data held in “Canonical Form” • Specific and Universal Adapters were used • SSL Security used • Hosting • Core Product • The broker host resides on a Sun 420R (redundant) server with 4 CPUs and 4 GB RAM • Most resident on Clients. • Some required hosting on a Separate box because of OS issues TVA Business Sensitive

3. Entergy EAI Development Program(Continued) • Mission • Create processes to respond to a De-Regulated business environment by integrating all applicable legacy applications. These were Market Mechanics, SAP, Customer Service, Billing, etc. • Integrate this new process into SAP while undergoing a transition to SAP • Mission was on Critical Path for Entergy to “Go Live” • Entergy EAI Delivery Team • SAIC- providing • Program Control – Interfacing/coordinating with the individual projects requiring EAI • Core Product procurement, development, testing and delivery • Configuration Management of the core product • Integration specifications • Integration resources via IS3C (EAI resource company – body shop) • Price Waterhouse – Initial delivery of SAP and integrating in into the De-regulated processes using A/W • Entergy – Integrating the new De-Regulation model into Market Mechanics, Billing, Customer service, etc. using the SAIC provided integration resources. Multi-internal projects. • Important Note: The following lessons are not meant to be critical or complementary of the WebMethods product, Entergy EAI Integration activities or any other area that may cause concern. These are simply perceptions and experiences of one member of the Entergy EAI Delivery Team. TVA Business Sensitive

4. Lessons • Individual projects wanted to run off on their own with designing integrations that effected other subscribers. Did not always want to participate in the common conventions that are required in the A/W environment. • Prior to development, All projects were supposed to have the interface functional requirements and specification documents peer reviewed and checked for compliance to conventions. This was very hard to control. • If the selected EAI product requires controls over this area, suggest implementing a strong EAI architectural resource in authority. • SAIC Configuration Management did not control this well in the beginning so later developments experienced some rework caused by inadequate naming conventions. • If the selected EAI product requires controls over this area, suggest implementing a strong EAI architectural resource in authority. • During Development, access to the core product was controlled via standard Development and Testing promotion process. When deliverables were in crisis delivery mode, projects running behind constantly wanted special treatment. • Configuration Management tried to be compliant but many feathers were ruffled and fingers were pointed about delay responsibility. Needed all of the team to understand the need for CM. This is an essential element in integrating A/W. • Suggest a required CM training exercise for all projects integrating into the EAI environment. Reinforce the need to control the Common Elements of the integration. TVA Business Sensitive

5. Lessons(Continued) • A DMZ was set up for external access. Security design was not established early, so this became a scramble getting it in place under the Entergy IT Group model. They selected SSL. • We scrambled together additional funds to set up the SSL model and reworked the affected integrations in crisis mode. • Suggest TVA address this in initial requirements. • Vendor changed his pricing strategy during Development that affected annual maintenance costs • This blew our estimates for on-going maintenance but WebMethods did compensate and help for the initial delivery and a few years of maintenance. This change affected long term EAI costs and was not received well by the customer. Unsure if the revised maintenance costs would have influenced initial product selection. • Suggest TVA select a vendor with maturity and get a long term commitment on their maintenance model costs. • Entergy did not allow for fault tolerance in initial selection of A/W and host requirements. • SAIC scrambled additional Entergy funds to add redundancy • Suggest TVA incorporate this in requirements. TVA Business Sensitive

6. Lessons (Continued) • Guaranteed delivery was indicated in the product specifications but we found that the A/W had some failure modes that caused loss of data. The Adapter queue failed on overflow and shut down the adapter. This occurred when the client app was out for a few days and messages overran the queue The vendor was notified and did correct some of the issues. There is still discussions about data integrity between the adapters and the client applications. So the story goes, the A/W adapter did/does not require the application to actually confirm the process the data request after delivering the message, only required an ACK of message receipt. • Suggest reviewing the vendors specifications in great detail about “Guaranteed Delivery” to understand exactly what you are purchasing. Need to look at: • Core product failure/recovery guaranteed processes • Adapter failure/recover guaranteed processes • Client application failure/recovery guaranteed processes • During installing needed updates to adapters, we found that we broke many of the integrations we just completed. Identified problems with vendor releasing adapter updates that were sensitive to client application versions. • Suggest TVA review each vendors release strategy. I.E. downward compatibility of adapters and sensitivity to client app versions. TVA Business Sensitive

7. Lessons (Continued) • Broker Server did not do batch processing well. There was a requirement for batch processing through the Broker Server and when tested, it overloaded the queues and crashed the Broker. • Do not know the final result of this. • Suggest TVA review the needed applications to see if batch processing will be needed and review vendor Reponses to this requirement if needed. (My personal take: This is a waste of EAI bandwidth and if batching is needed, do outside the EAI envelope.) • Hosting of Adapters became an issue after integration development started. The adapters could not always reside on the same host as the client application. The some of the A/W adapters were OS specific and had to be hosted on a separate box, Unix adapter, NT app. This was pushed back to the vendor, do not know what final results were. • To get around this, SAIC scrambled funds to host the adapters on external box. • Suggest TVA review the adapter requirements in vendor responses and insure: • infrastructure to host adapter is included in initial design or • Adapters will host in client App OS. TVA Business Sensitive