EAI & Enterprise Data Availability

1. EAI & Enterprise Data Availability A Software AG Perspective (?) Jim Wallace - Senior Architect

2. The IT Paradox • There is nothing new under the sun …..

3. Agenda • Introduction & Agenda • Maximizing Data Utility in the Enterprise • An Alternative Approach (EAI) • Enterprise Integration Patterns • Enterprise Integration Toolsets • Summary and Questions

4. Maximizing Data Utility in the Enterprise • Expansion of the IS domain – especially the rise of Electronic Business – has greatly accelerated the need and the usage of data across and between enterprises • Total Data Availability Means: • Information where you want it • When you need it to be there • In the form and state satisfies your needs • In short, everything that we’ve been taught but never quite been able to achieve

5. My Definition of Electronic Business Electronic Business: Ability to co-operatively initiate and complete transactions through the exchange of information over the network, preferably without having to resort to other modes of communication, and possibly without human intervention.

6. Historical Strategy & Tactics • Structure the data as flexibly as possible • Centralize information wherever possible • Leverage data replication • Build distributed databases • Separate transactional and analytical stores of data • Real time processing of data • Data cleansing and restructuring • We have eagerly seized on technological advances to do this more cost effectively

7. The Aim – The One Big Database Persistence Layer Data Access Modules Data Access Modules Application Layer Data Access Modules

8. Complexity Rules! • Considered in their own terms, all of these approaches were successful • Just the problem became bigger • Domain Complexity • Solution Complexity • Technological Complexity • It keeps changing faster than we can keep up

9. Control Even As Enterprises Try to Reduce It !!

10. One Big Database – Not! • In practical terms, it has proven to be impossible to develop and to implement all-inclusive, enterprise wide data models • While the effort should and must continue, additional approaches are required • This is where Enterprise Application Integration (EAI) has a contribution to make • It yields some of the advantages of the O(h)BD, in a more cost effective manner

11. A Digression – Process & Applications • The role of EAI in the enterprise has been driven by the need to support higher level business processes. Things like - • “Straight Through” processing • Customer Self Service • Unified (WEB) Interfaces Offering Multiple Services (Portals) • Greater movement across enterprise boundaries (Electronic Business) • The application of EAI neither exclusively nor primarily a data level focus • Although the end result both depends upon and results in increased data availability • EAI normally makes data more available – and more useful – by “sharing” it among collaborating applications

12. Multiple Levels Of Integration • Essentially a classification of techniques used for integration between applications or app components • Data Level • Method Level • Application Programming Interface (API) Level • User Interface Level • Data tends to be shared or exchanged regardless of what level integration is occurring • Often, data can not be shared or transferred without understanding the process or application

13. Clients Suppliers Partners Partners CORBA Clients MOM COM FTP RPC Solaris HTTP …... SAP C++ EDI HL7 Oracle XML SOAP ….. Application View As Complex As Data View The Enterprise OS 390 EDI Trns Windows NT Cobol Apps Natural Apps SQL Server HTML MQ Series Java Apps IIS DB2 Adabas CICS VSAM Windows 2000 VB Apps Pivotal SQL Server

14. Electronic Business = Zero-Latency Business {Gartner} Order Entry The Enterprise Financials Warehouse Mgmt. B2B Bank B2C A2A 4) “Please pay for it.” Logistics B2B Call Center 2) “I don’t have it yet!” E-Commerce Portal 1) “I want to buy it.” 3) “When are you going to get it delivered?” Customer Parcel Service

15. Zero or Low Data Latency • Just the old concept of timeliness with an additional twist • Refers to the real time (zero) or near-real time (low) transfer of information between applications • Zero data latency tends to imply transactional solutions, while low data latency or near-real time allows for assured delivery (messaging) options • Low data latency within the enterprise is no longer enough; it must be enabled across the enterprise perimeter

16. An Affirmation; Modeling the Problem • Adding EAI oriented solutions to the problem of data availability does not reduce the modeling effort required – it intensifies it • Each application view of the data must be defined and understood • Additionally the dependencies involved in sharing and transferring information must be supported • I find that use case modeling, at the level of the inter-application interfaces, is valuable for scoping the problem, and understanding it at a high level

17. Modes of Integration? A Functional View! Data Consistency Composite Application Multistep Process • • Manual or straight-through flows • Batch or immediate, individual transfers • One business process • • Multiple steps • • One-way, asynchronous interactions (loosely coupled) • • Systems are physically and logically independent • Potential long transactions • Batch or immediatetransfers • Multiple processes • • Multiple steps • • One-way, asynchronous interactions (loosely coupled) • • Systems are physically and logically independent • Immediate interaction • One business process • One step • Two-way, synchronous interactions (Tightly Coupled) • Systems are physically and logically dependent • Usually a client server (request reply) interaction

18. Front End versus Back End Integration • Front End Integration • Making Enterprise Resources available by means of tailored interfaces • Managing interaction between the enterprise and the user community • Providing gateways for information to and from enterprise systems • Back End Integration • Integration of various types among different enterprises resources (especially applications) • Back end applications collaborate to complete or realize a business process

19. Trend To XML Trend To XML Enterprise Integration (Internal & External) Backend Applications User Interfaces B2C A2A Automated Interfaces B2B WEB Services Specialized Application Interfaces Specialized Application Interfaces Specialized Application Interfaces

20. Order Entry Inventory Shipping Order Fulfillment Front End Integration Connected To Multiple Back Ends Wireless Device Portal DB Presentation & Communications Portal Business Logic Browser With Different Middleware “Typical” N-Tier Architecture

21. Critical Problems For Front End Integration • All the pieces of the n-tier application have to talk to one another • The various components may have different views of the information associated with the transaction • Must manage security (authentication and authorization); and e access to all of the back end resources involved • Typically, units of work must be supported over multiple resources (2 phase commit) • The solution as a whole must perform and scale • …….

22. Integration Engine Back End Integration Customer Service Order Entry SAP PeopleSoft Invoicing WMS

23. Characteristics Of Back End Integration • Connectivity more of an issue at the back end • Complexity of interaction may be greater • Security, while always critical, is more easily managed • Timing is not as much of an issue; near real time is often good enough • Consequently, suitable for multi-step processes • Transactions (units of work) are often reduced in scope • …...

24. Integration: The Enterprise View Web Based Services

25. Integration Technologies & Techniques • Conventional Techniques • File Transfers & Distributed File Systems, RPC, Teleprocessing Monitors, Database Replication, RJE, Screen Scraping, ….. • Contemporary Techniques • Message Oriented Middleware • Web Enablement (Browser Based) • Distributed Objects • Integration Brokers • Document Centric Integration (XML)

26. Message Oriented Middleware (MOM) • Products that pass messages from a source to a destination • Fundamentally asynchronous in nature • Based on queues, explicitly or implicitly; requiring programming to load and unload msgs at either end • Rely on assured delivery, rather than transactions • Principally a point-to-point solution ….. • Not a lot of value added features originally ….. • MQ Series, MSMQ, EntireX Broker, Sonic MQ …..

27. MOM Example: MQ Series This diagram from IBM shows MQ Series communicating between two platforms. MQ Series itself manages the flow of msgs between the local queue and the remote queues; and does low level marshalling. The OS on the sending and receiving platforms may vary.

28. Distributed Objects (really Components) • COM/DCOM , CORBA, and Java RMI/IIOP • Enables processes to invoke methods on external components (perhaps on remote platforms) • Supports Component Based Development (CBD); with access to individual components only via interfaces • Method level, point to point, synchronous integration par excellence • Unmatched flexibility, but complex to implement • Supports strongly typed interfaces

29. A Trivial Distributed Object Implementation Sybase Customer Account Manager JDBC RMI Client RMI Server Customer RMI Interface RMI Skeleton Distributed Objects Customer object calls the Account Manger with an Account ID. Manager retrieves the balance, and returns it to the Customer Object. RMI Stub Remote Reference Layer RMI Transport Layer InternetProtocol (IP)

30. Integration Brokers • Use the Broker architectural pattern to implement and manage flows of msgs between apps and resources • Uses a MOM (often a JMS) Messaging Backbone • Have sophisticated capabilities for: • Connecting to applications (connectors or intelligent adapters) • Transforming and augmenting messages (applies business rules) • Content Based Routing • Have an enterprise scale orientation & focus • Connects multiple apps in multi-step integration flows • Offer assured delivery; are not transactional oriented

31. Broker Architectural Pattern Database Application API Directory Service Application Screen Interface Integration Broker Application Distributed Component Middleware Service Application Interface Object

32. Integration Brokers - Continued • Provide a number of value added features centered on scalability, reliability, connectivity and graphical implementation tools • Semantically operate very similarly to workflow engines or Business Process Management products • Goal is to engineer a connection between multiple enterprise resources, performing whatever processing necessary to ensure that the connection adheres to business rules • In other words, they implement what is in effect an integrated application, with a minimum of effort • Especially suited for back end integration • Low data latency solution

33. Simple Integration Broker Implementation

34. Document Centric Integration • XML Focus - SOAP/Biztalk; X-Bridge; Rosetta Net; …. • Integration through the transfer of XML “documents” between Enterprises, platforms, applications, …. • HTTP or HTTP/S is often used to transmit the documents over TCP/IP connections • Additional protocols or processing standards often used to manage the communication • Defined XML dialects, like BiZtalk, Rosetta Net, ebXML, ….. • Eliminates much of the technical complexity in communicating between environments and domains

35. Orchestrator/XML Architecture StyleSheets Rules Transform Consumer (e.g. Browser) Consumer (e.g. Browser) Service Consumer (e.g . BusinessPartner) XML XML Gateway HTTP ContentBased Routing Replicate/ Aggregate World Wide Web Log Log XML XML

36. The Orchestrator/XML Solutions • Enables customers to dynamically compose new services in order to react to new business requirements • Provides a document driven reaction framework • Enables applications to be integrated in an XML way • Intelligent, flexible & scalable XML Routing Hub • Obtains an XML document • May transform it (based on style sheet) • Route document based on content (Content Based Routing) • Leverages XML-based and distributed-object technologies to provide enterprises with highly configurable and adaptable solution for integrating XML documents

37. Customers Submit XML Orders via HTTPS Server Submits Tracking Request & Returns Reply /XML Analyses Document, Transforms It, and Routes It to Services Can configure this as a Web Service Server Converts & Edits Data, and Forwards It to Order Fulfillment Server Redirects Orders To /XML Document Centric Integration: Order Entry Customer A Java Server Order Tracking Orchestrator /XML Customer B Web & Application Server Java Server Order Entry Customer C Tamino Audit DB

38. SOAP HTTP/S EntireX RPC A Multi Step Integration Process Workflow Engine Price Policy Verify Credit (Java) Issue Policy Invoice Policy (COM) >> >> Credit Bureau Web Service Check Balance (Natural) Update Account (Natural) Adabas

39. Integration & Web Enablement

40. N-Tier Application For Product Distribution

41. Conclusions • Multiple EAI Techniques and Technolgies Available • In order to properly use them, modeling and planning is necessary • Properly used, they can substantially enhance data availability within the enterprise Jim.Wallace@softwareag.ca Thank you!