Chapter 18 – Message Brokers The Preferred EAI Engine

1. Message brokers bridge many different platforms and development solutions than until now have been more isolated than open. The services provided by message brokers can be put into several distinct categories: message transformation, rules processing, intelligent routing, message warehousing, flow control, repository services, directory services, management, APIs, and adapters. An advantage of brokers is that they “mirror the business”, allowing for non-invasive integration, and automating functions currently performed manually. Brokers build on top of existing middleware technology, and could be called the “middleware of middleware” (See Fig 18.1). Chapter 18 – Message Brokers The Preferred EAI Engine

2. Fig. 18.1 Message brokers are able to integrate many different types of middleware, as well as applications and databases.

3. Message Broker is based on asynchronous, store-and-forward messaging. While its standards are not yet well defined, major vendor’s implementations include at least these three essential components (Fig 18.2): message transformation layer rules engine and intelligent routing mechanism An application publishes a message to the broker, while another application subscribes to the message. Applications need NOT be session connected, which alleviates scalability problems of other integration solutions. Moreover, brokers also able to transform and convert the payload, reformat the message, and route to any number of targets (which determined by the rules engine). All brokers must offer “any-to-any” and “many-to-many” messaging capabilities – publish/subscribe is an effective model to achieve both goals. They provide value-added services from the uppermost layer in the ISO model – the application layer.

4. Fig. 18.2 Message Brokers have three primary components: the message transformation layer, the rules engine, and the intelligent routing mechanism.

5. Broker could interface with source or target at the data level, application interface level, method level, and the user interface level. The strength of message brokers is that they are able to link any number of different types of systems, adjusting to the differences between all source and target systems by exposing an API, and sometimes an adapter. (Fig 18.3). Fig. 18.3 Message Brokers provide services to applications through an application programming interface, or an adapter.

6. Message Transformation Layer The Layer understands the format of all messages being passed among the applications and changes them “on the fly” – data from source is restructured to target(s) format (Fig 18.4). It is done through parsing and pattern-matching methods that are constructed to describe any message format. Information required for transformation is generally stored in a repository that keeps track of the source system, the format of the message, the target system, and the desired format of the target system. Format incompatibilities resolved through schema conversions; most data conversion are automatic, except for conversions between numeric and alphanumeric formats. All conversions must take place dynamically. In rear cases, conversion cannot be done (e.g. length/value of the message is unknown); rules-processing capabilities of brokers allow resolution of such situations.

7. Fig. 18.4 The Message Transformation Layer

8. Rules Processing Rules Processing is an application development environment supported by the message broker to address the special requirements of integrating applications. In effect, it is an application between applications, one that does nothing more than provide the logic for sharing information. Routing of message to more than one target, performing computations or look-ups on data as its being transformed, etc. are examples of Rules Processing. Rules are created programmatically, primarily with script languages, and with virtually no bound to their flexibility. Intelligent Routing Also known as “flow control” or “content-based routing” builds on capabilities of both rules layer and the message transformation. Broker can identify message source and route it to proper target(s) transforming it if required (Fig 18.5).

9. Fig. 18.5 Intelligent Routing means identifying the message and sending it on to the proper destination.

10. Message Warehousing Message Warehouse is a database that is able to store messages that flow through the message broker (Fig 18.6). Provision of message persistence is facilitated to meet several requirements: Message Mining Capability for extraction of business data to support decisions Message Integrity Warehouse provides persistent state for message traffic Message archiving Capability to store messages for archiving/other purposes Auditing Enabling to diagnose the health of the EAI solution through analysis of message traffic

11. Fig. 18.6 Message Warehousing

12. Repository Services Repository is a database of information about source and target applications, which may include data elements, inputs, processes, outputs, and the inter-relationships among applications. The goal is to keep track of sophisticated information about source/target applications (e.g. metadata, schemas, security and ownership perms). In essence, repository is a master directory for the entire EAI domain, and will ultimately become the enterprise metadata repository, able to track all systems and databases connected to the Message Broker. Directory Services Directory Services provide a single point of entry for applications and middleware, utilizing naming service standards. They are nothing more than a way to classify resources on the network in a way consistent with any other method of classification (e.g. DNS, X.500, etc.).

13. Adapters Adapters are layers between the message broker interface and the source or target application – for example, a set of “libraries” that map the differences between two distinct interfaces – the message broker interface and the native interface of the source or target application – and that hide the complexities of that interface from the end user or even from the EAI developer using the message broker. For example, broker may have adapters for applications (SAP, Baan, PeopleSoft), databases (Oracle, DB2, Sybase) and even to specific brands of middleware. They are also getting “smarter” with intelligence placed in adapters running on source/target systems for better event capture. There are two types of adapters in the world of message brokers: thin adapters and thick adapters; and adapters may behave in two different ways: dynamic and static.

14. Thin Adapters They are simple API wrappers, or binders, that map the interface of the source or target system to a common interface supported by the message broker (Fig. 18.7). Their advantage is simplicity of implementation and greater granular control. Disadvantage is impact of performance without increase of functionality, and considerable programming effort is required – thin adapters are nothing more then another interface software. Fig. 18.7 Thin Adapter

15. Thick Adapters In contract, these provide a lot of software and functionality between the message broker infrastructure and the source or target applications (Fig. 18.8). Because the abstraction layer and the manager manage the difference between all the applications requiring integration, there is almost no need for programming. The user sees only a business-like representation of the process and the metadata information as managed by the abstraction layer and the adapter. Although Thick Adapters are very complex to develop, greater complexity of EAI will ensure further sophistication, requiring no programming and providing as easy, business-like method to view integration of the enterprise; Thick Adapters are the future. Fig. 18.8 Thick Adapters place an abstraction layer on top of the application interface.

16. Static and Dynamic Adapters Static Adapters are the most common today, have to be manually coded with what’s contained within the source and the target systems (e.g. they must be configured by hand to receive data form a source schema). Dynamic Adapters are able to “learn” about the source/target systems connected to them through a discovery process that the adapters go through when first connected to the application or data source. Equally important – they can “re-learn” if changes occur. Using the API In addition to adapters, message brokers leverage APIs as a mechanism to access the services of source/target systems. It is similar to traditional message-oriented middleware, however broker could integrate multiple applications, while MOM is primarily for linking applications in one-to-one fashion.

17. Topologies Message Brokers use a hub-and-spoke topology. As a hub, Broker rests between the source and target applications in topology of a star (Fig. 18.9). While this topology considered traditional, others include bus and multi-hub. In the bus configuration, the broker sits on the network bus and provides services to other systems connected to the bus (Fig. 18.10). Such topology is effective when brokers play smaller role within EAI. Fig. 18.9 Hub-and-Spoke Configuration

18. Fig. 18.10 Bus Configuration Multi-Hub Configuration – a number of brokers are linked, with source and target applications linked to any of the brokers in the topology (Fig. 18.11). The advantage is ability to scale, making it possible to integrate a virtually unlimited number of source and target applications.

19. Fig. 18.11 Multi-Hub Configuration

20. As message brokers mature and become more sophisticated, there is a clear movement away from the simple hub-and-spoke configuration toward the multi-hub configuration. Issues must be resolved in orchestrating load-balancing and implementing fail-safe mechanisms within the multi-hub topology. Future of EAI and Brokers Vendors need to standardize core features as include layers of value – added software, with thin adapters giving way to thick. There is drive to hide the complexity behind abstraction layers and provide business “face”. Given this reality, message brokers may turn out to be more about business automation than middleware.