Event Processing in Business Applications

1. Event Processingin Business Applications Roy Schulte 15 March 2006

2. Event Processing is a Bounded, Definable Discipline We assume these definitions: • 1. First definition: An “event” (“ordinary event” or real-world event) is a state change (something that happened) which is significant to someone for some purpose. Implication: Not everything that happens is an event. A state change is an event only if you treat it as one. Many state changes are too fine grained, unimportant or lacking in identity to be events. • 2. Second definition: An “event” (“software event” or “event object,”) is an object, usually in the form of a message, that reports on a ordinary (type 1) event. • 3. “Event processing” means computing that uses events (definition 2, not definition 1). • Note 1: Ordinary events (type 1) may be • (i) Simple – or “atomic,” if they contain no meaningful (type 1) member events • (ii) Complex – if they reflect the significance of two or more member events, each of which are meaningful (type 1 events) in their own right • Note 2: An event object (type 2) may report either a simple event or a complex event. A complex-event object (type 2) may be • Raw or original, if it directly reports a complex event from the real world or • Derived (“synthesized”) if it is generated by processing two or more type 2 events

3. 1. Event-Driven Architecture (EDA) for Flexibility and Maintainability Order Entry Manufacturing Shipping Event Processing Brings Two Kinds Of Business Benefits 2. Complex-Event Processing (CEP) for Earlier and Better Insight

4. Agenda 1. Four Ways To Process Events 1.1. Event passing 1.2. Basic event mediation 1.3. Complex-Event Processing (CEP) 1.4. Combinations (e.g., events with BPM) • Commercial Adoption of Event Processing 2.1. Market drivers – is there a killer application? 2.2 Market inhibitors – why hasn’t this already happened? 2.3 Industry standards – critical to expanding the market

5. MOM, Web Services, ESB, E-Mail Event Passing Sink Source Event Sink Publish Subscribe Event Passing in Event-Driven Applications Asynchronous “staged” business processes may be called • “Event-driven,” • “Message-driven” or • “Document-driven”

6. Characteristics of Simple Event-Driven Applications • Event handler (sink) is triggered by the arrival of one event in one event stream • No real pattern matching – the rule always evaluates to “true” (i.e., begin execution upon arrival of any one well formed message in the event stream) • Benefit is experienced mostly in the IT department • Benefit comes through increased flexibility of software engineering. The absence of control coupling (source and sink are not interdependent) makes it easy to add new sinks, move a sink, or even eliminate a sink, all without touching the event source (of course, the source and sink must still share a common understanding of the event semantics) • Relevant middleware here is usually some form of MOM • Event messages usually don’t carry any genetic information that would indicate causality, or if there is such information, it is indirect (inserted by the application as a data element in the payload rather than deliberately planned and inserted on the header by an event processing agent (EPA) or manager) • These events are raw (original), they are not explicitly synthesized directly from other events, although an application might generate an event which is seen as raw (original) but which actually reflects the significance of some other event(s) which were also seen as raw

7. The Popular, Modern (Loose) Definition of “SOA” May Encompass EDA • If “SOA” means a modular, distributable, loosely-coupled, sharable application, then EDA is a type of SOA (because EDA sources and sinks are also modular, distributable, sharable, encapsulated and uncoupled) • But if “SOA” means that one module provides a service to another (consumer) module, then EDA is not a type of SOA • A ‘service’ is the performance of duties or responsibilities for the benefit of others, but an event sink does not provide a service to a consumer module • An SOA consumer delegates work to a subprocedure (the service provider), in contrast to an event source which does not delegate work or pass control • Be careful of the word “consumer.” In EDA, an event sink is said to be the “consumer” (of the event), while in SOA, the service “consumer” initiates the relationship and consumes a function supported by the service provider • The IT industry has acronym fatigue and hype fatigue, so wider adoption of EDA may be slow for another 5 years unless it is seen as wave 2 of “SOA”

8. Real Applications Leverage Both EDA and Request/reply SOA Relationships Order Entry Manufacturing Shipping Source Event Sink (event handler) Request/reply SOA within an application Sales Admin Use EDA when: • Application stages should run even when another adjacent stage is not running. • There will be a need to frequently add, drop or modify processing stages without affecting any previous or subsequent stage. • Multiple stages may execute simultaneously. • Two or more sinks (stages) need the same input event data.

9. Integration Broker Sink Broker Source Basic Event Mediation Service Provider Transform, Route on Contents MOM, Web Services, ESB, E-Mail Event Passing Sink Source Event Event’ Event Sink Publish Subscribe Business Events Are Used in Four Ways: 2. Mediated Events

10. Characteristics of Basic Event Mediation • One event stream goes into a broker/mediator, one or more streams come out (message splitting but not message combining). Broker/mediator is stateless; it acts on each input event separately (it has temporary state only while it is operating and it does not retain state after it has acted upon an incoming event) • Filtering, content-based routing, and transformation (“conditioning,” or “enrichment”) may be performed by the intermediary • Benefits directly accrue to IT people, indirectly to business end users • The main benefit is insulating source and sink applications, effectively offloading application integration work from end point applications • The relevant technology generally includes MOM and an integration broker • Events sent out by the mediator generally lack explicit causality information

11. CEP Tools Complex Event Complex Event Complex-Event Processing (CEP) BAM Event Source EPA EPA Filter, Aggregate Integration Broker Sink Broker Source Basic Event Mediation Service Provider Transform, Route on Contents MOM, Web Services, ESB, E-Mail Event Passing Sink Source Event Event Event Sink Publish Subscribe Business Events Are Used in Four Ways:3. Complex-Event Processing

12. Characteristics of CEP • Two or more events come in from one or more event streams • Pattern detection requires rules, usually involving sophisticated algorithms • Non-IT people often see direct benefits through a BAM dashboard or similar mechanism - the most common benefit is business insight rather than faster and easier software engineering of a business application • The relevant technology generally includes MOM and a stateful event-processing agent (EPA) or other event manager (custom 3GL coding would be onerous in many CEP situations) • Complex events are often consciously synthesized from simple events, and genetic information is sometimes inserted into the events • BPM is not required for CEP • A business process model is not required for CEP • But BPM may use BAM and CEP for monitoring of business processes

13. Complex Event Complex Event Complex Event Complex Event Dash- board Event Event Event EPA EPA EPA BPM CEP and BAM - ManipulatingMultiple Event Streams 1. Capture simple events (e.g., adapters) BAM Tool 2. Transport events (e.g., MOM) Subprocess SOA Service 3. Apply EPL rules: filter, correlate, apply constraints, aggregate, update event logs 4. Notify people, trigger BPM processes or invokeapplications and SOA services, optionally enriching events with context from event logs and other databases

14. ? The Majority of Business Activity Monitoring (BAM) Applications Use CEP Goals • Monitor key objectives • Anticipate operational risks or incidents • Reduce latency between material events and action Approach • Update real-time dashboard • Link to BI historical reporting and analysis • Issue context-rich alerts • Integrate with business process management • Manage incident life cycles Orders processed Manufacturing defects Web channel volume IT capacity Supply chain status Support call times

15. CEP Tools Complex Event Complex Event Complex-Event Processing (CEP) BAM Event Source EPA EPA Filter, Aggregate Event-Based BPM Combinations: e.g., Event-Enabled BPM Sink BPM Source Service Provider Process Management Integration Broker Sink Broker Source Basic Event Mediation Service Provider Transform, Route on Contents MOM, Web Services, ESB, E-Mail Event Passing Sink Source Event Event Event Event Event Sink Publish Subscribe Business Events Are Used in Four Ways:4. Combinations

16. Characteristics of BPM-enabled Events • One or more event streams are presented to a stateful BPM engine/mediator, one or more streams are generated as output (message splitting and message combining) or one or more services are invoked • Filtering, content-based routing, and transformation may be performed by the intermediary, but more importantly, the flow of control is governed by a state-aware BPM type mechanism that has a pre-specified model of the business process • Benefits mostly business end users • Benefits include making the process flow explicit, making the process flows easier to change and making it possible to monitor the integrity of a multistep process through multiple stages in its life cycle • The relevant technology includes MOM and a purpose-built stateful BPM engine. • Events are consciously synthesized from other events • BPM almost always uses message passing for some aspects of processing • BPM may also use event mediation and CEP for monitoring of business processes

17. BPM Manufacturing Business Process Management Tools Use Events to Trigger Activities and to Track Status Order Entry Shipping Notify BPM engine that activity has completed Billing Invoke a SOA Service Trigger an EDA event handler

18. Complex Events Event Passing Event Mediation BPM Events Adoption of EDA Application Types in Business Applications 100 Theoretical Usefulness 80 Actual Usage, 2005 Percentage Penetration in New, Large-Scale Application Systems 60 40 20 0

19. Agenda 1. Four Ways To Process Events 1.1. Event passing 1.2. Basic event mediation 1.3. Complex-Event Processing (CEP) 1.4. Combinations (e.g., events with BPM) • Commercial Adoption of Event Processing 2.1. Market drivers – is there a killer application? 2.2 Market inhibitors – why hasn’t this already happened? 2.3 Industry standards – critical to expanding the market

20. Market Drivers 1. Business strategies are enhanced by EDA. The real world is mostly event driven, and event-driven situations are best addressed by event-driven business applications (to eliminate the impedance mismatch) 2. Pressures for sense-and-respond behavior, fast action, prediction-based actions and agility are growing because of globalization, compliance demands and other forces (zero-latency enterprise, real-time enterprise and similar goals) 3. SOA is teaching developers about business components and will make event-capable middleware (ESBs) commonplace 4. Vendors are developing better event processing software tools 5. Event-oriented standards are expanding, especially within Web services and Java 6. Ongoing improvements in CPU and network price/performance and scalability 7. Candidate killer applications: Trading, defense, BAM, RFID, CRM, fraud detection, security, SCM, compliance

21. Market Inhibitors: Why this has not happened already • Gaps in software development methodologies even for simple staged event-passing EDA • Mainstream business application developers lack knowledge of event-capable tools: MOM, integration suites, BPM, CEP and BAM • Business analysts do not understand EDA (design issues, technology or applicability), and EDA specialists don’t understand the business situations that could use EDA. Potential projects are not even identified (lack of ROI information is not the real problem) • The lack of standards breeds fear of high costs, lock-in and failure

22. Business Process Modeling OO Analysis and Design Enterprise Architecture Frameworks Data Modeling Service-oriented Development of Applications (SODA) or Service-oriented Analysis and Design (SOAD) Next-generation Component-based Analysis and Design Event-oriented Analysis and Design 2003-2013 New Analysis and Design Approaches for SOA and EDA Are Emerging 1998-2008 • Most current work on developing SOA analysis and design methodologies still fails to make events first class citizens

23. Standards Relevant to Events (I) • Simple events • WS- RX, WS- Reliable Messaging and WS-Reliability • WS- Addressing • WS- Eventing, WS- Notification – getting back on track • WSDL v.2 is better than WSDL v.1 for EDA (MEP concept) • Java Message Service (JMS) • Message-Driven Beans (MDB) • Mediated events • XSLT, XPath, XQuery, JBI, SCA/SDO • BPM events • BPEL, BPMN

24. Standards Relevant to Events (II) • BAM/CEP Standards • Common Business Event (CBE) schema – first pass was pre-WS-Notification, is now seen as a part of the Oasis WSDM (Web services Distributed Management) event framework but it must be extended to handle business events • Event Processing Language (EPL) – No proposals currently on the table • Need standard APIs for handling events, event metadata and event databases (CEI is one possible starting point) • JAIN (Java for Advanced Intelligent Networks) • Includes SIP • SLEE (Service Logic Execution Environment)

25. Recommendations for User Companies • Use EDA on the business process and application architecture levels. • Design all new enterprise-class applicationsas sets of assignable modules, rather thanwith monolithic architectures. • Use request/response SOA and the simpler forms of EDA in mainstream projects now. • Use CEP in BAM applications and for leading-edge projects where real-time insight must be maximizedand you are willing to invest in emerging technology.

26. Event Processingin Business Applications Roy Schulte 15 March 2006