Dr. Opher Etzion, STSM Lead Architect, Event Processing Technologies IBM Software Group

1. Event Processing – Vision and RealityBPM-BAM-CEP ConferenceRegensburg, June 19, 2006 Keynote Address Dr. Opher Etzion, STSM Lead Architect, Event Processing Technologies IBM Software Group

2. Outline Event Processing – market view Event Processing - segments Terminology and architecture Towards an event processing Community

3. Event Processing – Market View

4. What drives this area ? • From the bird eye’s view: • Nothing is really new • We process events for many years • (e.g. exception handling in OS). • However, recently: • Significant amount of events – types, sources, instances • Variety of application need to process events • Some traditionally stand-alone applications need to be integrated with regular information systems (simulation, real-time) • Functionality requires sophistication – e.g. temporal capabilities, spatio-temporal capabilities. • These all contributed to COTS tools • It is not cost-effective to develop this functionality for a single application • It is recognized as middleware level capability, thus customer preference for COTS. • Drive for standards is next step .

5. Event Processing - Analyst view Event-Based Application Platforms Definition: Application platforms that offer a programming model for event-driven computing. Business application vendors will need complex event processing capabilities as part of their evolving service-oriented architectures (SOAs) to deliver adaptability and flexibility of their platforms. Justification for Hype Cycle Position/Adoption Speed: Some leading platform vendors have proposed, but none has strategically committed to, an event model as primary for business application design. JavaBeans is the early, and nearly failed, attempt at this model. The model has an intrinsic power that makes its gradual adoption likely, but not in the near term. Business Impact Areas: Event-based application platforms see the software environment as a relationship of events, as opposed to a relationship of programs in prevailing models. Business application vendors will design and develop event-driven principles into their traditional SOAs. This evolution will make unique application styles possible. Benefit Rating: High. Market Penetration: Less than 1 percent of target audience. Maturity: Embryonic.

6. More advanced domain: Algorithmic Trading Complex Event Processing for Trade Facilitation Definition: A component of business activity monitoring, complex event processing (CEP) entails the receipt or extraction of events as they are occurring in real- or near-to-real time, processing (analysis, filtering, aggregation and cleansing) to determine if the event requires action, and producing an output that can be used to trigger an appropriate action. Justification for Hype Cycle Position/Adoption Speed: Current implementations typically involve a limited number of feeds and relatively simple filtering or analysis. CEP is one component of the enterprise nervous system concept, and to straight-through processing and real-time operations. Business Impact Areas: Applications include algorithmic trading, quality of service monitoring, best execution assessment. Benefit Rating: High. Market Penetration: Five percent to 20 percent of target audience. Maturity: Adolescent. Example Vendors: Atsmai, iSphere, Progress Software (acquired Apama), Streambase. Recommended Reading: • “Clarifying the Terms 'Event-Driven' and 'Service-Oriented Architecture'” • “Innovative Vendors in Business Event Management”

7. Tibco Oracle Progress Circles of players StreamBase AptSoft RuleCore Coral8 Aleri Labs RedRabbit Event Processing as part of full-fledged middleware General Event Processing products that can work with multiple platforms Event Processing capabilities embedded inside solution / applications (sold as products also) Actimize Leanway G4

8. IBM complete Roles Application Development Tools Development tools provider Event-DrivenApplications Services Provider General Practice IBM SW services Solution Provider Middleware Producer Industry Solutions IBM Middleware Embedded Base (ISV) Middleware Based IBM software Integration platform

9. Event Processing – segments

10. Major Segments EPBL - Event processing as part of business logic EPBO - Event processing as part of business observation EPPD - Event processing as part of problem determination EPID - Event processing as part of information distribution EPPS - Event processing as part of predictive systems

11. EPBL - Event Processing as part of business logic • Types: • Application integration through event processing • Event-driven business policies/rules • Application examples: • Adaptive workflows (Telco and others) • Trade management (FSS) • Automated shipping and receiving (Retail) • Just-in-time car rental allocation (Travel and Transportation) • Time-critical targeting (military)

12. EPBL – characteristics • Entire spectrum of mediators: transformation, routing, validation, enrichment. • Pattern detection: varies, requires rich language. • Non functional requirements: • All mission critical systems requirements (integrity, transaction support, recoverability, high availability, failover, clustering, security, audit) • May have a need for high throughput • predictability is needed in some applications • Time-out management is important. • Require end-user ability to maintain rules/policies.

13. EPBO - Event processing as part of business observation • Business activity monitoring • KPI – performance management • Exception management in BPM. • Sense and respond • Time-constrained Decision processes and analytics may be needed • A loop that contains actuators and monitoring the actuators results. • Applications: • RT compliance/ Fraud detection/AML (FSS, Telco) • Manufacturing S&R (industrial) • Adaptive policy setting (operational resilience) • Business health assessment (cross-industry) • Promotion evaluation (Retail) • SLA management of a process (cross industries)

14. EPBO – charachterisitcs • Languages: • Aggregation-land: trends, time series processing, statistical measures etc… • Data-base connection – retrospective processing • Send and response may need: • Predictability • Embedded analytics/decision modules • Business performance may need: • Metrics and KPI management • Derived data (“active data warehousing”) • Causality model. • Require business analysts interfaces.

15. EPPD – Characteristics • Main functionality: • Eliminate redundant events – filtering, remove duplication • Event correlation to determine symptoms • Problem/symptoms relations modeling • Both real-time and post-mortem (log files) • May need high throughput • Time interval typically short • Assumption that events can be lost: • No fault tolerance is required. • Partial pattern satisfaction may be useful. • Pattern language is focused on – sequence within short time-frames, thresholds, filtering condition. • Typical users: system administrators, developers, product support team.

16. EPPD – Characteristics • Main functionality: • Eliminate redundant events – filtering, remove duplication • Event correlation to determine symptoms • Problem/symptoms relations modeling • Both real-time and post-mortem (log files) • May need high throughput • Time interval typically short • Assumption that events can be lost: • No fault tolerance is required. • Partial pattern satisfaction may be useful. • Pattern language is focused on – sequence within short time-frames, thresholds, filtering condition. • Typical users: system administrators, developers, product support team.

17. EPID - Event processing as part of information distribution • Enable individuals/applications to subscribe to personalized derived events rather than raw events • Customer alert systems (banking) • Content-based geo-spatial subcriptions

18. EPID – charachterisitcs • Personalized and specialized form of the publish/subscribe paradigm • Typically loosely-coupled from the operational system, • Subscription can be by individuals, “communities of interests” or applications – routing decisions and management of subscriptions is important • Subscription can be to derived events and not only to raw events, which requires pattern detection – trends, content filtering combined with basic relations (conjunction, disjunction, negation, sequence etc..). • Requires end-user interfaces for subscriptions in some cases, development interfaces in cases of applications’ subscriptions.

19. EPPS - Event processing as part of predictive systems • Impact analysis: prediction of consequences of already observed events towards future events, or entity states. • May be in response to future change proposals (“change management”) • May be combined with “sense and respond” to mitigate or eliminate this impact. • Applications: • Change management in dependent cases (automobile, industrial) • Impact of IT problems on business health (cross-industry) • Includes ITSM Availability, Incident, Problem Management • On-line/real-time detection of fraud, money laundering. • On-line risk management.

20. EPPS – characteristics • Requires predictive tools embedded in event processing • Requires dependency model of events and entities • Requires mining tools to find causality relations • requires rich causality model with possible uncertainty handling (e.g. probabilistic engine) • Rich pattern language – different type of condition causality.

21. Event Processing - Platform Independent Architecture and Model

22. Event Driven Applications

23. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Glossary Event Consumer Event: (1). Something of interest that occurred in reality (state changed, fact becomes true) (2). The computerized message to report the event Event Cloud Event Producer: An entity (e.g. software artifact) that creates events and reports them (e.g. workflow) Event Sensor: An entity that senses events and reports them

24. Event Creation • Produced or sensed • Push, periodic pull, on-demand pull. • challenge: Increase the “event scope” • Events extracted from video streams • Events from RSS feeds • Events extracted from textual information

25. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Event Consumer Glossary Event Cloud Event Topic: A data-type channel for transporting events of a certain type Event Consumer: An entity which is reported about event creation Event Actor: An entity which produces automatic actions in response to an event

26. Event-driven Action • Notify/Store – for consumer • Trigger, Orchestrate – for actor (BPM event processing)

27. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Event Consumer Glossary Event Cloud Context: A set of criteria to partition the space of events according to temporal (e.g. time window), spatial (e.g. space window) and partitioning entity (e.g. platinum customers) Event Stream: A collection of events that arrive to a single consumer over a single event topic within a single context Event cloud: A collection of all event streams that a single consumer receives

28. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Event Consumer Glossary Event Cloud Event Processing Mediator: A software artifact that gets an “event cloud” as an input, and produces a collection of events as output.

29. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Event Consumer Glossary Event Cloud Pattern Detector: A software artifact that obtains an event cloud as an input, and creates a “complex event” –Example: the stock quote is monotonically decreasing within 2 hours. Return the collection of stock quotes. Complex event: An event that contains reference to all events that participate in the pattern

30. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Actor Architecture Event Consumer Glossary Event Cloud Event Processor: A software artifact that obtains a composite event and creates collection of derived events using some function on the input events, such as: Enrichment, Transformation, Aggregation, Split. Derived Event: An event that is calculated as a function of other events.

31. Event Processing Mediators

32. Event Event Topic Event Producer Event Pattern Detector Complex event Event Sensor Event Topic processor Event Consumer Derived Event Emitter Event Processing Mediator Event Processing Mediator Event Consumer Architecture Event Consumer Glossary Event Cloud Emitter: A software artifact that makes routing decisions for the derived events to consumers and actors: subscribers, itinerary based or intelligent routing

33. Several assertions • There are several different approaches to event processing. • Rule-based approach • Extended SQL-based approach (stream and active database) • Script oriented approach • Other approaches • Our aim is to have a model-based approach that is in a “platform independent” level, have appropriate tooling, and consistent with standard approaches. • Probably there is no “one fits all” solution in the run-time level, but from the model and tooling level everything should be seamless • We should strive for open architecture with: • Separation of concerns: • each function can be implemented by various run-time artifacts, each of them satisfies a subset of the appropriate functions, and given set of non-functional requirements. • We should own the “integration platform” for event processing: the biggest market for event processing will be in “application integration through event processing”. • We should have our own run-time artifacts, but allow easy integration with other run-time artifacts, mainly specialized ones.

34. Business-to-Business Interactions Portal Service Enterprise Service Bus Enterprise Information System Adapter Distinguished Services Workflow Business Activity Script, POJO, Stateless Session Bean Information MgmtXML Database EDA as part of SOA This is the famous SOA picture Each service can emit events – all the time, periodically, or by request The events are being processed Using event processing mediators (e.g. enrich) The processed events notify or orchestrate These building blocks are not tightly coupled to the ESB Event Processing can also be retrospective

35. Event Processing Network • EPN consists of a set of nodes (event processing mediations AKA as event processing agents, service, and event processing endpoints) • Following Separation of concerns principle • Each processing node has an “event pattern detector” part and “event emitter” part , which are the same for all mediator types. • This is a design pattern, but can also be run-time artifact, providing distributed set of lightweight run-time artifacts. Event Processing Mediator Event Emitter Event Pattern Detector Event Processor

36. Retrospective processing of events • We need to do all event processing in retrospect • A naïve way is to select all the raw events from the database and run them through the “memory” event processing. • This may not be a cost-effective way to do it when there is a large quantity of raw events… • Solutions: • Built-in database capabilities that can optimize to this type of processing • Converting this functionality to plain SQL (probably the short term game) Time

37. Towards an Event Processing Community Conclusion

38. Activities in progress • Establishing international community: • The kick-off meeting of the community has been the first event processing symposium in March 2006 • The steering committee consists of representatives of IBM, Tibco, Oracle, Progress, SAP, Streambase, Gartner and some academic people. • Follow-up symposium in November 2006 • Several community work-groups: • Terminology • Reference Architecture • Interoperability • ACM SIG in construction • Starting work with OMG on standards • More workshops and conferences • Vendors announcements – SOA 2.0