Event Processing course Spring 2010 The Current State Of The Practice - Products Overview

1. Event Processing course Spring 2010 The Current State Of The Practice - Products Overview Ella Rabinovich ellak@il.ibm.com

2. Languages • StreamBase Event Flow • Sybase CCL (formerly Coral8 then Aleri) • Apama EPL • ruleCore • IBM Active Middleware Technology (AMiT) • Esper EPL • ETALIS

3. FFD Scenario

4. StreamBase • Background • Streambase is a VC owned company that focuses mainly of capital markets • It is a derivative of an academic project in MIT/Brown/Brendeis called “Aurora”. • Currently the largest “pure play CEP” in the market • It won this year a prestigious innovation award in Davos. • Started as StreamSQL it migrated its IDE to be based on some variation of event processing network, called: Event Flow, but the primitives are still closely related to the SQL concepts.

5. StreamBase • StreamBase Server – core processing engine • StreamSQL – SQL-like language for stream-event manipulations • Queries against stored data • Queries against real-time streams • Queries over time windows • Built-in operators • StreamBase adapters – connects to variety of source feeds • XML, CSV, Excel, SQL, JMS • Custom – Java, C++, .NET • StreamStudio – IDE for application authoring and debugging • Eclipse based

6. StreamBase Studio

7. StreamBase Debugger

8. StreamBase Pattern Matching

9. FFD Examples

10. FFD Examples

11. Sybase (Aleri(

12. Sybase • Background: • Sybase acquired Aleri (in early 2010), that acquired Coral8in early 2009. SAP announced Sybase acquisition in May 2010, which makes all big companies offering event-processing capabilities. • CCL is the original Coral8 language, a descendent of the academic stream project in Stanford, and the CQL language, the language is an hybrid language: SQL extension with some pattern primitives • Currently Sybase has several “EP platforms” and is trying to consolidate them

13. Sybase CCL • Server – processes incoming data streams againstregistered continuous queries. • Continuous Computation Language(CCL) Compiler – compiles and optimizes queries written in CCL (an SQL-like language) into an internal representation suitable for very efficient execution. • Input and Output Adapters – translates external data to internal data streams and back. • Aleri Studio – allows the user to create streams and CCLqueries, to control the server, to view all streams, and to debug applications in an interactive graphical development environment.

14. Sybase CCL Studio

15. Sybase CCL Engine • Filters • Various kinds of windows • sliding, jumping, counting, and others • Aggregators to aggregate state over windows • Count, Max, Min, Avg, Sum, StdDev etc. • Transformers - transform messages and evaluate expressions • Joiners to correlate streams • Stream-to-window correlations, stream-to-historical data correlations, inner/outer joiners, etc. • A GroupBy clause - allows applications to distinguish statesby individual column definitions • Matching

16. CCL - Continuous Computation Language • Windows and aggregation • Average stock price over a sliding window of five minutes by symbol Insert into StreamAvgPrices Select Symbol, Avg(Price) From StreamTrades Keep 5 minutes Where Volume >= 10000 Group By Symbol

17. CCL – Pattern Matching • Correlation • Network security application produces an alert if anIntrusion Detection System (IDS) and a Virus Checker detect an attack from the same IP address within 10 minutes: Insert into StreamAlertCommonIP Select StreamIDSAlerts.IP From StreamIDSAlerts Keep 10 minutes, StreamVirusCheckAlerts Keep 10 minutes, Where StreamIDSAlerts.IP= StreamVirusCheckAlerts.IP

18. CCL – Pattern Matching • RFID monitoring application • Checks if a tag has been seen by readers A and B, then C, but not D, within a 10 second window. Insert into StreamAlerts Select StreamA.id From StreamA a, StreamB b, StreamC c, StreamD d Matching [10 seconds: a && b, c, !d] On a.id = b.id = c.id = d.id

19. Apama

20. Apama • Background: • Apama started as academic project in Cambridge UK,the only one of the original EP start-ups established in the late 1990-ies that survived • Acquired in 2005 by Progress, and is considered one of the top divisions in Progress • Currently considered as the leading vendor in the market(in terms of number of customers and revenues) • The language is imperative, looks like C or Java • but not object oriented

21. Apama – Development env. • Apama Studio – an Eclipse based IDE • Event Processing Language (EPL) – Java-like language • Add and manage component files that make up the application • Specify the adapters, dashboards, and scenarios that are necessary for the application. • Specify the configuration properties necessary for launching the application • Automatic MonitorScript validation • Content assistance, Auto-completion • Running & debugging • …

22. Apama EPL – Handling Events • A Correlator can receive events in several ways: • Adapter • Events file (*.evt) – from the Apama Studio • Apama engine_send utility - send events manually intothe correlator • Apama Router - optimizes performance by partitioning events and sending them to various correlators

23. Apama EPL – FFD Examples

24. Apama EPL – FFD Examples

25. Apama EPL – FFD Examples

26. Apama Simulation Studio – cont.

27. ruleCore • Background: • ruleCore is a small company based in Sweden • It was the first one to introduce SAAS in event processing • The language is based on ECA rules

28. ruleCore • No programming required • Simple declarative approach; few new concepts to learn • Completely event-driven, easy to integrate • Uses known standards: XML, DOM, XML Schema, XPath and XSLT • Event based run-time monitoring • SAAS (Software as A Service)

29. ruleCore - SAAS • Business model – charging per incoming event • Subscription to the server using service instance identified by ID • Endpoint – JMS queue, Web service or e-mail • A single open connection at a time for input and output endpoint for the same service instance

30. ruleCore - Terminology • Event stream view - a unique context of events • a view contains a window into the inbound stream of events andcontains commonly only semantically related events • Situation - an interesting combination of multiple events as they occur over time • An item with an RFID tag being picked up from the shelf and then moving past the checkout without being paid for • Rule - an active event processing entity reacting to specific combinations of inbound events over time • Action - the last part of a rule's evaluation in response to a detected situation

31. ruleCore – A “clean” CEP instance

32. ruleCore – RuleDef

33. ruleCore - View • View – defined and updated from incoming event stream • Contains definitions of properties that must be hold • View properties • Age – maximum age of the events in the view • Count – maximum number of the events in the view • Match – fields of the event to be matched

34. ruleCore - Situation • Situation - interesting combination of multiple events as they occur over time • Situation detector – number of detection nodes • how different events must be related - in sequence, logically or temporally

35. ruleCore - Action • Action – last step of the rule evaluation • Executed when a situation is detected • New event is created • Event body is XSLT transformation

36. IBM Active Middleware Technology (AMiT) • Background • Research asset developed in IBM Haifa Research Labs • User friendly graphical interface • No coding required • Allows simulation of a predefined scenario • Supports J2SE, J2EE, J2ME

37. Set of CEP operators (patterns) Counting, joining, temporal, aggregation, absence etc. Actions are taken upon patterns detection AMiT Authoring Tool Events Event Processing Rules Definitions Build Time Run Time Detected Situations Runtime Engine Event Sources Actions Situation Detection

38. AMiT Terminology • Event type • Situation (EPA) • Database (global store) • Lifespan (temporal context) • Key (segmentation oriented context)

39. AMiT - Event Type

40. AMiT - Situation

41. AMiT - Lifespan • A lifespan is a time interval during which situation detection is relevant. A situation (agent) must use a lifespan, and it can be detected only when it’s lifespan is open. • A lifespan is bounded by an initiator and a terminator. • Initiator – beings the lifespan. The initiator can be an event, absolute time, startup (CEP engine startup). • A lifespan can use more than one initiator. • Terminator – ends the lifespan. The terminator can be event, expiration time, absolute time, or no-terminator (lifespan will always stay open). • A lifespan can use more than one terminator.

42. AMiT - Lifespan

43. OrderId Key String CustomerOrder OrderId CustomerOrderCancel OrderId OrderDelivery Id OrderShipmentReport Number AMiT - Keys • A key is a collection of event attributes that are used tomatch different events. • These attributes must have syntactic compatibility, and are usually semantically equivalent.

44. AMiT - Keys

45. Esper EPL • Background • Open source project – but sold as OEM to some vendors(BEA in the past, Event Zero) • The most popular open source in that area • EPL – Event Processing Language • SQL like language components embeddable in java • Some of the functionality needs to be written in Java • Esper for Java & Nesper for .NET • There is an attempt to create a commercial version under the name “EsperTech”

46. Esper EPL • Open source project • EPL – Event Processing Language • SQL like language components embeddable in java • Esper for Java & Nesper for .NET

47. Esper EPL – Event Representation • Supports event-type inheritance and polymorphism • Events can be plain Java objects, XML and java.util.Map including nested objects and hierarchical maps • Event properties can be simple, indexed, mapped or nested

48. Esper EPL – Aggregation

49. Esper EPL – FFD Example

50. Esper EPL – FFD Example /** finishing the delivery when we get no one to receive message */ on NoOneToReceiveMSG nc delete from DeliveryBidW d where d.requestId= nc.requestId; /* * alert for no one to receive msg */ insert into AlertW(requestId, message, driver, timestamp) select a.requestId, "no one to receive the delivery", a.driver,current_timestamp() from pattern[ every a=NoOneToReceiveMSG ];