Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010

1. Event processing – past, present, futureVLDB 2010 Tutorial, Singapore, September 15th, 2010 Opher Etzion (opher@il.ibm.com)

2. An example to kick-off this tutorial: Event processing can help here.. THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Act: Passenger has been re-routed to another destination – send the luggage Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight Act: Bag has reached to the wrong aircraft

3. Another example: premature baby monitoring Patient is hooked up to multiple monitors (in hospital or at home) - the physician can set up event- based rules on multiple measurements and patient’s history when to send an alert and to whom: Defined Pattern

4. Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges

5. What is behind event processing ? How is it related to other computing terms? Where are its roots?

6. What is “event processing” anyway? or Event processing is a form of computing that performs operations on events

7. In computing we ptocessed events since early days Network and System Management

8. Emerging technologies in enterprise computing(Gartner Hype Cycle, Summer 2009)

9. What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.

10. What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation

11. In daily life we often react to events..

12. Sometime we even react to the occurrence of multiple events I closed the deal with the Australians I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore

13. Pattern detection is one of the notable functions of event processing Event Patterns

14. What we actually want to react to are – situation Toll violation Frustrated customer TOLL VILOATOR Sometimes the events can approximate or indicate with some certainty that the situation has occurred Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. FRUSTRATED CUSTOMER

15. Event processing is being used for various reasons

16. Event Driven Architecture Event driven architecture: asynchronous, decoupled; each component is autonomic.

17. Ancestor: active databases Composite events were inherited to event processing On event When condition Do action With coupling mode

18. Ancestor: Data Stream management system Source: Ankur Jain’s website

19. Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added

20. Ancestor: Messaging – pub/sub middleware

21. Ancestor: Discrete event simulation

22. Ancestor: Network and system management

23. Event processing – architecture, building blocks

24. Location Service Bid System Fast Flower Delivery Delivery Request Location GPS Location Bid Request Store Preferences Ranked drivers / automatic assignment Delivery Bid Manual Assignment Assignment Assignment System Flower Store Assignments, Bid alerts, Assign Alerts Van Driver Pick Up confirmation Delivery confirmation Ranking and Reporting System Assignments, Pick Up Alert Delivery Alert Driver’s Guild Control System Ranking and reports

25. Event processing network

26. Example of EPN – part of the FFD example

27. The seven Building blocks Event Channel Event Type Event Consumer Event Producer Event Processing Agent Context Global State

28. Event type definition

29. Producer – State Observer in workflows State observer Push: Instrumentation points; Pull: Query the state

30. Producer – Code instrumentation

31. Producer – syndication

32. Producers – streams to events

33. Producer – sensors

34. Consumer - Performance monitoring dashboard

35. Consumer - Ambient Orb

36. Producer and consumer - Sixth sense

37. Twitter as a consumer

38. Event Processing Agents Event Processing Agent Filter Transform Detect Pattern Translate Aggregate Split Compose Enrich Project

39. The EPA picture

40. A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event. Filter EPA

41. Transform EPA sub types

42. Sample of pattern types • all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set • any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set • absence pattern is satisfied when there are no relevant events • relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument • value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion. • always pattern is satisfied when all the relevant events satisfy the always pattern assertion • sequence pattern is satisfied when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set. • increasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A < e2.A • relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion • moving toward pattern is satisfied when for any pair of relevant events e1, e2 we have e1 << e2  the location of e2 is closer to a certain object then the location of e1.

43. Find the five highest bids within the bid interval Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Taken from the Fast Flower Delivery use case

44. Why do we need policies - A simple example: heavy trading scenario Why defining patterns is not that easy? Because we need to tune up the semantics • Given: • A stream of events of a single topic, about the activity in the stock market for a certain stock. • An event is produced every 10 minutes when there is trade in the stock. • Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes). • A selection specification: “trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values”. • How many times the trade programming is triggered ; • Which arguments are used in each triggering?

45. Pattern policies Evaluation policy—This determines when the matching sets are produced Deferred Cardinality policy—This determines how many matching sets are produced within a single context partition Repeated type policy—This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…

46. Pattern policies – cont. Consumption policy—This specifies what happens to a participant event after it has been included in a matching set Consume Bounded reuse Reuse Order policy—This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute

47. Our entire culture is context sensitive • In the play “The Tea house of the August Moon” one of the characters says: Pornography question of geography • This says that in different geographical contexts people view things differently • Furthermore, the syntax of the language (no verbs) is typical to the way that the people of Okinawa are talking When hearing concert people are not talking, eating, and keep their mobile phone on “silent”.

48. Context has three distinct roles (which may be combined) The events that relate to each customer are processed separately Partition the incoming events Grouping together events that happened in the same hour at the same location Grouping events together Different processing for Different context partitions Determining the processing

49. Temporal Spatial State Oriented Segmentation Oriented Context Definition Acontext is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions. A context may have one or more context dimensions.

50. Context Types Examples Segmentation Oriented “All Children 2-5 years old” “All platinum customers” Temporal Spatial “Every day between 08:00 and 10:00 AM” “A week after borrowing a disk” “A time window bounded by TradingDayStart and TradingDayEnd events” “3 miles from the traffic accident location” “Within an authorized zone in a manufactory” Context State Oriented “Airport security level is red” “Weather is stormy”