Blending Complex Event Processing with the RETE Algorithm iCEP@FIS 2008

1. Blending Complex Event Processing with theRETE AlgorithmiCEP@FIS 2008 Kay-Uwe Schmidt, SAP Research 28.09.2008 Addl. Authors: Roland Stühmer (SAP), Ljiljana Stojanovic (FZI)

2. Agenda • 1. The Evolution of Rule-based Web Applications • State of the Art in Combining CEP and Rete • Enhancing RETE with SnoopIB • Evaluation and Conclusions

3. From Server- to Client-sideAdaptive Hypermedia Systems (AHSs) • Traditional adaptation strategies have been intensively studied • Web Page Paradigm: Every Web page in a series of pages is downloaded separately. • Conventional architecture: tracking of user clicks, the user modeling, as well as the adaptation take place on the server. • Limited user tracking possibilities • User requests seen by the server • Subset of user clicks • Adaptation only on behalf of an explicit user request • On-the-fly adaptation not obtainable

4. From Server- to Client-sideAdaptive Hypermedia Systems (AHSs) • RIAs / AJAX • New user tracking possibilities • New user interface adaptation possibilities • Look and feel of desktop applications • Highly responsive user interfaces • Adaptation rules still on the server • On-the-fly adaptation not obtainable

5. From Server- to Client-sideAdaptive Hypermedia Systems (AHSs) • Client-side rule processing • Reduction of client-server-communication to a minimum • Enhanced user tracking capabilities • In-time response to user actions • On-the-fly adaptation is obtainable

6. Run-time Architecture

7. Agenda • 1. The Evolution of Rule-based Web Applications • State of the Art in Combining CEP and Rete • Enhancing RETE with SnoopIB • Evaluation and Conclusions

8. State of the Art: Production Rule Systems • Pattern Recognition Algorithms • RETE algorithm [1, 2] • Others • TREAT [3], LEAPS [4] or GATOR [5] • CA-Rules • OPS5 [6] • Scope • Long-living Business Objects 1. Charles L. Forgy. On the efficient implementation of production systems. PhD thesis, Carnegie Mellon University, Pittsburgh, PA, USA, 1979. 2. Charles L. Forgy. Rete: a fast algorithm for the many pattern/many object pattern match problem. Artificial Intelligence, 19:17-37, 1982. 3. Daniel P. Miranker. Treat: A better match algorithm for ai production system matching. In AAAI, pages 42-47, 1987. 4. Don Batory. The leaps algorithms. Technical report, University of Texas at Austin, Austin, TX, USA, 1994. 5. E. N. Hanson and M. S. Hasan. Gator: An optimized discrimination network for active database rule condition testing, technical report tr-93-036. Technical report, CIS Department, University of Florida, 1993. 6. Charles L. Forgy. The ops5 user's manual. technical report cmu-cs-81-135, 1981.

9. Event Condition Action Rules • First mention of the ECA paradigm by U. Dayal, A. P. Buchmann, and D. R. McCarthy in 1988 • “Rules are objects too: A knowledge model for an active, object-oriented databasesystem”. In Lecture notes in computer science on Advances in object-oriented database systems, pages 129-143, New York, NY, USA, 1988. Springer-Verlag New York, Inc. • This paper describes work in progress on the knowledge model (an extended data model that includes constructs for representing rules) of HiPAC, an active, object-oriented DBMS. • “Central to our knowledge model is the concept of event-condition-action (ECA) rules, which generalizes the many different mechanisms introduced previously in the literature to support active DBMS functions. • The event part of an ECA rule specifies database operations, temporal events, or signals from arbitrary processes; • the condition part specifies database queries; • and the action part specifies a program. • When the event occurs (is signaled, the condition is evaluated; if the condition is satisfied, the action isexecuted.” • ON event IF condition DO action

10. Event Processing • The two kinds of event processing: • Complex Event Processing (CEP) • Event Stream Processing (ESP) • Dealing with different problems in event processing using different approaches • ESP – extraction of simple events from a stream • Events are totally ordered by time • Emphasis of ESP on efficiency for high throughput and low latency • Algorithmic stock trading • CEP – extraction of complex event patterns from a cloud • Only a partial temporal order of events • Other partial order of interest for CEP is for instance causality • More time and memory needed • Business Process Monitoring • CEP is a superset of ESP • CEP and ESP nowadays adopt each others approaches

11. Complex Event Detection Algorithms • Finite State Automata • Ode 1992 • Transformation of complex event expressions into deterministic finite automata • Convenient model to define the semantics of complex event operators • Downside no acceptance of overlapping occurrences of the same complex event • Colored Petri Nets • SAMOS 1994 • Convenient model to define the semantics of complex event operators • Also the detection of overlapping occurrences is possible • Graph-based Approaches

12. Graph-based Approaches • Sentinel based on Snoop, SnoopIB in the mid 90’s • Construction of the graph from the event expressions • Example: (E1,E2);E3 • The graph is a directed acyclic graph and generally does not form a tree • Nodes may have several parents, when their represented expression is part of more than one complex events • There is no single root node, when there is no overarching, single most complex event. ; , E1 E2 E3

13. Overview of the State of the Art inEvent Processing EventProcessing Complex EventProcessing Event StreamProcessing • Graph-based Approaches • Finite State Automata • Colored Petri Nets • Pattern Matching in Streams then DB Tables, Sliding Windows • StreamSQL

14. Adding Complex Event Processing to Rete • Research Question • How can both pattern recognition algorithms be integrated in an efficient manner? • Different approaches • Merging of event processing into rule processing • Merging of rule processing into event processing • Loose coupling of event with rule processing

15. Conceptual Differences between Facts and Events [7] [8] 7. Opher Etzion. On events and data. Online Article. http://epthinking.blogspot.com/2008/07/on-events-and-data.html, July 2008. 8. Francois Bry and Michael Eckert. Twelve theses on reactive rules for the web. In Torsten Grust, Hagen Höpfner, Arantza Illarramendi, Stefan Jablonski, Marco Mesiti, Sascha Müuller, Paula-Lavinia Patranjan, Kai-Uwe Sattler, Myra Spiliopoulou, and Jef Wijsen, editors, EDBT Workshops, volume 4254 of Lecture Notes in Computer Science, pages 842{854. Springer, 2006.

16. Maloof ’93 • Maloof et al. [9] • Reasoning about relative time representations • Temporal operators before, during and after • Interval-based semantics • Details of implementation unclear 9. M.A. Maloof and K.J. Kochut. Modifying rete to reason temporally. Tools with Artificial Intelligence, 1993. TAI '93. Proceedings., Fifth International Conference on, pages 472-473, Nov 1993.

17. Berstel ’02 • Berstel [10] • Temporal operators before and after • Time-point-based semantics (no correct detection of sequences) • Garbage-collection mechanism • Events are special facts which are asserted through a special API 10. Bruno Berstel. Extending the rete algorithm for event management. In Proc. Ninth International Symposium on Temporal Representation and Reasoning TIME 2002, pages 49-51, Washington, DC, USA, 7{9 July 2002. IEEE Computer Society.

18. Walzer ’08 • Walzer [11, 12] • 13 temporal relations from Allen [13] – all relations intervals can have • Temporal operators for each of the 13 relations are introduced, e.g. equal, before, after, during, overlaps, etc. • Interval-based semantics • Sliding window garbage collector 11. Karen Walzer, Alexander Schill, and Alexander Löser. Temporal constraints for rule-based event processing. In Aparna S. Varde and Jian Pei, editors, PIKM, pages 93-100. ACM, 2007. 12. Karen Walzer, Tino Breddin, and Matthias Groch. Relative temporal constraints in the rete algorithm for complex event detection. In DEBS '08: Proceedings of the second international conference on Distributed event-based systems, pages 147-155, New York, NY, USA, 2008. ACM. 13. James F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM, 26(11):832-843, 1983.

19. Comparison • Time-point vs. interval-based semantics • Number of temporal operators • Working memory holds all events • Garbage collection • No real publish / subscribe • New rules can match events from the past • Retained by an already existing rule • The match of the new rule might depend on older ones • Unclear semantics • Events are not facts • Immutable  Modify and Retract operation seem not applicable • No notion of event selection and consumption

20. Implications of Holding Events in Working Memory • Garbage collection • No real publish / subscribe • New rules can match events from the past • Retained by an already existing rule • The match of the new rule might depend on older ones • Unclear semantics • Events are immutable • Modify and Retract operation seem not applicable • No notion of event selection and consumption • Contexts: recent, chronicle, continuous, cumulative [18] 18. Sharma Chakravarthy, V. Krishnaprasad, Eman Anwar, and S. K. Kim. Composite events for active databases: Semantics, contexts and detection. In Jorge B. Bocca, Matthias Jarke, and Carlo Zaniolo, editors, 20th International Conference on Very Large Data Bases, September 12-15, 1994, Santiago, Chile proceedings, pages 606-617, Los Altos, CA 94022, USA, 1994. Morgan Kaufmann Publishers.

21. Agenda • 1. The Evolution of Rule-based Web Applications • State of the Art in Combining CEP and Rete • Enhancing RETE with SnoopIB • Evaluation and Conclusions

22. Coupling RETE with CEP Figure: • The RETE network, depicted at the top, is responsible for rule conditions and finds matching facts fulfilling rule conditions. • The event-detection graph, at bottom left, is responsible for event specifications and finds events matching an event pattern. • Both RETE and the event graph propagate their results to newly introduced top nodes. • These nodes trigger rule actions. Example: (E1,E2);E3 x.y=1 & x.z=2  Action

23. Advantages • The event graph can have nodes with more than two input edges • This more closely resembles higher-level event operators from Snoop which we are using. • Also garbage-collection comes more naturally • Events are stored and discarded in the queues of event nodes where they are still needed. There is no global repository for unused or partially unused events like the working memory from RETE. • Consumption modes are selectable (on a per-node basis, if necessary) • This allows for the controlled selection of events from a stream, if more than one events have arrived and might be fitting for a match. • For an ECA rule action to fire, an event must be detected, and for its complete interval, the condition must be fulfilled • Vice versa this means that no events need to be correlated as long as RETE supplies no matched tokens.

24. Agenda • 1. The Evolution of Rule-based Web Applications • State of the Art in Combining CEP and Rete • Enhancing RETE with SnoopIB • Evaluation and Conclusions

25. Conclusions • Rete seems not a natural fit for doing event processing • Reasoning over persistent and mutable vs. transient and immutable facts • Many changes towards temporal capabilities as well as event expiry must be added • Our approach: Blending CEP with RETE • Solution to avoid the indicated mismatch of requirements • Combination of dedicated event processing nodes with RETE nodes • Pending evaluation

26. Questions? Thank you!

27. Contexts