1 / 108

Event Processing with the PADRES Publish/Subscribe System

MIDDLEWARE SYSTEMS. RESEARCH GROUP. Event Processing with the PADRES Publish/Subscribe System. The PADRES Team. http://www.padres.msrg.utoronto.ca. Hans-Arno Jacobsen Bell University Laboratory Chair Middleware Systems Research Group University of Toronto. Current PADRES Team.

drea
Download Presentation

Event Processing with the PADRES Publish/Subscribe System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MIDDLEWARE SYSTEMS RESEARCH GROUP Event Processing with the PADRES Publish/Subscribe System The PADRES Team http://www.padres.msrg.utoronto.ca Hans-Arno Jacobsen Bell University Laboratory Chair Middleware Systems Research Group University of Toronto

  2. Current PADRES Team • Chen Chen • Alex Cheung • Alton Chiu • Amer Farroukh • Patrick Lee • Guoli Li • Bala Maniymaran • Serge Mankovskii, CA • Vinod Muthusamy • Reza Sherafat • Naweed Tajuddin • Chunyang Ye • Young Yoon Countless alumni (see our web site.) PADRES started around 2004 CANOE Summer School, Norway, 2009

  3. Try it out: Java Kick Start PADRES! http://www.padres.msrg.utoronto.ca CANOE Summer School, Norway, 2009

  4. Querying the Future CANOE Summer School, Norway, 2009

  5. Amazon to Chapters to You .... Monday, October 10th in Cyberspace Thursday, November 15th, in Toronto Your book “...” is available at .... $10 off CANOE Summer School, Norway, 2009

  6. Border Security Scenario [class,audit], [firm,ACME], [trust, 0] [class, reading],[shipID,ACME123],[level, 4] [class,manifest],[shipID,ACME123],[firm,ACME] CANOE Summer School, Norway, 2009

  7. Business Process Example Loan Application Processing Request Approval Store inDB … Reject Creditcheck 2 Checkscore Checkscore 2 Creditcheck Approve Send toofficer else … else Request Submission Check Completion CANOE Summer School, Norway, 2009

  8. Large-scale Business Processes Vendor Goods selection Goods delivery Dispatch B Packaging Pick-up goods Out-stock B FedEx Delivery Pick up Sale prediction Sign Contract Sale Fill order Determinate plan Process Check order CCC administrate Fill out-stock bill Check stock Manufactory Confirm features Design Fill dispatch bill Determinate plan Control Prototype Out Take Raw materials Execute plan Warehouse Material Out-stock B Pay Credit card Check Assign Audit Process control Make plan Target price Signature Raw Checkdealer Checkcredit Finance Confirm Approval Approval Monitoring Feature selection Print receipt Validate Statistic Monitor Marketing Requirement collection Feedback Affirm order Chart Strategy Design Marketing Manufactory Order Payment CANOE Summer School, Norway, 2009

  9. Many applications are driven by asynchronous state transitions. Something happens, … an appropriate reaction is expected and required. Asynchronous state transitions represent events. A process is triggered, a request submitted, … Many applications require event management and processingcapabilities to run effectively. What is the Common Denominator? CANOE Summer School, Norway, 2009

  10. These applications are driven by events Information matching the query is found and indexed Person walks by a bookstore Loan request is submitted online Radioactivity is detected Abstractly speaking events are disseminated and filtered against expressions or queries In Terms of the Examples queries events CANOE Summer School, Norway, 2009

  11. What Event Processing Support is Required ? • De-coupled & loosely coupled many-to-many interactions • Fine-grained filtering • In-network processing • Composite event detection and event correlation • Event dissemination CANOE Summer School, Norway, 2009

  12. A E B C F D Many Applications are Event-based Workflows, business processes and job scheduling Supply chain and logistics Job A done In flight Trigger Delivered Fault Order Event-Based Callback Razor SKU Invoke Loan Light Temperature Transform Service oriented architectures RFID and sensor networks CANOE Summer School, Norway, 2009

  13. What is an event? • Etymology • Latin: eventus; • evenireto happen; • e- + venire to come CANOE Summer School, Norway, 2009

  14. Various Interpretations A postulated outcome, something that follows as a result or consequence, condition, or eventuality; “in the event that I am not there, you may …” Any of the contests in a program of sports, a social occasion or activity, a world event The fundamental entity of observed physical reality represented by a point designated by three coordinates of place and one of time in the space-time continuum postulated by the theory of relativity A subset of the possible outcomes of an experiment (probabilities) Something that happens, a noteworthy happening, an occurrence CANOE Summer School, Norway, 2009

  15. Event – My Working Definition An event is an instantaneous and asynchronousstate transition in the environment of interest to applications, systems, users. Event versus eventoccurrence Like type or class versus instance In my definition events do not extend over time Interval events are defined by starting and ending events (a.k.a. composite events) My definition does not capture world events Maybe important for a theory of events, but less so for the effective processing of events My definition does not capture interval events Events associated with a time span, but those can be modeled as composite events CANOE Summer School, Norway, 2009

  16. Synonymously Used The terms Event Event occurrence (Event) message Event object are often used synonymously in practice. Also the pattern for detecting an event is often simply referred to as event. CANOE Summer School, Norway, 2009

  17. Event source Event producer Event generation Event publisher … sink … consumer … capture … subscriber Similarly, … CANOE Summer School, Norway, 2009

  18. Event Processing • Event Processing is an emerging area and discipline that aims to define and develop • Abstractions • Architectures • Systems • Languages • Patterns • Models • Standards for the processing of events in systems, applications, and by end users. CANOE Summer School, Norway, 2009

  19. A Note on CEP Complex Event Processing (CEP) Event processing must not be complex Or else, a marketing nightmare, I suspect  Meant was probably Complex-event Processing Or maybe the processing of complex events Not the complex processing of events Let’s just keep to Event Processing and try to make it as simple as possible,  CANOE Summer School, Norway, 2009

  20. Events • Many applications are inherently event-based. • An event is an instantaneous and asynchronousstate transition in the environment of interest to applications, systems, users. CANOE Summer School, Norway, 2009

  21. Agenda • What is the right abstractions for processing events? • Our point of view • PADRES Event & Services Bus (ESB ) • Innovative & Advanced Features • Building Applications with PADRES CANOE Summer School, Norway, 2009

  22. What Abstractions Do Not Work? Cum gran salis • Databases • Great for managing historic data • But what about future data (e.g., events) • Data streams • Great for managing structured streams of tuples • But what about un-structured, multi-typed, sporadic, un-ordered events from many sources • Rule-based expert systems • Great for inference and reasoning • But what about managing large numbers of fined-grainedfilters in distributed environments CANOE Summer School, Norway, 2009

  23. What Abstractions Enable Event Processing? • The afore-mentioned points can best be addressed by • The content-based publish/subscribe model • Realized by content-based message routing • Events are conveyed as publications. • Event listening, filtering and correlating is based on content-basedsubscriptions managed by the pub/sub system. CANOE Summer School, Norway, 2009

  24. Publish/Subscribe 101 • Not all publish/subscribe is equal • Publish/Subscribe models and evolution • Channel-based • OMG CORBA Event Service, … • Topic-based • WS Notifications, OMG Data Dissemination Service … • Type-based • OMG Data Dissemination Service (partially), … • Content-based • The PADRES ESB (see below), … • State-based • Subject Spaces CANOE Summer School, Norway, 2009

  25. Notification Notification Content-based Publish/Subscribe TSX Stock markets NASDAQ NYSE Publisher Publisher AMGN=58 Publications IBM=84 ORCL=12 JNJ=58 HON=24 INTC=19 MSFT=27 Broker(s) Subscriptions: IBM > 85 ORCL < 10 JNJ > 60 Subscriptions Subscriber Subscriber CANOE Summer School, Norway, 2009

  26. That’s Like Data Base Querying  !! query publication data tuples subscriptions About past About future sets of tuples sets of tuples Query and subscription are very similar. Data tuples and publication are very similar. However, the two problem statements are inverse. CANOE Summer School, Norway, 2009

  27. Language and data model Boolean functions over predicates Predicates are attribute-operator-value triples [class = reading] Subscriptions are conjunctions of predicates [class = reading],[level > 3] Publications are sets of attribute-value pairs [class,reading],[level,3],[id,5637] The Content-based Model events P/S notifications CANOE Summer School, Norway, 2009

  28. Given a set of subscriptions, S, and a publication, e, return all s in S matched by e. e is referred to as event or publication Splitting hairs The Event is the state transition of interest in the environment The Publication is the information about the event submitted to the publish/subscribe system Simple problem statement, widely applicable, and lots of open questions Publish/Subscribe Matching Problem CANOE Summer School, Norway, 2009

  29. Scope of the Content-based Model • Matching semantic • A subscription matches if all its predicates match • Approximate semantic (e.g., close to, cheap, sunny) • Semantic and similarity-based matching • Probabilistic … • Top-k matches CANOE Summer School, Norway, 2009

  30. Content-based Message Routing [class,=,stock],[symbol,=,YHOO] [class,=,stock],[symbol,=,YHOO],[price,>,20] S1 A1 S1 P1 [class, stock],[symbol, YHOO],[price,25] [class, stock],[symbol, YHOO],[price,45] [class, stock],[symbol, MSFT],[price, 55] P2 S2 A2 S2 [class,=,stock],[price,>,40] [class,=,stock],[symbol,=,MSFT],[price,>,50] Event-Based Content Routing Flexible Decoupled Declarative Responsive CANOE Summer School, Norway, 2009

  31. Publication Space Sub intersecting Adv Pub matching Sub height height 90 75 70 70 20 25 20 32 weight weight Adv: [height > 70],[weight > 25] Sub: [height > 75],[weight > 20] Sub: [height > 75],[weight > 20] Pub: [height , 90],[weight , 32] CANOE Summer School, Norway, 2009

  32. Standards CORBA Event Service CORBA Notification Service OMG Data Dissemination Service Java Messaging Service WS Eventing WS Notification, WS Topics, WS Brokered Notifications INFO-D (Grid Forum) AMQP Emerging technologies RSS aggregators PubSub.com, FeedTree Real-time data dissemination TIBCO, RTI Inc., Mantara Software Application integration Softwired Hardware-based brokers Sarvega (Intel), Solace Systems, DataPower (IBM) Publish/Subscribe in Industry CANOE Summer School, Norway, 2009

  33. Publish/Subscribe in Academia • Research projects • Elvin (Australia) • Gryphon (IBM) • Hermes (Cambridge) • LeSubscribe (INRIA) • PADRES (Toronto) • REBECA (Darmstadt) • SIENA (Boulder) • ToPSS (Toronto) • … many more CANOE Summer School, Norway, 2009

  34. ToPSS - The Toronto Publish/Subscribe System Family [2000 – present] • Matching algorithms • Language expressiveness vs. efficient matching • Routing protocols • Network architectures & scalability • Higher level abstractions • Workflow execution • Monitoring A-ToPSS (approximate) ToPSS (matching) CS-ToPSS (composite subs) S-ToPSS (semantic) L-ToPSS (location-based) Rb-ToPSS (rule-based) X-ToPSS (XML matching) persistent-ToPSS (subject spaces) M-ToPSS (mobile) P2P-ToPSS (peer-to-peer) LB-ToPSS (load balancing) Federated-ToPSS (federation of ToPSS brokers) Ad hoc-ToPSS (ad hoc networking) Historic-ToPSS (historic data) FT-ToPSS (fault tolerance) JS-ToPSS (job scheduling) BPEL-ToPSS (BPEL execution) CANOE Summer School, Norway, 2009

  35. Publish/Subscribe • Decoupling and interaction among many sources & many sinks • Event filtering & matching • Event & data dissemination • Query the future (relaxed later) • Service orchestration and choreography CANOE Summer School, Norway, 2009

  36. Agenda • PADRES Event & Services Bus (ESB ) • PADRES ESB & Vision • PADRES Event Language • PADRES Internals • Innovative & Advanced Features • Building Applications with PADRES CANOE Summer School, Norway, 2009

  37. PADRES Events & Services Bus • First generation of students, when I looked away  • Peng Alex David aRno Eli Serge • PADRES is Publish/subscribe Applied to Distributed Resource Scheduling • PAdres is Distributed REsource Scheduling • http://www.padres.msrg.utoronto.ca • http://padres.msrg.utoronto.ca Acknowledgements CANOE Summer School, Norway, 2009

  38. PADRES Vision 38 CANOE Summer School, Norway, 2009 2014/9/14

  39. PADRES Events & Services Bus • Consists of pub/sub message brokers • Content-based publish/subscribe interface • Content-based message routing • Store-and-forward message queuing • Comprised of a federation of brokers deployed as overlay • Offers a slim client library for development • Available under the Eclipse Public License (EPL) – a BSD-like open (source) license model ! • http://padres.msrg.utoronto.ca CANOE Summer School, Norway, 2009

  40. PADRES Event Language CANOE Summer School, Norway, 2009

  41. PADRES Event Language • Event schema • Event instances (a.k.a., publications or events) • Subscriptions (a.k.a., atomic subscriptions) • Composite subscriptions • Historic event queries (a.k.a., historic subscriptions) • Combinations of the former The above are unified in the PADRES SQL (PSQL), not covered today. CANOE Summer School, Norway, 2009

  42. PADRES Event Language • Event schema: Advertisements • A1: [class,=,reading], [shipID,=,*], [level,<,10] • A2: [class,=,manifest], [shipID,=,*], [firm,=,*], [content,=,*] • A3: [class,=,audit], [firm,=,*], [trust,>=,0] • Source: Publications / events • P1: [class, reading],[shipID,ACME123],[level, 4] (induced from A1) • P2: [class, manifest],[shipID,ACME123],[firm,ACME] (induced from A2) • P3: [class, reading],[shipID,ACME123],[level, 12] (not induced from A1) • Sink: Atomic & (composite) subscriptions • S1: [class,=,reading], [level,>,3] • S2: [class,=,audit], [firm,=,*],[trust, >=, 3] CANOE Summer School, Norway, 2009

  43. Composite Subscription I AND S5 AND OR OR S1 S2 S3 S4 Composite subscriptions (CS) are used for event correlation, in-network filtering, and the detection of composite events (complex event). A composite event is the constellation of events being detected by the composite subscription. Use of composite subscriptions: Event correlation Business process management (shown later) Business activity monitoring (shown later) CS={ {S1OR S2} AND {S3OR S4} AND S5 } Si are atomic subscriptions. I.e., they are satisfied by a single, multi-attribute publication / event. CANOE Summer School, Norway, 2009

  44. firm AND shipID AND S3 S1 S2 Composite Subscription II • A composite subscription consists of atomic subscriptions connected by operators • AND, OR • Variable join • Examples: [class,=,reading], [level,>,3] OR [class,=,audit], [firm,=,*], [trust, <=, 3] [class,=,reading], [shipID,=,$X], [level,>,3] AND [class,=,manifest], [shipID,=,$X], [firm,=,$Y], [content,!=,fertilizer] AND [class,=,audit], [firm,=,$Y], [trust,>,7] CANOE Summer School, Norway, 2009

  45. Atomic Historic Event Query • Future atomic subscription • Historic atomic subscription • Hybrid atomic subscription [class,=,reading], [level,>,3] [class,=,reading], [level,>,3], [start_time,<,NOW] # of events per match [class,=,reading], [level,>,3], [start_time,=,NOW-1h], [end_time,=,NOW+4h] now Past Future time CANOE Summer School, Norway, 2009

  46. Composite Historic Event Query [class,=,reading],[level,>,3],[shipID,=,$X] AND [class,=, manifest], [shipID,=,$X],[content,!=fertilizer] • Future composite subscription • Historic composite subscription • Hybrid composite subscription [class,=,manifest],[firm,=,$X],[start_time,<,NOW] AND [class,=,audit],[firm,=,$X],[trust,>,7],[start_time,<,NOW] # of events per match • [class,=,reading],[shipID,=,$X],[level,>,3] • AND • [class,=,manifest],[shipID,=,$X],[firm,=,$Y], • [content,!=,fertilizer] • AND • [class,=,audit],[firm,=,$Y],[trust,>,7], • [start_time,=,now-2 months] now Past Future time CANOE Summer School, Norway, 2009

  47. PADRES Event Language • Queries about future events • Queries about past events • Queries that express composite events over past and future CANOE Summer School, Norway, 2009

  48. PADRES Internals CANOE Summer School, Norway, 2009

  49. PADRES Broker P S B2 B Matching Engine B + Publications Routing Table B B1 output queue B2 input queue subscription dest B B B3 S P output queue B3 temperature > 37 B2 temperature > 40 B3 temperature = 36 temperature = 38 temperature = 42 = publisher = subscriber P S CANOE Summer School, Norway, 2009

  50. Broker Internals & Routing Tables PADRES Broker Input Queue Output Queues Matcher S1 B2 B4 B1 S1 SRT B3 S2 S1 s s S S2 S1 Pre Processor Post Processor [class,=,foo],[attr,<,4] [class,=,foo],[attr,>,30] S1 S2 PRT Forwarder Pub/Sub Messages [class,foo],[attr,24] [class,foo],[attr,11] Queue Handler Queue Handler the International CANOE Summer School, Norway, 2009 Advertisement Subscription Publication CANOE Summer School, Norway, 2009

More Related