<Insert Picture Here>

1. <Insert Picture Here> Event Processing Using Database Technology Dieter Gawlick and Shailendra Mishra Oracle, Mani Chandy, California Institute of Technology Sigmod Tutorial - Event Processing Using Database Technology

2. Agenda • Overview of event driven applications (Mani Chandy) • 5 parts • 10:30 to 11: 40 • Event processing using database technology (Shailendra Mishra) • 11:30 to 12:00 • 1:30 to 2:45 • Summary; next steps for you and challenges • 2:45 to 3:00 Sigmod Tutorial - Event Processing Using Database Technology

3. Part 1 • Benefits of event-driven applications • What is an event? • Components of event-driven systems • Most human organizations have been event-driven since the dawn of civilization. Why study event-driven IT now? What changed? Sigmod Tutorial - Event Processing Using Database Technology

4. Benefits of Event Driven Systems • Proactive response: respond to changes before they unfold. • Timely accurate response to changes. Sigmod Tutorial - Event Processing Using Database Technology

5. What is an event? • Event: a significant state change that requires a response. • The response may merely be registering the state change in memory. An insignificant state change is not “remembered.” • An event is a state change that often signals an opportunity or a threat • Event driven systems respond to events timely, accurately and (where possible) proactively Sigmod Tutorial - Event Processing Using Database Technology

6. What is an event? • Event: a significant state change that requires a response. • Change in time is a change in state. • So absence of change in a parameter over a time can be an event. • Example: No new deliveries after item was ordered with overnight shipping. Sigmod Tutorial - Event Processing Using Database Technology

7. External Event Actuator Information Fusion Actuation Information acquisition Components: Event Driven Systems Sigmod Tutorial - Event Processing Using Database Technology

8. Components: Event Driven Systems • Event sources • sensors; stock tickers; news stories • Dissemination network • Subscriptions; role registration • Dissemination protocols: push, pull, schedule • Processing agents • Databases; BI tools; rules engines • Responders • Event consumers: people, processes, actuators • System management • Registration of components, self-monitoring, “ilities” Sigmod Tutorial - Event Processing Using Database Technology

9. Most Systems are Event Driven • Nature • Single animal • Groups of animals • Human organizations • Why is their a need for event-driven systems now? What changed? Sigmod Tutorial - Event Processing Using Database Technology

10. Animals Are Event Driven • Streams of data: sight, sound, touch, smell, taste • Central nervous system detects the rare event that is a threat or an opportunity • Organism responds appropriately to threat or opportunity • Too many false positives, a false negative, or too many delayed or inappropriate responses results in death Sigmod Tutorial - Event Processing Using Database Technology

11. Animal Groups are Event Driven • Three enterprises: lions, hyenas, zebras • fuse information from inside and outside the enterprise Sigmod Tutorial - Event Processing Using Database Technology

12. Trade Finance Money Markets Secured Lines Foreign Exchange Problem:Elapsed time to ascertain limit breach of total credit exposure to single customer or industry segment is 90+ daysResult:Asset impairment leading to increased loss reserves Risk Management Sigmod Tutorial - Event Processing Using Database Technology

13. What changed? Pressure to: • Respond even more proactively and rapidly • React to conditions outside the organization • Handle huge data volumes and creation rates Opportunity • Availability of inexpensive computing elements • Ubiquity of sensors, wireless; miniaturization • Software and design technologies Sigmod Tutorial - Event Processing Using Database Technology

14. Part 1 finished • Benefits of event-driven applications • What is an event? • Components of event-driven systems • Most human organizations have been event-driven since the dawn of civilization. Why study event-driven IT now? What changed? Sigmod Tutorial - Event Processing Using Database Technology

15. Part 2 • Monitoring state-changes outside the enterprise. • Consequences of external monitoring. • Technology trends • THE SCARCE RESOURCE • Impact of the scarce resource on the future 10;50 AM Sigmod Tutorial - Event Processing Using Database Technology

16. Agents outside the enterprise • Customers: relationship management; trends • Suppliers: pricing; quality; delays • Competitors’ behaviors • Government regulators • External trends: delays; weather; accidents Sigmod Tutorial - Event Processing Using Database Technology

17. Web Page Extracting Engine MS Word, Excel MS Exchange Information Acquisition Software agent pulls and then normalizes information Human inputs data in HTML forms XML Repository & Matching Engine Monitor chat rooms and bulletin boards Seamless integration w/legacy information Sigmod Tutorial - Event Processing Using Database Technology

18. An Energy Trading Application Multiple concurrent threads insert into environment interface tables Weather Pipelines Power prices Electricity grid utilization DB Event Detector Event handling Put into MQ Event or message Sigmod Tutorial - Event Processing Using Database Technology

19. External Agents for Individuals Entertainment and information: • You Tube • Face Book • Second Life • wikipedia • eBay • Amazon • Flickr • Google • Last.fm • myspace • msn games; yahoo games; … Sigmod Tutorial - Event Processing Using Database Technology

20. Monitoring External Agents: Consequences • Asynchronous data: external agencies – not you – determines when data becomes available. • Noisy data: Schemas may not exist and may change at any time. • Volume of external relevant data is growing rapidly. Sigmod Tutorial - Event Processing Using Database Technology

21. Monitoring External Agents: Consequences Responses may be wrong because: • external data is inaccurate. • the appropriate external data wasn’t monitored • Changes in external data weren’t detected in a timely way Sigmod Tutorial - Event Processing Using Database Technology

22. Technology Trends • Costs of computers, storage, communication dropping rapidly, exponentially and continuously. • Sensors becoming much more widely available. • Responder mechanisms widely available: whom to communicate with, how, when • Miniaturization • Location sensing; geographical data Sigmod Tutorial - Event Processing Using Database Technology

23. What are the scarce resources? Given rapid exponential decrease in costs of: • Storage • Bandwidth • Computing capacity • Energy requirements Given pressure to: • Respond even more proactively and rapidly • React to conditions outside the organization: • Handle huge data volumes and creation rates Sigmod Tutorial - Event Processing Using Database Technology

24. What are the scarce resources? • Time and attention Sigmod Tutorial - Event Processing Using Database Technology

25. Managing Scarce Resources Right information to right people at right times Sigmod Tutorial - Event Processing Using Database Technology

26. Future impact of the scarce resource • The biggest software companies 10 years from now will not think of themselves primarily as vendors of: • Databases • Application servers • Operating systems • The will be companies that help you manage your time and attention. Sigmod Tutorial - Event Processing Using Database Technology

27. Scare Resources and Technology Time and Attention: EDA Integration and Services: The Web Scarce resource Communication:The Internet Technology Computation: Mainframe; then PC Time Sigmod Tutorial - Event Processing Using Database Technology

28. Monitoring Externals Interaction outside and inside the organization Interaction within the organization Event-Driven Architecture Service Oriented Architecture Remote procedure calls Client-Server Procedure calls Time Sigmod Tutorial - Event Processing Using Database Technology

29. Isn’t the Enterprise Stack Enough? What’s needed Focus of the current enterprise stack Time and Attention: EDA Integration and Services: The Web Communication:The Internet Computation: Mainframe; then PC Time Sigmod Tutorial - Event Processing Using Database Technology

30. Part 2 finished • Monitoring state-changes outside the enterprise. • Consequences of external monitoring. • Technology trends • THE SCARCE RESOURCE • Impact of the scarce resource on the future Sigmod Tutorial - Event Processing Using Database Technology

31. Part 3 Types of communication • Schedule-driven; pull; push • Advantages and disadvantages of each type • Every organization uses all three types • Lead in to relationship between EDA (Event Driven Architectures) and SOA (Service Oriented Architectures) 11:05 AM Sigmod Tutorial - Event Processing Using Database Technology

32. Types of Communication • Communication types: Schedule, pull and push. • Most organizations have all 3 types of communication. • IT has been slow to support push. Sigmod Tutorial - Event Processing Using Database Technology

33. Types of Communication • CEO and VP meet Monday at 9AM – Schedule Driven; timed • CEO calls VP to check on status of manufacturing – SOA; pull • VP manufacturing calls CEO when there is a fire in a factory. – EDA; push Sigmod Tutorial - Event Processing Using Database Technology

34. Schedule-Driven Communication: Advantages • Checks “health” of agents. • if an agent doesn’t participate then that agent is probably not functional. • Energy spent when needed, i.e., when “woken” up by schedule; • important for certain types of wireless devices. • Effective interaction with a group of agents scheduled to meet at the same time Sigmod Tutorial - Event Processing Using Database Technology

35. Schedule-Driven CommunicationDisadvantages • CEO and VP have group meetings every Monday morning. • There is a fire in a factory on Monday evening. Does the CEO hear about the fire only next Monday morning? Sigmod Tutorial - Event Processing Using Database Technology

36. Push Advantage • Service monitors “reality” – data sources – and proactively informs consumer when necessary. Disadvantage • Some messages may be irrelevant (false positives) and some conditions may be missed (false negatives) Sigmod Tutorial - Event Processing Using Database Technology

37. Pull Advantage: • Requestor often has a good idea of information that the requestor needs. • e.g., Doctor asks nurse for specific information about blood sugar Disadvantage • Requestor doesn’t know when to pull the information. Sigmod Tutorial - Event Processing Using Database Technology

38. Ideal: Combination of Communication Types Example of a hospital: • Push: From sensors to nurses to residents to specialist doctors • Pull: Doctors ask for specific information from specific people and databases to help with determining the doctor’s actions • Schedule: Every morning specialists, residents, nurses meet to discuss cases Sigmod Tutorial - Event Processing Using Database Technology

39. Database: Interaction Types • Push: • Trigger; continuous query • Pull: Typical database operations • Query, Updates • Schedule: • Time-driven queries and updates Sigmod Tutorial - Event Processing Using Database Technology

40. Key Points • Most systems benefit from a combination of schedule, push and pull • what information is communicated with each type of operation, and when? • Human organizations use combinations of schedule, push and pull. Consider: • healthcare, supply chain, trading • IT has been slow to support integration of push with pull and schedule-driven interaction Sigmod Tutorial - Event Processing Using Database Technology

41. Part 3: Finished Types of communication • Schedule-driven; pull; push • Advantages and disadvantages of each type • Every organization uses all three types • Lead in to relationship between EDA (Event Driven Architectures) and SOA (Service Oriented Architectures) Sigmod Tutorial - Event Processing Using Database Technology

42. Part 4 • SOA (Service Oriented Architecture) and EDA (Event Driven Architecture) • Difference in emphasis between SOA and EDA • Trends in SOA and EDA • Composition and contracts in SOA and EDA • SOA and EDA are complementary 11:15 AM Sigmod Tutorial - Event Processing Using Database Technology

43. SOA and EDA • IT already supports SOA (Service Oriented Architecture) • What does IT have to do differently from SOA to support events? Sigmod Tutorial - Event Processing Using Database Technology

44. SOA and EDA: Difference in Emphasis • SOA: • client makes request in well-defined schema • Server responds in well-defined format • Clear relationship between call and response • Natural progression of request-reply / client-server form of module composition Sigmod Tutorial - Event Processing Using Database Technology

45. SOA and EDA: Difference in Emphasis • EDA: • Data from sources outside and inside the enterprise • Data arrives asynchronously • Respond proactively • Workflow, business process management, agile supply chain management, timely business intelligence are all event driven. • EDA: Feedback loop – analyzing flow of events helps to optimize event processing. Sigmod Tutorial - Event Processing Using Database Technology

46. Difference in Emphasis Users request specific information from specific services Users are sent relevant information based on their interests, roles, subscriptions Time and Attention: EDA Integration and Services: The Web Communication:The Internet Computation: Mainframe; then PC Time Sigmod Tutorial - Event Processing Using Database Technology

47. Event Programming Breakthroughs Traditional Request – Reply Trends: EDA and SOA Event-Driven Architecture Events: Loose coupling and BAM Service Oriented Architecture • Event-Mainstream Apps: • C4I • Program Trading • Process Control Process • Event Meta-Apps: • Fraud • Management • Security • Alerts • BPM • CRM • SCM Client/Server Transactions Procedural/ Legacy time 1965 now Sigmod Tutorial - Event Processing Using Database Technology

48. SOA: Sequences Call service to compute minimum cost in dollars for ball bearings to be delivered to LA. Service calls other services to obtain costs in local currencies. Computes best dollar price and sends reply EDA: Monitors, integrates Alert manager if potential delay in shipment to a customer can result in penalties. Continuously monitor Suppliers Manufacturing process New contracts & deadlines Integrate data from multiple sources; send alert Composition in SOA and EDA Sigmod Tutorial - Event Processing Using Database Technology

49. EDA Contracts between Components • What is the contract between producers and consumers of information in EDA? • Producer monitors and estimates the state of some aspect of the world. • Contract: When there is a state change that is significant for the consumer, the producer alerts the consumer. • How is this contract implemented? Sigmod Tutorial - Event Processing Using Database Technology

50. Implementation of EDA Contracts • Producer and consumer share a model of what state changes are significant for the consumer. • Two types of shared models: • Deviation: Send alert when reality deviates from model • Match: Send alert when reality matches model • Conceptual mismatch when models of producer and consumer are not identical. • Conceptual drift may require new shared model. Sigmod Tutorial - Event Processing Using Database Technology