Event-Driven Applications: Where they Apply and How they are Built

1. Event-Driven Applications: Where they Apply and How they are Built K. Mani Chandy California Institute of Technology mani@cs.caltech.edu Special thanks to Roy Schulte and David Luckham.

2. EDA Business Value Respond to events – threats and opportunities.

3. Take Away 1: EDA Characteristics • Monitors and correlates events outside and inside the enterprise. • No communication IMPLIES reality matches model.

4. Take Away 2: You’re ready for EDA now • Your company is already event driven. • You already have the components of the architecture in your enterprise stack, and it’s complementary to SOA. Key Question for you: Do the incremental benefits of IT for EDA exceed its incremental costs?

5. Organisms Sense and Respond: EDA • Streams of data: sight, sound, touch, smell, taste • Central nervous system detects the rare event that is a threat or an opportunity • Organism responds appropriately to threat or opportunity • Too many false positives, a false negative, or too many delayed or inappropriate responses results in death

6. Group Sense and Respond: EDA • Three enterprises: lions, hyenas, zebras • Critical conditions: fuse information from inside and outside the enterprise

7. External Awareness: Questions for you • Does your enterprise monitor its competitors? Government agencies? • Do people in your enterprise correlate information from multiple sources? e.g., correlate flood at a supplier’s factory with deadlines for critical customers.

8. Responding to Unexpected Situations Question for you: • A fire has just occurred in a factory that is going to effect customers severely. • Which of two scenarios represents your enterprise? • The CEO doesn’t expect VP Mfg to say anything unless the CEO asks. • The CEO expects VP Mfg to tell the CEO.

9. No Communication => Reality = Model • CEO has an expectation – a model – of the CFO: • No communication implies reality matches the model. • Space shuttle director has a model for the engineer responsible for shuttle foam insulation. • No communication implies reality matches the model.

10. Enterprise-Wide Situational Awareness Military situational awareness Corporate situational awareness London Edmonton Corporate VP, risk Denver NY, NY Scheduler cockpit Risk management cockpit Houston Risk manager Houston Sydney Trader cockpit

11. Take-Away Points • EDA: Global situational awareness • EDA: Respond when reality deviates from model. • Your enterprise is: • Already event-directed. • Has the software components including SOA • Question for you: incremental costs vs. benefits? 4:05 PM

12. What is EDA? • System that manages and executes rules of the form: WHEN reality deviates significantly from expectations THEN initiate appropriate response.

13. Respond Sense Detect events across extended environment in real-time Invoke distributed services in real-time Analyze Aggregate events across multiple sources EDA Characteristics

14. EDA: Software Characteristics • Asynchronous coupling. • The timing, place, and characteristics of threats and opportunities are not determined by you. • So sensors are responsible for pushing information; responders are not responsible for pulling it. • Defensive programming to the extreme. • Data may be unstructured and inaccurate. Protocols may be unspecified.

15. Defensive Programming to the Extreme Airline A Airline B Monitoring • One airline can make few assumptions about another airline. • EDA should be very robust; or it is very brittle • The robustness comes at a price • EDA is at the limit of coupling “looseness”

16. Defensive Programming to the Extreme Division A Division B Monitoring • One division can make few assumptions about another division. • EDA should be very robust • The robustness comes at a price • EDA is at the limit of coupling “looseness”

17. EDA Structure: Sense, Analyze, Respond

18. Defensive Programs: Sensors & Responders SENSORS PROGRAM OUTWARD-FACING COMPONENTS EXTREMELY DEFENSIVELY RESPONDERS

19. Less Defensive: Event Processing Network PROGRAM INWARD-FACING COMPONENTS LESS DEFENSIVELY

20. Compare EDA Requirements with SOA • SOA: Components are collaborators • Accounting “client” calls a “sales pipeline expectation” method on a sales “service” which returns with a report • SOA: Time of interaction determined by client • SOA: Service protocols and schemas are well defined. • SOA: Units obtain global situational awareness by invoking multiple services.

21. EDA: Software Characteristics - Review • Asynchronous coupling. • The timing, place, and characteristics of threats and opportunities are not determined by you. • So sensors are responsible for pushing information; responders are not responsible for pulling it. • Defensive programming to the extreme. • Data may be unstructured and inaccurate. Protocols may be unspecified. • Sense – Analyze – Respond. • When-Then rules.

22. Alert Engine End Users Approximate Matching Complex Event Patterns Web sites Web Service An Event-Driven Architecture Hostile Web sites Screen Scrape News feeds News Handler Stock tickers Ticker Handler E S B Scheduler Applications Application Handler Text Analysis Electronic Markets Market Interaction Parametric Analysis DB Database Interaction Time series Analysis When-Then Rule Mgmt System Configuration Monitoring

23. Take-Away 2 (repeated) • Your enterprise already has most of the components of the architecture: • EAI tools; Messaging, ESB or event distributors; Databases; Alerts engines; Web Services; Rules engines • EDA is compatible with and complements SOA. • Your enterprise is already event-directed. Your key question: Do the incremental benefits exceed the incremental costs? 4:15 PM

24. Inevitable Down-Sides of EDA • Effort required to specify and tune rules. • Errors: • False positives: Response to non-event Drowning in Events: “turn that darn thing off!” • False negatives: Non-response to event Why are these down-sides inevitable?

25. Inevitable EDA Down-Sides: Errors • When-then rules are imperfectly specified. • When clauses evaluated incorrectly or late. • Changing rules to reduce false positives increases false negatives (and vice-versa).

26. Inevitable EDA Down-Sides: Drown in Events • Most responses alert people. • Perception: false positives cost much less than false negatives. • Too many false positives: “so, turn that thing off.” Correct evaluation should be based on total costs of all the false positives and negatives over time.

27. Inevitable Down-Sides of EDA: No Rules • People are busy. • Learning tools for specifying rules takes time. • So, rules aren’t specified, or are specified in a cursory fashion. • Destroys business value of EDA.

28. Rule Development Solutions • Have IT (or some central org.) specify rules. • Doesn’t work. • Work with business users to specify rule templates; individuals fill in templates. • Have role-based rule repositories.

29. Is Anything Missing in your Software Stack? • Event Process Agents • Machine can “learn” the critical condition from positive and negative examples • Users can specify critical conditions • SQL-like queries • Fuzzy matches • Statistical operators • Regular expressions • CEP

30. Estimating Performance Requirements • Delay from occurrence of condition to initiation of response: Minutes? Sub-seconds? • Number of data sources: Tens, Hundreds? • Numbers of rule templates: Tens, Hundreds? • Numbers of users? • Numbers of rules? Observation: Many enterprises overshoot: they estimate greater performance requirements than they need.

31. Yes Use Your Existing Software Stack If • Delay from occurrence of condition to initiation of response: Tens of seconds • Number of data sources: Ten, • Numbers of rule templates: Ten, • Numbers of users? 100s • Numbers of rules? 1000s This is the more common case.

32. Speed of EDA uptake • Enterprises are event-driven. So, why is EDA uptake slow? • Simple event processing is widely used. • Enterprise service buses and EAI tools allow when clauses on single documents (events). • Multiple event streams are correlated in some applications But these apps are not seen as part of an EDA paradigm.

33. Speed of EDA uptake • Correlation across multiple event streams in: • Financial trading; IT Infrastructure management; Plant control; Defense • Why these spaces? • Small designated groups responsible for responding to critical events. • Clear additional value • Performance

34. Should you build EDA? • Your company is already event driven; you have many of the software components. Question: Do incremental benefits exceed incremental costs?

35. Should you build EDA? Do you have: • Applications: • Apps that sense and respond to the environment and that benefit from automation? • Evolving middleware that can benefit from asynchronous coupling? • Responsibility in a single group or LOB? • Performance requirements not met? • Small numbers of data sources and rule templates satisfying large numbers of users?

36. Should you build EDA? • Are costs of false positives and negatives smaller than benefits of EDA? • Do you have a senior manager in a LOB who will make all the hard business work happen? (Building effective EDA is 95% effort by business people and only 5% effort in technology.)

37. Future Talks • How not to build EDA. Lessons from the trenches! • Steps and gotchas in building EDA.

38. Take-Away Points and THANKS! • EDA: Global situational awareness • EDA: Unit “knows” reality matches model until the unit gets a message. • Your enterprise: • Is already event-directed. • Has the software components including SOA • Your Question: incremental costs vs. benefits? 4:45 PM

39. Getting Business Users to Specify Rules is Hard • Different roles have different sets of rules. Who specifies the rules? • How many rule templates? Tens, hundreds? • How many rules? Hundreds, thousands? • How are rules specified? Language, visual UI, positive and negative examples? Fill in templates? • Who can turn a rule off? • What if rules are at cross-purposes?