Event Processing in Operational Information Systems: Two Case Studies and BAM/EDA Implications

1. Event Processing in Operational Information Systems:Two Case Studies and BAM/EDA Implications Karsten Schwan, Brian Cooper, Greg Eisenhauer Georgia Institute of Technology Center for Experimental Research in Computer Systems (CERCS) NSF Industry University Co-operative Research Center

2. I. Delta Air Lines Operational Information Systems (OIS) – Internal View High event rates for simple/mediated events Complex events processed/produced by business logic

3. I. Delta Air Lines OIS – External View

4. I. Continuous Event Processing in Delta’s OIS • Complex systems and large event volumes • TPF, DTMI, TIBCO, Tuxedo, Web Services; Mainframes, Clusters, End Systems • event services across multiple system `silos’ • interoperability APIs • event filtering, replication, morphing • JIT XML and event conversion – for outsources services • runtime trust management vs. security? • data tapping – for legacy systems (hardware support?) • deep packet inspection/event morphing (system/network support?)

5. I. Integrated BAM: Continuously Managed Event Flows • Complex system interactions and 24/7 operation: • high reliability and availability: with stateful operation • continuous monitoring and repair • abnormal behavior (e.g., timeout behavior) detection, with human intervention after thresholds exceeded • `poison messages’ and poison message sequences • avoid recovery and/or bound recovery time • online performance management • utility-based event scheduling/routing • ability to distinguish service levels • link to immediate business needs • e.g., revenue management • performance isolation vs. optimization • e.g., isolation from recovery traffic • NOTES: highly distributed event processing; most events carry business data (additional BAM events); BASE, not ACID, for most events; multi-model event processing, not SQL; STATEful processing

6. II. Worldspan: Need for QoS in Business Monitoring Utility Obtained from Worldspan’s Flight Search Engine • SLA-driven operation and online event scheduling: • QoS in Business Monitoring for differentiated services • 24/7 operation and stateful services: • Management must include incremental updates of service state • Huge event volumes

7. Summary Event-based Systems for the Enterprise Domain: • GT Focus: Adaptive/Autonomic Distributed Information Flows • IBM, Tata (iFlow: utility-based, autonomic management of distributed information flows; performance isolation in web-based event flows; online monitoring and management with Eclipse) • HP (automated application deployment; QMon: QoS in business activity monitoring) • Worldspan (`power udpates’: non-intrusive dynamic state updates; utility-based activity monitoring) • Delta, Raytheon (performance isolation/robustness; utility-driven failure management; monitoring web-based infrastructures) • Cisco, Intel (network-level services for event-based systems) • NSF, DARPA, DOE (continual queries; ECho/IQ-ECho:publish/subscribe event system, with resource-aware operation; EV(ent)Path: dynamic overlay creation and management, with runtime event scheduing; event flows and mobility) Security Systems

8. EDA/BAM Implications • Multiple event/processing models • Monitoring events, Business events, ... • Interoperability • Differently structured event data, eventually should include unstructured data • Complex, domain-specific event processing • Importance of state • state recovery/expiration • Distributed data and processing • Security/performance/reliability implications • Importance of online management • integrated into business event processing • driven by end user utility • strong QoS/real-time constraints • Overlap/conflicts with AC (ICAC) (many companies involved!) • Terminology: • CBEs (events), touchpoints, symptoms/symptom databases, SLAs, SLOs, ... • Technology: • non-intrusive instrumentation, ...

9. Georgia Tech Information Flow Research To construct the interactive information grids of the future and to create the intellectual capital that can advance these technologies and fuel future advances. Enterprise Computing Embedded Systems Scientific.Grid Remote access to the Information Grid Continual Queries ECho/IQ-ECho Fusion Channels IFlow/EVPath Information anytime, anywhere  Timeliness! Robustness! Quality! Security and Trust! Brian Cooper Ling Liu Calton Pu Kishore Ramachandran Karsten Schwan

10. Additional Insights Enterprise Systems • Utility-based mapping and configuration in: • shared execution environments High Performance Computing • Large-data events in: • simulation monitoring: e.g., remote data visualization • GT Smartpointer application Pervasive Systems • Online path management in: • situation monitoring and assessment • Location-aware operation in: • mobile end user systems

11. Research Agenda for Event-based Systems • I. Stateful Event Services: • Dynamic service and code deployment (DCG, dynamic compilation) • Runtime code modification and adaptation, dynamic data conversion • Dynamic state saving and updates (e.g., power updates) • Dynamic overlays, … • II. Resource- and Needs-Awareness: • Diverse metrics: bandwidth, power, trust, ... • Changing end user needs, application behaviors • Performance monitoring/understanding: integrate across user and system levels • III. Runtime Management: • Utility-driven operation • New reliability and availability methods • `Vertical’ integration: user/system/network levels • Multi-dimensional optimization vs. performance robustness • IV. Open Infrastructures: • App-level (e.g., `inside’ JMS) or `instrumented networks’ • `Black box’ operating systems vs. dynamic extension and VM technologies • `Closed’ networks vs. application-level services `in’ network devices • e.g., Cisco’s AONS, Intel’s IXP network processors

12. Event Processing in EScience – SmartPointer Example SmartPointer: Data-intensive scientific collaboration Dynamic composition of user-specified services.