1 / 19

Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing

DEBS Grand Challenge 2012. Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing. Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey , Aakash Nigam, Chen Wang, Hans-Arno Jacobsen Middleware Systems Research Group, University of Toronto.

cai
Download Presentation

Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DEBS Grand Challenge 2012 Solving Manufacturing EquipmentMonitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, NavneetKumar Pandey, Aakash Nigam, Chen Wang, Hans-Arno Jacobsen Middleware Systems Research Group, University of Toronto

  2. Agenda • Complex Event Processing Scenarios • System Architecture • Evaluation • Demo msrg.org

  3. Motivation • Course “Large-Scale Data Management” • Big data storage • Large scale event processing • Complex event processing scenarios • Application performance management • Smart traffic monitoring • Energy monitoring • Resulting team project • DEBS Grand Challenge msrg.org

  4. Scenario I: Application Performance Management • Monitoring of enterprise systems • Find bottlenecks, problems • Trace transactions, measure utilization msrg.org

  5. Scenario II: Smart Traffic Monitoring • Traffic data from • cars, mobile devices, road sensors • Event aggregation, filtering, correlation • Traffic status, accident detection, etc. msrg.org

  6. Scenario III: Energy Monitoring • Green computing • Application-level energy monitoring • API-based energy consumption estimation Applications API Formulae CPU Network I/O … Store Sensors Operating System & Hardware msrg.org

  7. Common Denominators • High data rates • 1000 – 1000000+ events / sec • Small data points • < 1 KB • Complex queries • Filtering, aggregation, correlation • Persistent storage • Distributed setup • DEBS Grand Challenge msrg.org

  8. High-Level Architecture • Monitoring Service • Input data stream, marshalling • Event Dissemination Substrate • (Optional) pub/sub layer, queues • Continuous Query Evaluation • Consumes input, computes results • Storage Manager • Stores data, enables querying • Client • Visualizes query results • Java-based implementation Monitoring Service Continuous Query Evaluation EventDisseminationSubstrate Client Storage Manager Stable Storage msrg.org

  9. Storage Architecture • Data is stored in tables • Key-value pairs • Table Index • Fast lookup • Compressor • Efficient data storage • Run length encoding • Grand Challenge data • Run-length: 20000 • Compression: 99.99% msrg.org

  10. Client • Google Web Toolkit-based • Client Java-code compiled to JavaScript • Displays all Grand Challenge results • Plots, results, alarms msrg.org

  11. Evaluation • Distributed setup • Separated servers for data generator and monitoring tool • Configuration • 2 servers • 2 x dual core Xeon processor • 4 GB RAM • Gigabit Ethernet • Data set: 5 min + 18 days + synthetic • Synthetic data • Many errors (every 2 - 3 min) • Maximum throughput (no network) • Metrics: latency & throughput msrg.org

  12. Processing Overhead Real Workload • 0 – 200 queries (Q1,Q2 repeated) • Linear in the number of queries • Stable with increasing generator speedup msrg.org

  13. ThroughputReal Workload • Throughput controlled by data generator • Data generator does not saturate system • Peak 9000 events/sec ~ 0.11 ms arrival rate • Well below maximum throughput msrg.org

  14. LatencySynthetic Workload • Maximum throughput • Minimum latency • 0.019 ms (10 queries) • Maximum latency • 0.63 ms (1400 queries) msrg.org

  15. Throughput Synthetic Workload • Maximum throughput • 40000 events / sec (20 queries) • Minimum throughput • 1500 events / sec (1400 queries) msrg.org

  16. Demo • Live! msrg.org

  17. Conclusions • Complex event processing scenarios • Application performance management • Smart traffic monitoring • Energy monitoring • DEBS Grand Challenge • Efficient implementation of the Grand Challenge • Java-based • Google Web Toolkit GUI • Synthetic data generator • Up to 40000 events per second • Up to 1400 queries msrg.org

  18. Thank You! • Questions? Contact: Tilmann Rabl tilmann.rabl@utoronto.ca msrg.org msrg.org

  19. Back-Up: DEBS Grand Challenge • Manufacturing equipment monitoring • Large, real data set • 18 days, 100 Hz • 2 queries • State of sensors and valves • Difference • Threshold • Power consumption • Range • Average • Threshold msrg.org

More Related