1 / 11

Application of Event Stream Processing toward QA/QC in GLEON

Application of Event Stream Processing toward QA/QC in GLEON. Sameer Tilak, Chris Solomon, Peter Shin, Tony Fountain, and Peter Arzberger. Science: Ability to process and manage large amounts of data in a scalable and efficient fashion

weylin
Download Presentation

Application of Event Stream Processing toward QA/QC in GLEON

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Application of Event Stream Processing toward QA/QC in GLEON Sameer Tilak, Chris Solomon, Peter Shin, Tony Fountain, and Peter Arzberger.

  2. Science: Ability to process and manage large amounts of data in a scalable and efficient fashion • Engineering/Operations and Management: Reliable and smooth operation of the system Motivation

  3. Non Real-Time (Traditional) QA/QC Approach • Set up an experiment, collect data in flat files/ database. • Look at acquired data after few days/weeks/months. • QA/QC is done either: • Manually: Visual inspection • Automated: Matlab/Perl scripts etc.

  4. Sample QA/QC Procedure • Out-of-range values • Missing data • Sensor flat lines • Signal shifts etc.

  5. Issues with current approach • Real World is Messy. QA/QC might take a lot of time! Even up to 50% • System Level: Clock drifts, batteries fail, sensors malfunction, servers and networks fail. You can’t respond to these in real-time • Application/Science Level: Rear event occurs, abnormal event happens you can’t respond to it in real-time

  6. Event Stream Processing (ESP) Overview • ESP and CEP have emerged out of years of research in the field of active database, continuous queries and stream processing and are now being widely used in modern latency-sensitive architectures such as algorithmic trading systems, fraud or intrusion detection and real-time business intelligence or customer relationship management.

  7. Real-Time Event Stream Processing (ESP) • Database up-side down • Persistent Queries versus single-shot queries • Seamless integration of persistent data with streaming data • Time-windows are user-defined can span anywhere from few milliseconds to hours or days

  8. Proposed Architecture

  9. Advantages of the Proposed Architecture • Real-Time notification -- on the order of few seconds/minutes • Scalable and modular architecture • Enables new types of science • Novel system monitoring capabilities • Essential QA/QC happens in more efficient manner e.g. pre-processing.

  10. Tsunami Data

  11. Future Work • Work more with domain scientists to fine-tune QA/QC procedures. • Employ Cloud Computing. • Live-Migration of virtual machines in real-time in response to critical events to provide more compute power, network bandwidth etc.

More Related