1 / 16

Telegraph: An Adaptive Global-Scale Query Engine

Explore the adaptive technology of The Telegraph Query Engine, enabling interactive control, online aggregation, and continuous flow optimization for handling massive data queries across diverse sensors, devices, and networks globally.

mccaw
Download Presentation

Telegraph: An Adaptive Global-Scale Query Engine

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Telegraph: An Adaptive Global-Scale Query Engine Joe Hellerstein

  2. Scenarios • Ubiquitous computing: more than clients! • sensors and their data feeds are key • smart dust, biomedical (MEMS sensors) • each consumer good records (mis)use • disposable computing • video from surveillance cameras, broadcasts, etc. • Global Data Federation • all the data is online – what are we waiting for? • The plumbing is coming • XML/HTTP, etc. give LCD communication • but how do you query robustly over many sites in the wide area?

  3. There’s a Data Flood Coming

  4. There’s a Data Flood Coming • What does it look like? • Never ends: interactivity required • Big: data reduction/aggregation is key • Unpredictable: this scale of devices and nets will not behave nicely

  5. The Telegraph Query Engine • Key technologies • Interactive Control • interactivity with early answers • online aggregation for data reduction • Continuously adaptive flow optimization • massively parallel, adaptive dataflow via Rivers and Eddies

  6. CONTROLContinuous Output, Navigation & Transformation with Refinement On Line • Data-intensive jobs are long-running. How to give early answers and interactivity? • online interactivity over feeds: data “juggle” • online query processing algs: ripple joins • statistical estimators, and their performance implications • Appreciate interplay of massive data processing, stats, and UIs

  7. CONTROLContinuous Output and Navigation Technology with Refinement On Line

  8. CONTROLContinuous Output and Navigation Technology with Refinement On Line

  9. Q River • We built the world’s fastest sorting machine • On the “NOW”: 100 Sun workstations + SAN • But it only beat the record under ideal conditions! • River: performance adaptivity for data flows on clusters • simplifies management and programming • perfect for sensor-based streams

  10. Eddy Eddy • How to order and reorder operators over time • based on performance, economic/admin feedback • Vs.River: • River optimizes each operator “horizontally” • Eddies optimize a pipeline “vertically”

  11. Telegraph: Putting it Together • Scalable, adaptive dataflow infrastructure. Apps include… • sensor nets • massively parallel and wide-area query engines • net appliances: chaining xform8n/aggreg8n/etc. proxies • any unpredictable dataflow scenario • Technology: a marriage of… • CONTROL, River & Eddy • Many research questions here • E.g. how to combine River and Eddy adaptivity • E.g. how to tune Eddies for statistical performance goals • Combinations of browse/query/mine at UI • Storage management to handle new hardware realities

  12. Integration with Endeavour • Give • Be data-intensive backbone to diverse clients • Be replication dataflow engine for OceanStore • Telegraph Storage Manager provides storage (xactional/otherwise) for OceanStore • Provide platform for data-intensive “tacit info mining” • Take • Leverage OceanStore to manager distributed metadata, security • Leverage protocols out of TinyOS for sensors

  13. Additional Slides • For use in questions, etc.

  14. Connectivity & Heterogeneity • Lots of folks working on data format translation, parsing • we will borrow, not build • currently using JDBC & Cohera Net Query • commercial tool, donated by Cohera Corp. • gateways XML/HTML (via http) to ODBC/JDBC • we may write “Teletalk” gateways from sensors • Heterogeneity • never a simple problem • Control project developed interactive, online data transformation tool: Potter’s Wheel

  15. Potter’s Wheel Anomaly Detection

More Related