1 / 31

Fjording The Stream An Architecture for Queries over Streaming Sensor Data

Fjording The Stream An Architecture for Queries over Streaming Sensor Data. Samuel Madden, Michael Franklin UC Berkeley. Introduction. Telegraph Sensor Query Processing Architecture Fjords Enable push and pull in query plans Operators for streaming data Sensor proxy

judah
Download Presentation

Fjording The Stream An Architecture for Queries over Streaming Sensor Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fjording The StreamAn Architecture for Queries over Streaming Sensor Data Samuel Madden, Michael Franklin UC Berkeley

  2. Introduction • Telegraph Sensor Query Processing Architecture • Fjords • Enable push and pull in query plans • Operators for streaming data • Sensor proxy • Sensor - Query Mediator User Proxy Web Sensors Fjording ICDE

  3. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  4. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  5. Sensor Challenges • Battery Powered • 2AA Cells (2800 mAh; 1.01x104 J), Coin Cell (100mAh) • Communication Dominates Power Cost • Can be exhausted - tens to hundreds of MBs of data / sensor • Wireless • High loss rates (20% @ 5meters , typical) • Low bandwith (10kbps) • Near Real Time • Streaming Data • Pushed at (user defined) regular intervals TinyOS “Mote” Fjording ICDE

  6. Requirements of Sensor Query Processing • Power Sensitivity • Tolerance to unbounded streams of data • Tolerance to push • Tolerance to intermittent, lossy connections • Tolerance to failed sensors Fjording ICDE

  7. Traffic Scenario • CA Department of Transportation (CalTrans) has sensors all over bay area freeways • Inductive loop sensors give speed, flow, vehicle size • Motes could collect this data cheaply • Many possible queries • Commuters want to find congestion • California Highway Patrol (CHP) wants to find accidents Fjording ICDE

  8. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  9. Fjords • Query plan implementation • Useful for streams and distributed environments • Combine push (streaming) data and pull (static) data • E.g. traffic sensors with CHP accident reports Fjording ICDE

  10. Push vs. Pull • Problem: Need an API to combine push & pull • Operators (e.g. join) data-direction agnostic • Push vs. pull implemented in queues (connectors) • Contrast with • Iterator model • Exchange operator • Allows arbitrary combinations of push and pull Fjording ICDE

  11. get() process() Pull Example Pull Queue Operator parent, child; Tuple get() { Tuple t = null; while (t == null) { t = child.process(); } return t; } Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; If (t != null) { <process t> } else { … do something else … } return outt; } s • Notice: • Iterator semantics by making get() blocking • Get() can return null • Process() can return null Pull Connection Scan Fjording ICDE

  12. get() Push Example Push Queue Operator parent, child; Vector v = new Vector(); Tuple get() { if (v.size() > 0) return v.removeFirst(); else return null; } Tuple enqueue(Tuple t) { v.put(t); } Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; If (t != null) { <process t> } else { … do something else … } return outt; } Thread while(true) { Tuple t = op.process(); if (t != null) op.outq.enqueue(t); } s Push Connection Scan Fjording ICDE

  13. Fjord Example Push Push Pull Fjording ICDE

  14. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  15. Continuous Queries • Given user queries over current sensor data • Expect that many queries will be over the same data sources (e.g. traffic sensors) • Queries over current data always looking at same tuples • Those queries can share • Current tuples • Work (e.g. selections) • Sharing reduces messages, thereby power! • A new query can be “folded” into an existing query • Use old instances of scans and selections Fjording ICDE

  16. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  17. Relational Operators And Streams • Selection and Projection Apply Naturally • No Blocking Operators • Sorts and aggregates over the entire stream • Nested loops and sort-merge join • Windowed Operators • Sorts, aggregates, etc. • Online, Interactive QP Techniques • In memory symmetric hash join • Alternatives: ripple-join, Xjoin, etc. • Partial Results Fjording ICDE

  18. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  19. Query Registration Parsed Queries [sources, ops] [fields, filters, aggregates, rates] Query [tuples] Operators Sensor Proxies • Mediate between Sensors and Fjords • Push operators out to sensors • Hide query processing, knowledge of multiple queries from sensors • Hide details of sensors from query processor • Enable power-sensitivity Query Processor Fjording ICDE

  20. What runs where? • (Multi-query) Distributed Optimization Challenge: • Given set of operators, proxy must choose: • Run on sensor or, • Run on local query processor • Running on sensors saves power • Simple computations cheaper than messages • Selection, aggregation reduce communication cost • Sensors have limited resources • All queries can’t run in all sensors simultaneously • Limited state precludes big joins or lots of groups • Queries share operators • Operators vary in selectivity Fjording ICDE

  21. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  22. accident = true Index NL Join Building a Query • How to translate a declarative query into a Fjord? • Just like traditional query processing, except: • Branches originating in sensors connected by push connectors • Sensor proxy handles scans, selections over sensors • Proxy delivers tuples from sensors • Proxy pushes-down selections transparently • Output of join is push if one or both inputs is push • Join carefully chosen Pull: Data Request Data Request Speed < 30 Push Push Data Fjording ICDE

  23. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  24. Loc  {userLocs} Query & Fjord • Simple Test Query: SELECT AVG(s.speed, w) FROM sensors AS s WHERE s.loc in {userLocs} Telegraph Server Average BHL Server Fjording ICDE

  25. Loc  {userLocs} Loc  {userLocs} Loc  {userLocs} Multiple Queries in a Fjord Telegraph Server BHL Server Fjording ICDE

  26. Fjord Performance (In Telegraph) Fjording ICDE

  27. w t : Vehicle Covers S t : Vehicle Covers S S and 0 u 1 u d w w t : Vehicle Covers S t : Vehicle Covers Neither Sensor 2 d 3 Sensor Proxy for Traffic • Measure benefit of pushing computation into sensors; in this case, vehicle identification. • Simple “aggregation” dramatically reduces power costs Fjording ICDE

  28. Pushing Aggregates Saves Power Atmel (TinyOS CPU) Simulator 100 samples / sec 5 vehicles / sec 7x power savings Fjording ICDE

  29. Roadmap • Background • Sensors • Requirements • Traffic Scenario • Fjords • Continuous Queries • Stream-sensitive operators • Sensor Proxies • Querying a Sensor Network • Results • Graphs • Related Work Fjording ICDE

  30. Related Work • Cougar • Interactive & Adaptive Query Processors • Tukwila, Xjoin, Eddy, CONTROL • Continuous Query Processors • NiagaraCQ, Xfilter, CACQ • Directed Diffusion • Volcano / Exchange Operator • Temporal Databases Fjording ICDE

  31. Conclusions & Future Work • Required for sensor query processing: • Fjords : API for combining push & pull • Sensor Proxy: Mediator between sensors and QP • Streaming Operators • Big benefit from pushing selections, aggregates into network • Combining multiple queries is a win • Extensions & Future Work: • Multi-query optimization & adaptivity (CACQ, SIGMOD 2002) • Push down selections & aggregates (Submitted to VLDB 2002) • Sensor proxy policies Fjording ICDE

More Related