Streaming Queries over Streaming Data

1. Streaming Queries over Streaming Data Sirish Chandrasekaran (UC Berkeley) Michael J. Franklin (UC Berkeley) Presented by Andy Williamson

2. About Me • 3rd Year ISYE major • Minor in Computer Science • From Austin, TX • Have visited every state but Alaska • Intern at Deloitte Consulting focusing on SAP implementation

3. Agenda • Background/Motivation • PSoup • Introduction • System Overview • Query Processing Techniques • Implementation • Performance • Aggregation Queries • Conclusions • Critique

4. Background/Motivation • Continuous Query (CQ) Systems • Treat queries as fixed entities and stream data over them • Previous systems only allowed streaming of either data or queries • Continuously deliver results as they are computed (infeasible/inefficient) • Data Recharging • Monitoring

5. PSoup: Introduction • Query processor based on Telegraph query processing framework • Allows both data and queries to be streamed • Partially stores results to support disconnected operation and improve data throughput and response time

6. PSoup: System Overview • User initially registers query specification with system • System returns handle which can be used to invoke results of query later • Example Query: SELECT * FROM Data_Stream D_s WHERE (D_s.a < x ^ D_s.b > y) BEGIN(NOW – 10) END(NOW); • Begin-End Clause allows: • Snapshot (constant beginning and ending time) • Landmark (constant beginning and variable ending time) • Sliding window (variable beginning and ending time) • Limited by size of memory

7. PSoup: System Overview • PSoup treats execution of query streams as a join of query and data streams • Maintains State Modules (SteMs) for queries and data • One query SteM for all queries in the system, and one data SteM for each data stream

8. PSoup: Query Processing Techniques • Overview • PSoup assigns unique queryID that it returns to the user • Client can disconnect, reconnect and execute query to obtain updated results • PSoup continuously matches data to query predicates in background and stores the results in its Results Structure • When a query is invoked, PSoup applies the appropriate input window to the Results Structure to return the current results

9. PSoup: Query Processing Techniques • Entry of new Query specs • New queries split into two parts: • Standing Query Clause (SQC): consists of the SELECT-FROM-WHERE clauses • BEGIN-END clause, stored in separate WindowsTable structure • SQC inserted into Query SteM • Used to probe Data SteMs corresponding to tables in FROM clause • Resulting tuples stored in Results Structure

10. PSoup: Query Processing Techniques • Entry of new data • New tuples assigned globally unique tupleID and physical timestamp (physicalID) based on system clock • Inserted into appropriate Data SteM • Then used to probe Query SteM to determine which SQCs it satisfies • TupleIDs and physicalIDs stored in Results Structure

11. PSoup: Query Processing Techniques • Selection Queries over a single stream

12. PSoup: Query Processing Techniques • Join Queries Over Multiple Streams

13. PSoup: Query Processing Techniques • Query Invocation and Result Construction • Results Structure maintains info about which tuples in Data SteM(s) satisfy which SQCs in Query SteM • For each result tuple of each query, it stores tupleID and physicalID of all constituent base tuples of result tuple • Results of a query can be accessed by its queryID • Ordered by timestamp (physicalID)

14. PSoup: Implementation • Eddy • Each tuple has a predicate attribute and an Interest List dictating where it is to be routed • Provides Stream Prefix Consistency by storing new and temporary tuples separately in New Tuple Pool and Temporary Tuple Pool • Begins by selecting a tuple from the NTP and then processing everything in the TTP before pickign another tuple from the NTP

15. PSoup: Implementation • Data SteM • Use tree-based index for data to provide efficient access to probing queries • One red-black tree for every attribute • Maintains hash-based index over tupleIDs for fast access

16. PSoup: Implementation • Query SteM • Allows sharing of work between queries that have overlapping FROM clauses • Use red-black trees to index single-attribute single-relation boolean factors of a query

17. PSoup: Implementation • Query SteM • For queries involving joins of multiple attributes, tree structure doesn’t work • Instead, a linked list called the predicateList is used • Query SteM contains an array in which each cell represents a query • At beginning of probe by a data tuple, each cell is set to the number of boolean factors in corresponding query • Every time tuple satisfies a boolean factor, cell value is decremented • At end of probe, if cell = 0, that means the data tuple satisfies the given query

18. PSoup: Implementation • Results Structure • Stores metadata indicating which tuples satisfy which SQCs • Can either be accomplished by previously-mentioned bitmap or by associating a linked list of satisfactory data tuples for each query • Ordering by timestamp is simple for single-table queries • For Join queries, typically use oldest timestamp

19. PSoup: Performance • Implemented in Java with customized versions of Eddy and SteMs • Examined performance of two versions: • PSoup-Partial (PSoup-P): Maintain results corresponding to SQCs in Results Structure, and apply BEGIN-END clauses to retrieve current results on query invocation • PSoup-Complete (PSoup-C): Continuously maintains results corresponding to current input window for each query in linked lists • NoMat: Measurements of a system that doesn’t materialize results

20. PSoup: Performance • Storage Requirements • NoMat: Storage cost = space taken to store base data streams within maximum window over which queries are supported, plus size of structures • PSoup-P: Storage cost = storage cost of NoMat + size of Results Structure (either bitarray or linked-list) • PSoup-C: Storage cost >> storage cost of PSoup-P since C always stores current results of standing queries at a given time

21. PSoup: Performance • Experimental Setup • Varied window sizes (27-216) and number(1-8)/type of boolean factors • Measured response time and maximum supportable data arrival rate • Examined both P and C with and without predicate indexes • Tested scheme to remove redundancies arising from joins • Used synthetic generated query(27-212) /data streams

22. PSoup: Performance • Response Time vs. Window Size

23. PSoup: Performance • Response Time vs. # Interval Predicates

24. PSoup: Performance • Data Arrival Rate vs. # SQCs

25. PSoup: Performance • Summary of Results • Materializing results of queries supports higher query invocation rates • Indexing queries and lazily applying windows improves maximum data throughput • PSoup-C requires more memory • PSoup-C optimizes query invocation rate • PSoup-P optimizes data arrival rate

26. PSoup: Performance • Removing Redundancy in Join processing • Entry of a query specification or new data • Composite tuples in joins

27. PSoup: Aggregation Queries • PSoup can support aggregate functions • Only possible to share data structures across queries with identical SELECT-PROJECT-JOIN clause

28. PSoup: Conclusions • Treats data and query streams analogously • Can support queries that require access to data that arrived before and after the query • Materializes results to cut down on response time and to support disconnected operation • Enables data recharging and monitoring • Future work: • Write data streams to disk and execute queries over them • Transfer queries between disk and memory, allowing query execution to be scheduled • Confront resource constraints when dealing with infinite streams • Query browser for temporal data

29. Critique • Strengths • Very well written, easy to follow • Clear examples, excellent explanation of performance results • Strong method that reduces processing time with increase in interval predicates • Weaknesses • Lacking sufficient data on storage costs • Experimentation only tested one multiple-relation boolean factor for joins; unrealistic • Didn’t address whether same (or similar) query could be entered twice and accidentally given two ID’s