ΗΥ-562 Advanced Topics on Databases Αλέκα Σεληνιωτάκη Ηράκλειο, 22/05/2012

1. Efficient Query Evaluation over Temporally Correlated Probabilistic StreamsBhargavKanagal, AmolDeshpande ΗΥ-562 Advanced Topics on Databases Αλέκα Σεληνιωτάκη Ηράκλειο, 22/05/2012

2. Contents • Motivation • Previous Work • Approach • Markov Sequence • Formal Semantics • Operator Design • Query Planning Algorithm • Results • Feature Work

3. Motivation • Correlated probabilistic streams occur in large variety of applications including sensor networks, information extraction and pattern/event detection. • Probabilistic data streams are highly correlated in both time and space. Impact final query results

4. Motivation • A habitat monitoring application: Query example: Compute the likelihood hat a nest was occupied for all 7 days in a week. Other issues: • Handle high-rate data streams efficiently and produce accurate answers to continuous queries in real time • Query semantics become ambiguous since they are dealing with sequences of tuplesand not set of tuples.

5. Previous Work • Probabilistic databases (Mystiq,Trio,Orion,MayBMS) focus on static data and not streams, also cannot handle complex correlations • Lahar, Caldera are applicable to probabilistic streams, but focus on pattern identification queries. • Unable to represent the types of correlations that probabilistic streams exhibit • Can not be applied directly because of their complexity

6. Approach • Observation: Many naturally occurring probabilistic streams are both structured and Markovian in nature • Structured: Same set of correlations and independences repeat across time • Markovian: The values of variables at time t + 1 are independent of those at time t − 1 given their values at time t • Design compact, lightweightdata structures to representand modify • them • Enable incremental queryprocessing using the iterator framework A Markov sequence as a DGM

7. ApproachProbabilistic Databases • Probabilistic Databases exhibit two types of uncertainties: • Tuple existence (captures the uncertainty whether a tuple exists in a database) • Attribute value (captures the uncertainty about the value of an attribute) • Equivalence between Probabilistic Databases and Directed • Graphical Models: • nodes in the graph: denote the random variables • edges: correspond to the dependencies between the random variables (dependencies quantified by a conditional probability distribution function (CPD) for each node that describes how the value of that node depends on the value of its parent)

8. ApproachSystem’s Description

9. ApproachSystem’s Description • Input:a query on probabilistic streams • Output: query’s results • Steps: • Query conversion to a probabilistic sequence algebra expression • Query plan construction by instantiating each of the operators with the schemas of their input sequences • Each operator executes its schema routine and computes its output schema • Check the input to the projection and determine if projection operator is safe. • Optimize query plan

10. Markov Sequence • A probabilistic sequence Sp, is defined over a set of named attributes S = {V1,V2…Vk}, also called its schema, where each Vi is a discrete probabilistic value. • A probabilistic sequence is equivalent to a set of deterministic sequences, where each deterministic sequence is obtained by assigning a value to each random variable. • Two operators for conversion from probabilistic sequence to deterministic sequence: • MAP: returns the possible sequence that has the highest probability • ML: returns a sequence of items each of which has the highest probability for its time instant.

11. Markov Sequence • Special case of a probabilistic sequence. Completely determined by specifying successive joint distributions for all time instants p(Vt, Vt+1) • Efficient Representation of Markov sequences using a combination of the schema graph and the clique list: • Schema graph: graphical representation of the two step joint distribution that repeats continuously. • Clique list: the set of direct dependencies present between successive sets of random variables. Schema graph Clique List

12. Formal Semantics Possible World Semantics (+small modification for sequences) • Operators: Select, Project, Join, Aggregate, Windowing • MAP, ML: Convert a Markovian sequence into a deterministic sequence • The set of Markov sequences is not closed under these operators, i.e., some operators return non-Markovian sequences (Projection, Windowing) • Op is safe for input schema I , Op(I) is a Markovian sequence

13. Operator Design • Operators treat the input Markov sequence as a data stream, operating on one tuple(an array of numbers) at a time and produce the output in the same fashion (passed to the next operator). • Each operator implements two high level routines: • Schema routine: Output schema is computed based on the input schema • get next() routine: Operates on each of the input data tuples to compute the output tuples.

14. Operator DesignSelection Generally: Add new boolean random variable to each slice. Always safe • Steps: • Start with the DGM corresponding to the input schema • Add a new node corresponding to the exists variable to both time steps of the DGM • Connect this node to the variables that are part of selection predicate through directed edges (selection predicate X>Y) • Update the clique list of the schema to include the newly created dependencies. Add new boolean random variable to each slice. Schema Routine

15. Operator DesignSelection • get_next() routine: • determine the CPD of the newly created node • add it to the input tuple’s CPD list • return the new tuple The algorithm does not change the Markovian property of the sequence Safe operator.

16. Operator DesignProjection • Steps: • Remove the nodes that are not in the projection list - Eliminate operation on the graphical model-Update clique list with eliminations • Determine if new edges need to be added to the schema graph Generally: Eliminate all variables not of interest. Unsafe for schemas Schema Routine

17. Operator DesignProjection • get_next() routine: perform the actual variable elimination procedure to eliminate the nodes that are not required Projection is not safe for all input sequences, and in some cases, even if the input is a Markov Sequence, the output may not be a Markov sequence.

18. Operator DesignJoin • schema routine: concatenate the schemas of the two sequences in order to determine the resulting output schema (combine the schema graphs, and concatenate the clique lists) • get_next() routine: concatenate the CPD lists of the two tuples, whenever both tuples have the same time value. • a join can be computed incrementally, and is always safe.

19. Operator DesignAggregation • Schema routine: the output schema is just a single attribute corresponding to the aggregate • get_next() routine: Case1: no selection predicates (Gi=Agg(X1,X2…Xi)) dotted variables added to the DGM, the boxed nodes are eliminated and computing p(Xi,Gi) as input tuples arrive. At the end of the sequence, we get p(Xn,Gn), from which we can obtain p(Gn) by eliminating Xn. Case2: with selection predicates a value Xi contributes to the aggregate only if Ei (exists attribute) is 1 and not otherwise Always safe operator

20. Operator DesignSliding Window Aggregates • A sliding window aggregate query asks to compute the aggregate values over a window that shifts over the stream (characterized by the length of the window, the desired shift, and the type of aggregate) DGM for sliding window aggregate. Gi denotes the aggregates to compute After eliminating the Xi variables, make a clique list on the Gi variables. • The aggregate value for a sliding window influences the aggregates for all of the future windows in the stream Unsafe Operator

21. Operator DesignSliding Window Aggregates • Ignoring the dependencies between the aggregate values produced at different time instances • Computing the distribution over each aggregate value independently • Splitting the sliding window DGM into separate graphical models (one for each window), run inference on each of them separately and compute the results Unmarked nodes are intermediate aggregates

22. Operator DesignTumpling Window Aggregates • The length of the sliding window is equal to its shift  can compute exact answers in a few cases. For tumbling window aggregates, only eliminate only eliminate boxed nodes to obtain the obtain Markov sequence : Eliminating X3και Χ5is postponed to a later projection.

23. Operator DesignMAP-ML • MAP operator takes in a Markov sequence and returns a deterministic sequence. It is usually the last operator in the query plan, and hence it does not have a schema routine. The get next() routine uses the dynamic programming based approach of Viterbi'salgorithm. (Viterbi-style dynamic Programming) ML: at each time step, compute the probability distribution for each time instant from each tuple. Based on this eliminate the variables that are not required and determine the most likely values for the variables. Simply marginalize the joint distribution.

24. Query EvaluationQuery Syntax • The user has the choice between using MAP or ML operators for converting the final probabilistic answer to a deterministic answer • Support for specifying sliding window parameters

25. Query Planning Algorithm • Unsafe Operators: Projection, Tumbling window operator reduced to a projection, window operator use approximate method • Query Planning = Determining the correct position for Projection • Strategy: Pull up projection operator until it is safe. If no safe position for projection, check its parent: ML  we can determine a safe plan, MAP  we cannot determine a safe plan • plan

26. Query Planning Algorithm • For a given query, convert it to a probabilistic sequence algebra expression:Q0: SELECT_MAP MAX(X) FROM SEQ WHERE Y<20MAP(Gp(ΠpΧ(σpΥ<20SEQ))) • Each operator then executes its schema routine and computes its output schema, which is used as the input schema for the next operator in the chain • Check the input to the projection, and determine if the projection operator is safe • If a projection-input pair is not safe, we pull up the projection operator through the aggregate and the windowing operators and continue with the rest of the query plan: MAP(ΠpMAX(X)(Gp(σpY<20SEQ))) • If the operator after the projection is ML, then determine the exact answer. If it is a MAP operator, replace both the projection and the MAP operator with the approximate-MAP operator

27. Results • Markov sequence generator: Generate Markov sequences for a given input schema • Capturing and reasoning about temporal correlations is critical for obtaining accurate query answers • Operators can process up to 500 tuples per second • Get_next() query processing framework that exploits the structure in Markov sequences is much more efficient that previous generic approaches

28. Results The % error in query processing for various operators when temporal correlations are ignored. Q1: SELECT MAP Agg(A) FROM S Q3: SELECT MAP MAX(A) FROM S[size,size]

29. Feature Work Scalability improvement of the system by resorting to approximations with guarantees.

30. References • B.Kanagal, A.Deshpande: “Efficient Query Evaluation over Temporally Correlated Probabilistic Streams.” • http://www.cs.umd.edu/~bhargav/ICDE_0703.pdf • http://www.cs.umd.edu/~bhargav/ICDE09poster.pdf