Optimal aggregation algorithms for middleware Ronald Fagin, Amnon Lotem , and Moni Naor

1. Optimal aggregation algorithms for middlewareRonald Fagin, AmnonLotem, and MoniNaor Presented by Suresh Barukula 2011csz8090

2. What is top-k query processing • Top-k query processing means finding k-objects, that have highest overall grades. • A query in multimedia database *combines different graded attributes through an aggregation function *Overall grade for each object will be calculated using an aggregation function, and we can return top-k objects.

3. Why is it important? In general multimedia databases contains fuzzy data. For example: We want to retrieve all red objects What we can say about the below object? Is it red or not? We can’t say whether it is red or not, but we can grade it by the amount of redness. attribute values are typically graded [0,1]

4. What are the ways? • FA-Fagin’s Algorithm • TA-Threshold Algorithm • TAZ Algorithm • NRA- No Random Access • CA- Combined Algorithm

5. The database model • N-Number of Objects • m-No of attributes • xi Є [0,1] • Database is consisting of m sorted lists L1…Lm; each of length N . We may refer to Li as list i. Each entry of Li is of the form (R, xi), where xi is the ithfield of object R, Each list Li is sorted in descending order by the xi value.

6. Modes of accessing the database • Sorted access • Random access • The cost of the middleware is sCS+ rCR Where s is the no of sorted accesses, r is no of random accesses , CS is sorted access cost and CR is random access cost.

7. M Object ID Attribute 1 Attribute 2 (a, 0.9) (d, 0.9) 0.9 0.85 a (b, 0.8) (a, 0.85) 0.8 0.7 b (c, 0.72) (b, 0.7) 0.72 0.2 c . . . . . . . . 0.6 0.9 d . . . . . . . . . . . . N (d, 0.6) (c, 0.2) Example – Simple Database model Sorted L1 Sorted L2

8. Example – Simple Query Find the top 2 (k = 2) objects on the following ‘query’ executed on the middleware: A1 & A2(eg: color=red & shape=round) A1 & A2 as a ‘query’ to the middleware results in combining of the grades of A1 andA2 by min(A1,A2)

9. L2 L1 (a, 0.9) (d, 0.9) A1 A2 Min(A1,A2) (b, 0.8) (a, 0.85) (c, 0.72) (b, 0.7) . . . . . . . . (d, 0.6) (c, 0.2) Example – Fagin’s Algorithm STEP 1: Read attributes from every sorted list • Stop when k objects have been seen in common from all lists ID a 0.85 0.9 d 0.9 b 0.8 0.7 0.72 c

10. L2 L1 (a, 0.9) (d, 0.9) ID A1 A2 Min(A1,A2) (b, 0.8) (a, 0.85) (c, 0.72) (b, 0.7) . . . . . . . . c (d, 0.6) (c, 0.2) Example – Fagin’s Algortihm STEP 2: Random access to find missing grades a 0.85 0.9 0.6 d 0.9 b 0.8 0.7 0.72 0.2

11. (d, 0.9) ID A1 A2 Min(A1,A2) (a, 0.85) (b, 0.7) . . . . c (c, 0.2) Example – Fagin’s Algortihm STEP 3 • Compute the grades of the seen objects. • Return the k highest graded objects. L2 L1 (a, 0.9) (b, 0.8) 0.85 a 0.85 0.9 (c, 0.72) 0.6 0.6 d 0.9 . . . . b 0.8 0.7 0.7 0.2 0.2 0.72 (d, 0.6)

12. d: 0.9 a: 0.85 b: 0.7 . . . . c: 0.2 Threshold Algorithm (TA) Read all grades of an object once seen from a sorted access • No need to wait until the lists give k common objects Do sorted access (and corresponding random accesses) until you have seen the top k answers. • How do we know that grades of seen objects are higher than the grades of unseen objects ? • Predict maximum possible grade unseen objects: L2 L1 a: 0.9 Seen b: 0.8 c: 0.72 T = min(0.72, 0.7) = 0.7 f: 0.6 . . . . f: 0.65 Possibly unseen Threshold value d: 0.6

13. L2 L1 (a, 0.9) (d, 0.9) ID A1 A2 Min(A1,A2) (b, 0.8) (a, 0.85) (c, 0.72) (b, 0.7) . . . . . . . . (d, 0.6) (c, 0.2) Example – Threshold Algorithm Step 1: - parallel sorted access to each list For each object seen: - get all grades by random access - determine Min(A1,A2) - amongst 2 highest seen ? keep in buffer a 0.9 0.85 0.85 d 0.9 0.6 0.6

14. L2 L1 a: 0.9 d: 0.9 ID A1 A2 Min(A1,A2) b: 0.8 a: 0.85 a 0.9 c: 0.72 b: 0.7 0.9 d . . . . . . . . d: 0.6 c: 0.2 Example – Threshold Algorithm Step 2: - Determine threshold value based on objects currently seen under sorted access. T = min(L1, L2) - 2 objects with overall grade ≥ threshold value ? stop else go to next entry position in sorted list and repeat step 1 0.85 0.85 0.6 0.6 T = min(0.9, 0.9) = 0.9

15. L2 L1 (a, 0.9) (d, 0.9) ID A1 A2 Min(A1,A2) (b, 0.8) (a, 0.85) (c, 0.72) (b, 0.7) . . . . . . . . (d, 0.6) (c, 0.2) Example – Threshold Algorithm Step 1 (Again): - parallel sorted access to each list For each object seen: - get all grades by random access - determine Min(A1,A2) - amongst 2 highest seen ? keep in buffer a 0.9 0.85 0.85 d 0.6 0.9 0.6 b 0.8 0.7 0.7

16. L2 L1 a: 0.9 d: 0.9 ID A1 A2 Min(A1,A2) b: 0.8 a: 0.85 a 0.9 c: 0.72 b: 0.7 0.7 b . . . . . . . . d: 0.6 c: 0.2 Example – Threshold Algorithm Step 2 (Again): - Determine threshold value based on objects currently seen. T = min(L1, L2) - 2 objects with overall grade ≥ threshold value ? stop else go to next entry position in sorted list and repeat step 1 0.85 0.85 0.7 0.8 T = min(0.8, 0.85) = 0.8

17. L2 L1 a: 0.9 d: 0.9 ID A1 A2 Min(A1,A2) b: 0.8 a: 0.85 a 0.9 c: 0.72 b: 0.7 0.7 b . . . . . . . . d: 0.6 c: 0.2 Example – Threshold Algorithm Situation at stopping condition 0.85 0.85 0.7 0.8 T = min(0.72, 0.7) = 0.7

18. TA vs FA • The middleware cost of the FA is same no matter what the aggregation function is .  • TA stops at least as early as FA • TA may perform more random accesses than FA • TA requires only bounded buffers • TA can be stopped early(θ-approximation)

19. Algorithm B isinstance optimal over A and D if : B ЄA and Cost(B,D ) = O(Cost(A,D )) A ЄA,D ЄD Which means that: Cost(B,D ) ≤ c . Cost(A,D ) + c’, A ЄA,D ЄD , optimality ratio Concept of instance optimality A = class of algorithms, AЄA represents an algorithm D = legal inputs to algorithms (databases), D ЄD represents a database Cost(A,D ) = middleware cost when running algorithm A over database D

20. Some facts about TA • Theorem: If the aggregation function t is monotone, TA correctly finds the top K answers. • Theorem: TA is instance optimal for every monotone aggregation function, over every database (Note: if we exclude wild guesses).

21. (d, 0.9) (a, 0.85) (b, 0.7) . . . . (c, 0.2) TAZ Algorithm(No Sorted Access) L3 L2 L1 (b, 0.6) (a, 0.9) (a, 0.83) (b, 0.8) (d, 0.61) (c, 0.72) 1 . . . . . . . . (c, 0.9) (d, 0.6) T=min(0.72,0.7,1)=0.7

22. 1/3 1/3 1/3 . . . . 1/3 NRA(No Random Access) • Can we determine rank of an object without seeing all of it’s grades? • The main essence of this algorithm is estimating the rank using best and worst possible values 1 1/3 1/3 . . . . 1/3

23. CA(Combined Algorithm) • CA is merge between TA and NRA. • The idea of CA is to run NRA but after every h steps to perform random access step. • Both NRA and CA are instance optimal over all databases, when the aggregation function is monotone

24. Conclusion • In this paper the authors have studied a simple and elegant algorithm called TA. • They have also studied the variants of TA, when there are no sorted access, no random access etc.., • They have emphasized on instance optimality and they have proved that their algorithms are instance optimal over all algorithms for all databases under normal assumptions. • But they have not considered the computational costs and the data structures that are required to implement the algorithms.

25. Questions?