Querying Infinite Databases

1. Querying Infinite Databases Safety of Datalog Queries over infinite Databases (Sagiv and Vardi ’90) Queries and Computation on the Web (Abiteboul and Vianu ’97) Itay Maman 049011 Student Symposium, 5 July 2006

2. Simple Technion Queries… (Domain: The Technion’s students database) • Q1: Which courses did Gidi attend? • SELECT course FROM students WHERE name='Gidi' • Q2: Which students took 234218? • SELECT name FROM students WHERE course='234218'

3. Simple Web Queries… Q3: Which pages does my home page link to? SELECT target FROM links WHERE source='www.geocities.com/mysite' Q4: Which pages link to my home page? SELECT source FROM links WHERE target='www.geocities.com/mysite' Q4 is challenging: No matter how long my web-crawler works… … I can never find all incoming links of a page! This is an infinite query The more you crawl the more answers you get (In Q3 the size of the result set is bounded)

4. Leading questions • What does an infinite DB look like? • Can we evaluate a query over an infinite DB? • Can we determine the finiteness of a query? • But first, some Datalog…

5. Datalog • Why Datalog? • Supports recursion/transitive closure (unlike SQL) • Recursion is essential in large data-sets • Terminates if DB is finite • Very simple • program = A collection of rules • rule = A sequence of terms • In our program: • Three rules • Two queries (AKA: IDB): g(X), small(X,Y) • One Table (AKA: EDB): before(X,Y) • A goal predicate from which execution starts • We choose g(X) as the goal g(X) :- small(X,2). small(X,Y) :- before(X,Y). small(X,Y) :- small(X,Z), before(Z,Y).

6. Finiteness • A DB is finite If every table is a finite set • before(X,Y)  { (0,1), (1,2), (2,3) } • Possible evaluation schemes: • Brute force • Bottom up • Optimizations • The Requirement: Finiteness of tables • The guarantee: Termination of the Datalog program

7. Infinity • Here is another definition for our table • before(X,Y)  { (X,X+1) | X  0 } • We now have an infinite DB • The Problem: we cannot iterate over the tuples in the set • The solution: Top-down algorithm • Such tables are quite common • The internet links relation links(X,Y)  { (X,Y) | page X links to page Y } • Java’s subclassing relation extends(X,Y)  { (X,Y) | class X extends Y } Leading question: What does as infinite DB look like?

8. Example: Top-down evaluation g(W) :- small(W,2). small(A,B) :- before(A,B). small(X,Y) :- small(X,Z), before(Z,Y). before(X,Y)  { (X,X+1) | X  0 } s(X,Y) = b(X,Y)  s(X,Z)  b(Z,Y) g(W) = s(W,2) = b(W,2) s(W,Z) b(Z,2) = {(1,2)}s(W,1) {(1,2)} = {(1,2)} [b(W,1)s(W,Z) b(Z,1)]  {(1,2)} = {(1,2)} [{(0,1)}s(W,0){(0,1)}]  {(1,2)} = {(1,2)} [{(0,1)} [b(W,0) s(W,Z) b(Z,0)]  {(0,1)}]  {(1,2)} = {(1,2)} [{(0,1)} [s(W,Z) ]  {(0,1)}]  {(1,2)} = {(1,2)} [{(0,1)}   {(0,1)}]  {(1,2)} = {(1,2)} {(0,1)}  {(1,2)} = {(1,2)} {(0,2)} = {(1,2), (0,2)} • b : before • s : small •  : Join

9. Top-down evaluation • The Top-down algorithm • Init: assign r  body of the goal • Loop: • (Intelligently) Pick a term, t, from r • If t is a query term: • Replace it with the union of the rules indicated by t • If t is a table term: • Replace it with the set generated by the table • Replace s  expressions (in r) with  • Replace s   expressions (in r) with s • Evaluate relational algebra expressions (if both sides are known) • Stop if no further replacements can be made Leading question: Can we evaluate a query over an infinite DB? Yes

10. Infinite Queries • Can the top-down algorithm run forever? • Yes • Case 1: An table that returns an infinite result • evenProduct(X,Y)  { (X,Y) | X*Y mod 2 = 0 } • divides(X,Y)  { (X,Y) | X mod Y = 0 } • links(X,Y)  { (X,Y) | page X links to page Y } • weak-safety: all intermediate results are finite • Result #1 (Sagiv and Vardi ’90): • Weak-safety is decidable given F/C (finiteness constraints) of tables • F/C of evenProduct: None • F/C of divides: X => Y • F/C of links: X => Y • Algorithm: Tracking flow of values from assigned variables

11. Infinite Queries (cont.) • Can the top-down algorithm run forever? • Yes • Case 2: The algorithm’s recursion never stops • A query/table is used in its “unbounded” direction g(W) = s(2,W) = b(2,W) s(2,Z)  b(Z,W) = {(2,3)}s(2,Z) b(Z,W) = {(2,3)} [b(2,Z) s(2,Z’) b(Z’,Z)]  b(Z,W) … s(X,Y) = b(X,Y)  s(X,Z)  b(Z,Y) g(W) :- small(2,W). small(A,B) :- before(A,B). small(X,Y) :- small(X,Z), before(Z,Y). before(X,Y)  { (X,X+1) | X  0 } • Results #2-3 (Sagiv and Vardi ’90): • Termination is undecidable in the general case • Termination is decideable if all queries are unary

12. Infinite Queries (summ.) Leading question: Can we determine the finiteness of a query? No • We can automatically determine weak-safety • We cannot (automatically) determine termination • But, one can analytically prove that a given query over a given DB is finite • E.g., our small(W,2) program

13. The Web as a DB • The web data model (WDM): • A scheme of a DB that can represent the web graph • Just three tables: urls = { u | u is a url of a web-page } links = { (u1,u2) | u1 links to u2; u1, u2  urls } Words = { (u,w) | w appears in page u; u  urls } • Result #4 (Abiteboul and Vianu ’97): • If a Datalog program with no literals halts over an infinite DB, its result is  • => A non-trivial query (over an infinite DB) must have a literal

14. Web - Machines • Browsing Machine • A weakly safe Datalog program (over WDM) • At least one URL literal • Searching/Browsing Machine • An unsafe Datalog program (over WDM) • Evaluates queries in parallel • Allowed literal types: URLs, Words • Claims #1-2 (Abiteboul and Vianu ’97): • Browsing machine: • Represent a user following static links from a page • Searching/Browsing machine: • Also allows the user to access search engine

15. Discussion: Finite approximation • Relational Database servers are very popular • Such DBs are finite • Also, computing a table on demand may be slow • Better performance at batch processing The challenge: Build a finite replacement for an infinite DB • Formally: • Given a finite query, q, over an infinite DB,  • (Finiteness of q proved analytically) • Build a finite Database, , such that q over  yield the same result as q over 

16. Discussion: Finite approximation • Example: Our small(W,2) program • A finite, sound table: before(X,Y)  { (0,1), (1,2) } • A finite, unsound table: before(X,Y)  { (0,1) } • The process: • Compute the transitive closure of the before relation • Start from the literal ‘2’ at the right-hand side position • Condition: the table graph must end with a sink • In before the sink is the vertex ‘0’ • => We can build a finite DB • Sadly, In the web-graph no such sink exists

17. Discussion: Temporality • Crawling takes time • The subject may change while crawling • The DB is a snapshot which never happened • (Open Question): • Can we decide whether a result was really “true” at some point?

18. More issues • Relational algebra over large relations • BDD • Negation • Stratified Datalog

19. - Questions ? -

21. Datalog • Semantics: ??? • Straight forward mapping to Relational Algebra?? g(X) :- small(X,2). small(X,Y) :- before(X,Y). small(X,Y) :- small(X,Z), before(Z,Y).

22. Example: Bottom-up evaluation Initialization: Translate the EDBs into relations

23. Example: Bottom-up evaluation apply small(X,Y) :- before(X,Y).

24. Example: Bottom-up evaluation apply small(X,Y) :- small(X,Z), before(Z,Y). Join

25. Example: Bottom-up evaluation apply g(X) :- small(X,2).

26. Finiteness before(X,Y)  { (0,1) (1,2) (2,3) } • The Bottom-up algorithm: • Init: • For each EDB, p, assign r(p)  Relation of all tuples satisfying p • For each IDB, p, assign r(p) • Loop: • Choose a rule p(…) :- t1(…), t2(…), … tn(…) • t join of all r(ti), where 1  i  n • r(p)r(p)t • Continue until a fix-point is reached • Requires: Finiteness of EDBs • Ensures: Termination