Lore: A Database Management System for Semistructured Data

1. Lore: A Database Management System for Semistructured Data

2. Why? • Although data may exhibit some structure it may be too varied or irregular to map to a fixed schema. • Relational DBMS might use null values in this case. • May be difficult to decide in advance on a specific schema. • Data elements may change types. • Structure changes a lot (lots of schema modifications).

3. Semistructured Data • Examples: • Data from the web • Overall site structure may change often. • It would be nice to be able to query a web site. • Data integrated from multiple, heterogeneous data sources. • Information sources change, or new sources added.

4. Object Exchange Model (OEM) • Data in this model can be thought of as a labeled directed graph. • Schema-less and self-describing. • Vertices in graph are objects. • Each object has a unique object identifier (oid), such as &5. • Atomic objects have no outgoing edges and are types such as int, real, string, gif, java, etc. • All other objects that have outgoing edges are called complex objects.

5. OEM (Cont.) • Examples: • Object &3 is complex, and its subobjects are &8, &9, &10, and &11. • Object &7 is atomic and has value “Clark”. • DBGroup is a name that denotes object &1.(Names are entry points into the database).

7. OEM to XML • Example: • <Member project=“&5 &6”> <name>Jones</name> <age>46</age> <office> <building>gates</building> <room>252</room> </office></member> • This corresponds to rightmost member in the example OEM, where project is an attribute.

8. Lorel Query Language • Need query language that supports path expressions for traversing graph data and handling of ‘typeless’ data. • A simple path expression is a name followed by a sequence of labels. • DBGroup.Member.Office. • Set of objects that can be reached starting with the DBGroup object, following edges labels member and then office.

9. Lorel (cont.) • Example: • select DBGroup.Member.Officewhere DBGroup.Member.Age < 30 • Result: • Office “Gates 252” • Office Building “CIS” Room “411”

10. Lorel Query Rewrite • Previous query rewritten to: • select Ofrom DBGroup.Member M, M.Office Owhere exists y in M.Age : y < 30 • Comparison on age transformed to existential condition. • Since all properties are set-valued in OEM. • A user can ask DBGroup.Member.Age < 30 regardless of whether Age is single valued, set valued, or unknown.

11. Lorel Query Rewrite • Why? • Breaking query into simple path expressions necessary for query optimization. • Need to explicitly handle coercion. • Atomic objects and values. 0.5 < “0.9” should return true • Comparing objects and sets of objects. DBGroup.Member.Age is a set of objects.

12. Lorel (cont.) • General path expressions are loosely specified patterns for labels in the database.(‘|’ disjunction, ‘?’ label pattern optional) • Example: • select DBGroup.Member.Namewhere DBGroup.Member.Office(.Room%|.Cubicle)? like “%252” • Result: • Name “Jones”Name “Smith”

13. Query and Update Processing • Query is parsed • Parse tree is preprocessed and translated to new OQL-like query. • Query plan constructed. • Query optimization. • Opt. query plan executed.

14. System Architecture

15. Iterators and Object Assignments • Use recursive iterator approach: • execution begins at top of query plan • each node in the plan requests a tuple at a time from its children and performs some operation on the tuple(s). • pass result tuples up to parent.

16. Object Assignments (OAs) • OA is a data structure containing slots for range variables with additional slots depending on the query. • Each slot within an OA will holds the oid of a vertex on a path being considered by the query engine. • Example: if OA1 holds oid for “Smith” then OA2 and OA3 can hold the oids for one of Smiths Office objects and Age objects.

18. Query Operators • For example, the Scan operator returns all oids that are subobjects of a given object following a specified path expression. • Scan (StartingOASlot, Path_expression, TargetOASlot) • For each oid in StartingOASlot, check to see if object satisfies path_expression and place oid into TargetOASlot. • Other operators include Join, Project, Select, Aggregation, etc. • Join node like nested-loop join in relational DBMS.

19. Query Optimization • Does only a few optimizations: • Push selection ops down query tree. • Eliminate/combine redundant query operators. • Explores query plans that use indexes where possible. • Two kinds of indexes: • Lindex (link index) provide parent pointers impl. as hashing. • Vindex (value index) impl. as B+-trees

20. Indexes • Because of non-strict typing system, have String Vindex, Real Vindex, and String-coerced-to-real Vindex. • Separate B-Trees for each type are constructed. • Using Vindex for comparison (e.g. Age < 30) consider the following: • If type is string, do lookup in String Vindex • If can convert to real the do lookup in String-coerced-to-real Vindex. • If type is real?

22. Other issues • Update query operator example: • Update(Create_Edge, OA1, OA5, “Member”) • Create edge from results in OA1 to OA5 labeled “Member”. • Lore arranges objects in physical disk pages, each page with a number of slots with a single object in each slot. • Objects placed according to first-fit algorithm. • Supports large objects spanning multiple pages. • Objects clustered in depth-first manner (since Scan traverses depth-first). • Garbage collector removes unreachable objects.

23. External Data Manager • Enables retrieval of information from other data sources, transparent to the user. • An external object in Lore is a “placeholder” for the external data and specifies how lore interacts with an external data source. • The spec for an external object includes: • Location of a wrapper program to fetch and convert data to OEM, time interval until fetched information becomes stale, and a set of arguments used to limit info fetched from external source.

24. Data Guides • A DataGuide is a concise and accurate summary of the structure of an OEM database (stored as OEM database itself, kind of like the system catalog). • Why? • No explicit schema, how do we formulate meaningful queries? • Large databases (can’t just view graph structure). • What if a path expression doesn’t exist (waste). • Each possible path expression is encoded once.

25. {9, 13}

26. DataGuides As Histograms • Each object in the dataguide can have a link to its corresponding target set. • A target set is a set of oids reachable by that path. • TS of DBGroup.Member.Age is {9, 13}. • This is a path index. Can find set of objects reachable by a particular path. • Can store statistics in DataGuide (more in next paper). • For example, the # of atomic objects of each type reachable by p.

27. Conclusions • Takes advantage of the structure where it exists. • Handles lack of structure well (data type coercion, general path expressions). • Query language allows users to get and update data from semistructured sources. • DataGuide allows users to determine what paths exist, and gives useful statistical information

28. Query Optimization for Semistructured Data

29. OEM vs. XML • OEM’s objects correspond to elements in XML • Sub-elements in XML are inherently ordered. • XML elements may optionally include a list of attribute value pairs. • Graph structure for multiple incoming edges specified in XML with references (ID, IDREF attributes). i.e. the Project attribute.

30. Indexes • Vindex(op, value, l, x) places into x all atomic objects that satisfy the “op value” condition with an incoming edge labeled l. • Vindex(“Age”, <, 30,y) places into y objects with age < 30. • Lindex(x, l, y) places into x all objects that are parents of y via edge labeled l. • Lindex(x, “Age”, y) places into x all parents of y via label “Age”.

31. Indexes (cont.) • Bindex(l, x, y) finds all parent-child object pairs connected by a label l. • Bindex(“Age”, x, y) locates all parent-child pairs with label Age. • Pindex(PathExpression, x) placed into x all objects reachable via the path expression. • Pindex(“A.B x, x.C y”, y) places into y all objects reachable by going from A to B to C. • Uses DataGuide.

32. Simple Query • select Ofrom DBGroup.Member M, M.Office Owhere exists y in M.Age : y < 30 • Possible plans: • Top-down (similar to pointer-chasing, nested-loops join) • Use Vindex to check y < 30, traverse backwards from child to parent using Lindex(bottom-up). • Hybrid, both top down and bottom up. Meet in middle.

33. Select xFrom A.B xWhere exists y in x.C: y = 5

34. Query Plan Generation (Overview) • Logical query plan generator creates high-level execution strategy. • Physical query plan enumerator uses statistics and a cost model to transform logical query plan into an estimated best physical plan that lies within their search space.

35. Logical Query Plans (cont.) • Glue node represents a ‘rotation point’ that has as its children two independent subplans. • Rotating the order between independent components yields different plans. • Marks place where execution order is not fixed. • Discover node chooses best way to bind variables x and y. • Chain node chooses best evaluation of a path expression.

36. Logical query plan for:Select xFrom DBGroup.Member xWhere exists y in x.Age: y<30 from clause where clause

37. Physical Query Plans

38. Physical Query Plans (cont.) • Scan(x, l, y) places into y all objects that are subojects of x via edge labeled l. • Top-down (pointer chasing). • Lindex plan is bottom-up approach. • Bindex: Locate edges whose label appears infrequently in database. • NLJ: left subplan passes variables to right subplan.

39. Statistics • I/O metric uses estimated # of objects fetched. • For every label subpath p of length <= k: • # Of atomic objects of each type reachable by p • Min, and max values of all atomic objects of each type reachable by p • # Of instances of path p, denoted |p| • # Of distinct objects reachable by p, denoted |p|d • # Of l-labeled subobjects of all objects reachable by p • # Of incoming l-labeled edges to any instance of p, denoted |pl|

40. Plan Enumeration • Doesn’t consider joining two simple path expressions together unless they share a common variable. • Pindex is used only when path expression begins with a name and no variable except the last is used in the query. • Select clause always executes last. • Doesn’t try to reorder multiple independent path expressions.

42. Results • Used XML database about movies. Database graph contained 62,256 nodes and 130,402 edges. • Experiment 1: Select DB.Movie.Title • Best plan is Pindex, followed by top-down • Worst plan is Bindex, with hash joins.

43. Results (cont.) • Experiment 2: All Movies with a Genre of “Comedy” • Where clause is very selective, bottom-up does a Vindex for “Comedy” with incoming edge Genre

44. Results (cont.) • Experiment 3: Query with two existentially quantified variables in the where clause. • Errors due to bad estimates of atomic value distributions and set operation costs.

45. Results (cont.) • Experiment 4: Select movies with certain quality rating. • Quality ratings uncommon in database so optimizer chooses to find all ratings via Bindex, and then work bottom-up.

46. Conclusions • Cost estimates are accurate and select the best plan most of the time • Execution times of best and worst plans for a given query can differ by many orders of magnitude. • Best strategy is highly dependent upon the query and database (Query optimization is good for XML data).