DBMS Storage and Indexing

1. DBMS Storage and Indexing 198:541

2. Disk Storage

3. Disks and Files • DBMS stores information on (“hard”) disks. • This has major implications for DBMS design! • READ: transfer data from disk to main memory (RAM). • WRITE: transfer data from RAM to disk. • Both are high-cost operations, relative to in-memory operations, so must be planned carefully!

4. Why Not Store Everything in Main Memory? • Costs too much. • Main memory is volatile. We want data to be saved between runs. (Obviously!) • Typical storage hierarchy: • Main memory (RAM) for currently used data. • Disk for the main database (secondary storage). • Tapes, DVD for archiving older versions of the data (tertiary storage).

5. Disks • Secondary storage device of choice. • Main advantage over tapes: random access vs.sequential. • Data is stored and retrieved in units called disk blocks or pages. • Unlike RAM, time to retrieve a disk page varies depending upon location on disk. • Therefore, relative placement of pages on disk has major impact on DBMS performance!

6. See textbook for in-depth discussion on disk storage • Physical storage of files to avoid high I/O delays • Seek time and rotational delay dominate. • Seek time varies from about 1 to 20msec • Rotational delay varies from 0 to 10msec • Transfer rate is about 1msec per 4KB page • Key to lower I/O cost: reduce seek/rotation delays! Hardware vs. software solutions? • RAID organization • Reliability • Redundancy

7. DB Buffer Management in a DBMS Page Requests from Higher Levels • Data must be in RAM for DBMS to operate on it! • Table of <frame#, pageid> pairs is maintained. BUFFER POOL disk page free frame MAIN MEMORY DISK choice of frame dictated by replacement policy

8. Buffer Replacement Policy • Frame is chosen for replacement by a replacement policy: • Least-recently-used (LRU), Clock, MRU etc. • Policy can have big impact on # of I/O’s; depends on the access pattern. • Sequential flooding: Nasty situation caused by LRU + repeated sequential scans. • # buffer frames < # pages in file means each page request causes an I/O. MRU much better in this situation (but not in all situations, of course). • DBMS buffer policy has specific requirements

9. Record Organization

10. Record Formats: Fixed Length • Information about field types same for all records in a file; stored in systemcatalogs. • Finding i’th field does not require scan of record. F1 F2 F3 F4 L1 L2 L3 L4 Base address (B) Address = B+L1+L2

11. 4 $ $ $ $ Record Formats: Variable Length • Two alternative formats (# fields is fixed): F1 F2 F3 F4 Fields Delimited by Special Symbols Field Count F1 F2 F3 F4 Array of Field Offsets • Second offers direct access to i’th field, efficient storage • of nulls(special don’t know value); small directory overhead.

12. Page Formats: Fixed Length Records Slot 1 Slot 1 • Record id = <page id, slot #>. In first alternative, moving records for free space management changes rid; may not be acceptable. Slot 2 Slot 2 Free Space . . . . . . Slot N Slot N Slot M N . . . 1 1 1 M 0 M ... 3 2 1 number of records number of slots PACKED UNPACKED, BITMAP

13. Page Formats: Variable Length Records Rid = (i,N) Page i • Can move records on page without changing rid; so, attractive for fixed-length records too. Rid = (i,2) Rid = (i,1) N Pointer to start of free space 20 16 24 N . . . 2 1 # slots SLOT DIRECTORY

14. Files of Records • Page or block is OK when doing I/O, but higher levels of DBMS operate on records, and files of records. • FILE: A collection of pages, each containing a collection of records. Must support: • insert/delete/modify record • read a particular record (specified using record id) • scan all records (possibly with some conditions on the records to be retrieved)

15. File Organization

16. Alternative File Organizations Many alternatives exist, each ideal for some situations, and not so good in others: • Heap (random order) files:Suitable when typical access is a file scan retrieving all records. • Sorted Files:Best if records must be retrieved in some order, or only a `range’ of records is needed. • Indexes: Data structures to organize records via trees or hashing. • Like sorted files, they speed up searches for a subset of records, based on values in certain (“search key”) fields • Updates are much faster than in sorted files.

17. Unordered (Heap) Files • Simplest file structure contains records in no particular order. • As file grows and shrinks, disk pages are allocated and de-allocated. • To support record level operations, we must: • keep track of the pages in a file • keep track of free space on pages • keep track of the records on a page • There are many alternatives for keeping track of this.

18. Heap File Implemented as a List • The header page id and Heap file name must be stored someplace. • Each page contains 2 `pointers’ plus data. Data Page Data Page Data Page Full Pages Header Page Data Page Data Page Data Page Pages with Free Space

19. Data Page 1 Header Page Data Page 2 Data Page N DIRECTORY Heap File Using a Page Directory • The entry for a page can include the number of free bytes on the page. • The directory is a collection of pages; linked list implementation is just one alternative. • Much smaller than linked list of all HF pages!

20. Index Structures

21. Indexes • An index on a file speeds up selections on the search key fields for the index. • Any subset of the fields of a relation can be the search key for an index on the relation. • Search key is not the same as key(minimal set of fields that uniquely identify a record in a relation). • An index contains a collection of data entries, and supports efficient retrieval of all data entries k*with a given key value k. • Given data entry k*, we can find record with key k in at most one disk I/O. (Details soon …)

22. Alternatives for Data Entry k*in Index • In a data entry k* we can store: • Data record with key value k, or • <k, rid of data record with search key value k>, or • <k, list of rids of data records with search key k> • Choice of alternative for data entries is orthogonal to the indexing technique used to locate data entries with a given key value k. • Examples of indexing techniques: B+ trees, hash-based structures • Typically, index contains auxiliary information that directs searches to the desired data entries

23. Alternatives for Data Entries (Contd.) • Alternative 1: • If this is used, index structure is a file organization for data records (instead of a Heap file or sorted file). • At most one index on a given collection of data records can use Alternative 1. (Otherwise, data records are duplicated, leading to redundant storage and potential inconsistency.) • If data records are very large, # of pages containing data entries is high. Implies size of auxiliary information in the index is also large, typically.

24. Alternatives for Data Entries (Contd.) • Alternatives 2 and 3: • Data entries typically much smaller than data records. So, better than Alternative 1 with large data records, especially if search keys are small. (Portion of index structure used to direct search, which depends on size of data entries, is much smaller than with Alternative 1.) • Alternative 3 more compact than Alternative 2, but leads to variable sized data entries even if search keys are of fixed length.

25. B+ Tree Indexes Non-leaf Pages Leaf Pages (Sorted by search key) • Leaf pages containdata entries, and are chained (prev & next) • Non-leaf pages have index entries; only used to direct searches: index entry P K P K P P K m 0 1 2 1 m 2

26. Example B+ Tree Note how data entries in leaf level are sorted Root 17 • Find 28*? 29*? All > 15* and < 30* • Insert/delete: Find data entry in leaf, then change it. Need to adjust parent sometimes. • And change sometimes bubbles up the tree Entries < 17 Entries > = 17 27 5 13 30 39* 2* 3* 5* 7* 8* 22* 24* 27* 29* 38* 33* 34* 14* 16*

27. Hash-Based Indexes • Good for equality selections. • Index is a collection of buckets. • Bucket = primary page plus zero or moreoverflow pages. • Buckets contain data entries. • Hashing functionh: h(r) = bucket in which (data entry for) record r belongs. h looks at the search key fields of r. • No need for “index entries” in this scheme.

28. Index Classification • Primary vs. secondary: If search key contains primary key, then called primary index. • Unique index: Search key contains a candidate key. • Clustered vs. unclustered: If order of data records is the same as, or `close to’, order of data entries, then called clustered index. • Alternative 1 implies clustered; in practice, clustered also implies Alternative 1 (since sorted files are rare). • A file can be clustered on at most one search key. • Cost of retrieving data records through index varies greatly based on whether index is clustered or not!

29. Clustered vs. Unclustered Index • Suppose that Alternative (2) is used for data entries, and that the data records are stored in a Heap file. • To build clustered index, first sort the Heap file (with some free space on each page for future inserts). • Overflow pages may be needed for inserts. (Thus, order of data recs is `close to’, but not identical to, the sort order.) Index entries UNCLUSTERED CLUSTERED direct search for data entries Data entries Data entries (Index File) (Data file) Data Records Data Records

30. Comparing Storage Techniques

31. Cost Model for Our Analysis We ignore CPU costs, for simplicity: • B: The number of data pages • R: Number of records per page • D: (Average) time to read or write disk page • Measuring number of page I/O’s ignores gains of pre-fetching a sequence of pages; thus, even I/O cost is only approximated. • Average-case analysis; based on several simplistic assumptions. • Good enough to show the overall trends!

32. Comparing File Organizations • Heap files (random order; insert at eof) • Sorted files, sorted on <age, sal> • Clustered B+ tree file, Alternative (1), search key <age, sal> • Heap file with unclustered B + tree index on search key <age, sal> • Heap file with unclustered hash index on search key <age, sal>

33. Operations to Compare • Scan: Fetch all records from disk • Equality search • Range selection • Insert a record • Delete a record

34. Assumptions in Our Analysis • Heap Files: • Equality selection on key; exactly one match. • Sorted Files: • Files compacted after deletions. • Indexes: • Alt (2), (3): data entry size = 10% size of record • Hash: No overflow buckets. • 80% page occupancy => File size = 1.25 data size • Tree: 67% occupancy (this is typical). • Implies file size = 1.5 data size

35. Assumptions (contd.) • Scans: • Leaf levels of a tree-index are chained. • Index data-entries plus actual file scanned for unclustered indexes. • Range searches: • We use tree indexes to restrict the set of data records fetched, but ignore hash indexes.

36. Cost of Operations • Several assumptions underlie these (rough) estimates!

37. Cost of Operations • Several assumptions underlie these (rough) estimates!

38. Common Indexing Structures:B+ Tree

39. Index Entries (Direct search) Data Entries ("Sequence set") B+ Tree: Most Widely Used Index • Insert/delete at log F N cost; keep tree height-balanced. (F = fanout, N = # leaf pages) • Minimum 50% occupancy (except for root). Each node contains d <= m <= 2d entries. The parameter d is called the order of the tree. • Supports equality and range-searches efficiently.

40. Example B+ Tree • Search begins at root, and key comparisons direct it to a leaf. • Search for 5*, 15*, all data entries >= 24* ... Root 30 13 17 24 39* 3* 5* 19* 20* 22* 24* 27* 38* 2* 7* 14* 16* 29* 33* 34* • Based on the search for 15*, we know it is not in the tree!

41. B+ Trees in Practice • Typical order: 100 • capacity is 200 • min 100 keys per node, except root) • Typical fill-factor: 67%. • average fanout = 133 • Typical capacities: • Height 4: 1334 = 312,900,700 records • Height 3: 1333 = 2,352,637 records • Can often hold top levels in buffer pool: • Level 1 = 1 page = 8 Kbytes • Level 2 = 133 pages = 1 Mbyte • Level 3 = 17,689 pages = 133 MBytes

42. Inserting a Data Entry into a B+ Tree • Find correct leaf L. • Put data entry onto L. • If L has enough space, done! • Else, must splitL (into L and a new node L2) • Redistribute entries evenly, copy upmiddle key. • Insert index entry pointing to L2 into parent of L. • This can happen recursively • To split index node, redistribute entries evenly, but push upmiddle key. (Contrast with leaf splits.) • Splits “grow” tree; root split increases height. • Tree growth: gets wider or one level taller at top.

43. Inserting 8* into Example B+ Tree Root 30 13 17 24 39* 3* 5* 19* 20* 22* 24* 27* 38* 2* 7* 14* 16* 29* 33* 34*

44. Entry to be inserted in parent node. (Note that 17 is pushed up and only 17 this with a leaf split.) 5 13 24 30 Inserting 8* into Example B+ Tree Entry to be inserted in parent node. • Observe how minimum occupancy is guaranteed in both leaf and index pg splits. • Note difference between copy-upand push-up; be sure you understand the reasons for this. (Note that 5 is s copied up and 5 continues to appear in the leaf.) 3* 5* 2* 7* 8* appears once in the index. Contrast

45. Example B+ Tree After Inserting 8* Root 17 24 5 13 30 39* 2* 3* 5* 7* 8* 19* 20* 22* 24* 27* 38* 29* 33* 34* 14* 16* • Notice that root was split, leading to increase in height. • In this example, we can avoid split by re-distributing entries; however, this is usually not done in practice.

46. Deleting a Data Entry from a B+ Tree • Start at root, find leaf L where entry belongs. • Remove the entry. • If L is at least half-full, done! • If L has only d-1 entries, • Try to re-distribute, borrowing from sibling (adjacent node with same parent as L). • If re-distribution fails, mergeL and sibling. • If merge occurred, must delete entry (pointing to L or sibling) from parent of L. • Merge could propagate to root, decreasing height.

47. Example Tree After (Inserting 8*, Then) Deleting 19* and 20* ... Root 17 24 5 13 30 39* 2* 3* 5* 7* 8* 19* 20* 22* 24* 27* 38* 29* 33* 34* 14* 16*

48. Example Tree After (Inserting 8*, Then) Deleting 19* and 20* ... • Deleting 19* is easy. • Deleting 20* is done with re-distribution. Notice how middle key is copied up. Root 17 27 5 13 30 39* 2* 3* 5* 7* 8* 22* 24* 27* 29* 38* 33* 34* 14* 16*

49. ... And Then Deleting 24* • Must merge. • Observe `toss’ of index entry (on right), and `pull down’ of index entry (below). 30 39* 22* 27* 38* 29* 33* 34* Root 5 13 17 30 3* 39* 2* 5* 7* 8* 22* 38* 27* 33* 34* 14* 16* 29*

50. Prefix Key Compression • Important to increase fan-out. (Why?) • Key values in index entries only `direct traffic’; can often compress them. • E.g., If we have adjacent index entries with search key values Dannon Yogurt, David Smith and Devarakonda Murthy, we can abbreviate DavidSmith to Dav. (The other keys can be compressed too ...) • In general, while compressing, must leave each index entry greater than every key value (in any subtree) to its left. • Insert/delete must be suitably modified.