Index Construction: sorting

1. Index Construction:sorting Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading Chap 4

2. Indexer steps • Dictionary & postings: How do we construct them? • Scan the texts • Find the proper tokens-list • Append to the proper tokens-list the docID • How do we: • Find ? …time issue… • Append ? …space issues … • Postings’ size? • Dictionary size ? … in-memory issues …

3. Indexer steps: create token • Sequence of pairs: • < Modified token, Document ID > • What about var-length strings? Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious

4. Indexer steps: Sort • Sort by docIDs Core indexing step

5. Sec. 1.2 Indexer steps: Dictionary & Postings • Multiple term entries in a single document are merged. • Split into Dictionary and Postings • Doc. frequency information is added.

6. Sec. 1.2 Some key issues… Lists of docIDs Terms and counts • Now: • How do we sort? • How much storage is needed ? Pointers

7. Keep attention on disk... • If sorting needs to manage strings Key observations: • Array A is an “array of pointers to objects” • For each object-to-object comparison A[i] vs A[j]: • 2 random accesses to 2 memory locations A[i] and A[j] • Q(n log n) random memory accesses (I/Os ??) You sort A But this is an indirect sort Memory containing the strings A Again chaching helps, But it may be less effective than before

8. 1 2 8 10 7 9 13 19 1 Binary Merge-Sort Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Divide Conquer Combine 2 7 Merge is linear in the #items to be merged

9. Few key observations • Items = (short) strings = atomic... • On english wikipedia, about 109 tokens to sort • Q(n log n) memory accesses (I/Os ??) • [5ms] * n log2 n ≈ 3years In practice it is a “faster”, why?

10. Use the same algorithm for disk? • multi-way merge-sort aka BSBI: Blocked sort-based Indexing • Mapping term  termID • to be kept in memory for constructing the pairs • Consumes memory to pairs’ creation • Needs two passes, unless you use hashing and thus some probability of collision.

11. 4 15 2 10 13 19 1 5 7 9 1 2 5 7 9 10 13 19 3 8 3 4 6 8 11 12 15 17 12 17 6 11 1 2 5 10 7 9 13 19 3 4 8 15 6 11 12 17 1 2 3 4 5 6 7 8 9 10 11 12 13 15 17 19 19 12 7 15 4 8 3 13 11 9 6 1 5 2 10 17 Implicit Caching… 2 passes (one Read/one Write) = 2 * (N/B) I/Os Log2 (N/M) 2 passes (R/W) 2 passes (R/W) log2 N M N/M runs, each sorted in internal memory (no I/Os) — I/O-cost for binary merge-sort is ≈ 2 (N/B) log2 (N/M)

12. 1 2 4 7 9 10 13 19 3 5 6 8 11 12 15 17 A key inefficiency After few steps, every run is longer than B !!! Output Run 1, 2, 3 B B B Output Buffer Disk 1, 2, 3 4, ... B We are using only 3 pages But memory contains M/B pages ≈ 230/215 = 215

13. Multi-way Merge-Sort • Sort N items with main-memory M and disk-pages B: • Pass 1: Produce (N/M) sorted runs. • Pass i: merge X =M/B-1 runs  logX N/M passes Pg for run1 . . . . . . Pg for run 2 . . . Out Pg Pg for run X Disk Disk Main memory buffers of B items

14. 1 2 5 10 7 9 13 19 1 2 5 7…. 3 4 6 8 11….. 1 2 3 4 5 6 7 8 9 10 11 12 13 15 17 19 How it works 2 passes (one Read/one Write) = 2 * (N/B) I/Os LogX (N/M) X 2 passes (R/W) X M M N/M runs, each sorted in internal memory = 2 (N/B) I/Os — I/O-cost for X-way merge is ≈ 2 (N/B) I/Os per level

15. Cost of Multi-way Merge-Sort • Number of passes = logX N/M  logM/B (N/M) • Total I/O-cost is Q( (N/B) logM/B N/M ) I/Os • In practice • M/B ≈ 105#passes =1 few mins Tuning depends on disk features • Large fan-out (M/B) decreases #passes • Compression would decrease the cost of a pass!

16. SPIMI: Single-pass in-memory indexing • Key idea #1: Generate separate dictionaries for each block (No need for term  termID) • Key idea #2: Don’t sort. Accumulate postings in postings lists as they occur (in internal memory). • Generate an inverted index for each block. • More space for postings available • Compression is possible • Merge indexes into one big index. • Easy append with 1 file per posting (docID are increasing within a block), otherwise multi-merge like

17. SPIMI-Invert

18. Distributed indexing • For web-scale indexing: must use a distributed computing cluster of PCs • Individual machines are fault-prone • Can unpredictably slow down or fail • How do we exploit such a pool of machines?

19. Google data centers • Google data centers mainly contain commodity machines. • Data centers are distributed around the world. • Estimate: Google installs 100,000 servers each quarter. • Based on expenditures of 200–250 million dollars per year

20. Distributed indexing • Maintain a master machine directing the indexing job – considered “safe”. • Break up indexing into sets of (parallel) tasks. • Master machine assigns tasks to idle machines • Other machines can play many roles during the computation

21. Parallel tasks • We will use two sets of parallel tasks • Parsers and Inverters • Break the document collection in two ways: • Term-based partition • one machine handles a subrange of terms • Doc-based partition • one machine handles a subrange of documents

22. Data flow: doc-based partitioning Master assign assign Postings Set1 Parser Inverter IL1 Set2 Parser Inverter IL2 Inverter splits ILk Parser Setk Each query-term goes to many machines

23. Data flow: term-based partitioning Master assign assign Postings Set1 Parser Inverter a-f g-p q-z a-f Set2 Parser a-f g-p q-z Inverter g-p Inverter splits q-z Parser a-f g-p q-z Setk Each query-term goes to one machine

24. MapReduce • This is • a robust and conceptually simple framework for distributed computing • … without having to write code for the distribution part. • Google indexing system (ca. 2002) consists of a number of phases, each implemented in MapReduce.

25. Guarantee fitting in one machine ? Guarantee fitting in one machine ? Data flow: term-based partitioning 16Mb -64Mb Master assign assign Postings Parser Inverter a-f g-p q-z a-f Parser a-f g-p q-z Inverter g-p Inverter splits q-z Parser a-f g-p q-z Segment files (local disks) Map phase Reduce phase

26. Dynamic indexing • Up to now, we have assumed static collections. • Now more frequently occurs that: • Documents come in over time • Documents are deleted and modified • And this induces: • Postings updates for terms already in dictionary • New terms added/deleted to/from dictionary

27. Simplest approach • Maintain “big” main index • New docs go into “small” auxiliary index • Search across both, and merge the results • Deletions • Invalidation bit-vector for deleted docs • Filter search results (i.e. docs) by the invalidation bit-vector • Periodically, re-index into one main index

28. Issues with 2 indexes • Poor performance • Merging of the auxiliary index into the main index is efficient if we keep a separate file for each postings list. • Merge is the same as a simple append [new docIDs are greater]. • But this needs a lot of files – inefficient for O/S. • Assumption: The index is one big file. • In reality: Use a scheme somewhere in between (e.g., split very large postings lists, collect postings lists of length 1 in one file etc.)

29. Logarithmic merge • Maintain a series of indexes, each twice as large as the previous one: 20 M, 21 M , 22 M , 23 M , … • Keep smallest (Z0) in memory (of size M) • Larger ones (I0, I1, …) on disk (of sizes 21 M , 22 M , …) • If Z0 gets too big (> M), write to disk as I0 or merge with I0 (if I0 already exists) as Z1 • Either write merge Z1 to disk as I1 (if no I1) or merge with I1 to form Z2 • etc. # indexes = log2 (T/M)

30. Some analysis (T = #postings = #tokens) • Auxiliary and main index: index construction time is O(T2) as each posting is touched in each merge. • Logarithmic merge: Each posting is merged O(log (T/M) ) times, so complexity is O( T log (T/M) ) Logarithmic merge is more efficient for index construction, but its query processing requires the merging of O( log (T/M) ) lists of results

31. Web search engines • Most search engines now support dynamic indexing • News items, blogs, new topical web pages • But (sometimes/typically) they also periodically reconstruct the index • Query processing is then switched to the new index, and the old index is then deleted

32. Query processing:optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3

33. Sec. 2.3 128 17 2 4 8 41 48 64 1 2 3 8 11 21 Augment postings with skip pointers (at indexing time) 128 41 • How do we deploy them ? • Where do we place them ? 31 11 31

34. Sec. 2.3 17 2 4 8 41 48 64 1 2 3 8 11 21 But the skip successor of 11 on the lower list is 31, so we can skip ahead past the intervening postings. Using skips 128 41 128 31 11 31 Suppose we’ve stepped through the lists until we process 8 on each list. We match it and advance. We then have 41 and 11 on the lower. 11 is smaller.

35. Sec. 2.3 Placing skips • Tradeoff: • More skips  shorter skip spans  more likely to skip. But lots of comparisons to skip pointers. • Fewer skips  few pointer comparison, but then long skip spans  few successful skips.

36. Sec. 2.3 Placing skips • Simple heuristic: for postings of length L, use L evenly-spaced skip pointers. • This ignores the distribution of query terms. • Easy if the index is relatively static; harder if L keeps changing because of updates. • This definitely used to help; with modern hardware it may not unless you’re memory-based • The I/O cost of loading a bigger postings list can outweigh the gains from quicker in memory merging!

37. Faster = caching? Two opposite approaches: • Cache the query results(exploits query locality) • Cache pages of posting lists(exploits term locality)

38. Query processing:phrase queries and positional indexes Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.4

39. Sec. 2.4 Phrase queries • Want to be able to answer queries such as “stanford university” – as a phrase • Thus the sentence “I went at Stanford my university” is not a match.

40. Sec. 2.4.1 Solution #1: Biword indexes • For example the text “Friends, Romans, Countrymen” would generate the biwords • friends romans • romans countrymen • Each of these biwords is now a dictionary term • Two-word phrase query-processing is immediate.

41. Sec. 2.4.1 Longer phrase queries • Longer phrases are processed by reducing them to bi-word queries in AND: • stanford university palo alto can be broken into the Boolean query on biwords: stanford university AND university palo AND palo alto Need the docs to verify + They are combined with other solutions Can have false positives! Index blows up

42. Sec. 2.4.2 Solution #2: Positional indexes • In the postings, store for each term and document the position(s) in which that term occurs: <term, number of docs containing term; doc1: position1, position2 … ; doc2: position1, position2 … ; etc.>

43. Sec. 2.4.2 Processing a phrase query • “to be or not to be”. • to: • 2:1,17,74,222,551;4:8,16,190,429,433;7:13,23,191; ... • be: • 1:17,19; 4:17,191,291,430,434;5:14,19,101; ... • Same general method for proximity searches

44. Sec. 7.2.2 Query term proximity • Free text queries: just a set of terms typed into the query box – common on the web • Users prefer docs in which query terms occur within close proximity of each other • Would like scoring function to take this into account – how?

45. Sec. 2.4.2 Positional index size • You can compress position values/offsets • Nevertheless, a positional index expands postings storage by a factor 2-4 in English • Nevertheless, a positional index is now commonly used because of the power and usefulness of phrase and proximity queries … whether used explicitly or implicitly in a ranking retrieval system.

46. Sec. 2.4.3 Combination schemes • BiWord + Positional index is a profitable combination • For particular phrases (“Michael Jackson”, “Britney Spears”) it is inefficient to keep on merging positional postings lists  better to store a bi-word • More complicated mixing strategies do exist!

47. Sec. 7.2.3 Soft-AND • E.g. query rising interest rates • Run the query as a phrase query • If <K docs contain the phrase rising interest rates, run the two phrase queries rising interest and interest rates • If we still have <K docs, run the “vector space query”rising interest rates • Rank matching docs

48. Query processing:other sophisticated queries Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 3.2 and 3.3

49. Sec. 3.2 Wild-card queries: * • mon*: find all docs containing words beginning with “mon”. • Use a Prefix-search data structure • *mon: find words ending in “mon” • Maintain a prefix-search data structure for reverse terms. How can we solve the wild-card query pro*cent?

50. Sec. 3.2 What about * in the middle? • co*tion • We could look up co* AND *tion and intersect the two lists • Expensive • se*ateANDfil*er This may result in many Boolean AND queries. • The solution: transform wild-card queries so that the *’s occur at the end • This gives rise to the Permuterm Index.