CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan

1. CS276Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture: Index construction

2. Ch. 4 Index construction • How do we construct an index? • What strategies can we use with limited main memory? • What about dynamic text collections?

3. Sec. 4.1 Recall hardware basics • Disk I/O is block-based: Reading and writing of entire blocks (as opposed to smaller chunks). • Block sizes: 8KB to 256 KB. • Servers used in IR systems now typically have several GB of main memory, sometimes tens of GB. • Available disk space is several (2–3) orders of magnitude larger. • Fault tolerance is very expensive: It’s much cheaper to use many regular machines rather than one fault tolerant machine.

5. Sec. 4.2 Recall: index construction • Documents are parsed to extract words and these are saved with the Document ID. Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious

6. Sec. 4.2 After all documents have been parsed, the inverted file is sorted by terms. Key step We focus on this sort step.

7. Sec. 4.2 Scaling index construction • In-memory index construction does not scale. • How can we construct an index for very large collections? • Taking into account. . .memory, disk, speed, etc.

8. Sec. 4.2 Sort-based index construction • As we build the index, we parse docs one at a time. • While building the index, we cannot easily exploit compression tricks (you can, but much more complex) • The final postings are incomplete until the end. • At 12 bytes per non-positional postings entry (term, doc, freq), demands a lot of space for large collections. • T = 100,000,000 in the case of Reuters • So … we can do this in memory in 2009, but typical collections are much larger. E.g. the New York Times provides an index of >150 years of newswire • We need to store intermediate results on disk.

9. Sec. 4.2 Use the same algorithm for disk? • multi-way merge-sort aka BSBI: Blocked sort-based Indexing

10. Multi-way Mergesort • Take Wikipedia in Italian, and compute word freq: • Few GBs  n  109 words • How do you proceed ?? • Tokenize into a sequence of strings • Sortthe strings • Create tuples < word, freq >

11. 1 2 8 10 7 9 13 19 1 Binary Merge-Sort Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Divide Conquer Combine 2 7 Merge is linear in the #items to be merged

12. Few key observations • Items = (short) strings = atomic... • Q(n log n) memory accesses (I/Os ??) • [5ms] * n log2 n ≈ 3years In practice it is a “faster”, why?

13. 4 15 2 10 13 19 1 5 7 9 1 2 5 7 9 10 13 19 3 8 3 4 6 8 11 12 15 17 12 17 6 11 1 2 5 10 7 9 13 19 3 4 8 15 6 11 12 17 1 2 3 4 5 6 7 8 9 10 11 12 13 15 17 19 19 12 7 15 4 8 3 13 11 9 6 1 5 2 10 17 Implicit Caching… 2 passes (one Read/one Write) = 2 * (N/B) I/Os Log2 (N/M) 2 passes (R/W) 2 passes (R/W) log2 N M N/M runs, each sorted in internal memory (no I/Os) — I/O-cost for binary merge-sort is ≈ 2 (N/B) log2 (N/M)

14. 1 2 4 7 9 10 13 19 3 5 6 8 11 12 15 17 A key inefficiency After few steps, every run is longer than B !!! Output Run 1, 2, 3 B B B Output Buffer Disk 1, 2, 3 4, ... B We are using only 3 pages But memory contains M/B pages ≈ 230/215 = 215

15. Multi-way Merge-Sort • Sort N items with main-memory M and disk-pages B: • Pass 1: Produce (N/M) sorted runs. • Pass i: merge X =M/B-1 runs  logX N/M passes Pg for run1 . . . . . . Pg for run 2 . . . Out Pg Pg for run X Disk Disk Main memory buffers of B items

16. Cost of Multi-way Merge-Sort • Number of passes = logX N/M  logM/B (N/M) • Total I/O-cost is Q( (N/B) logM/B N/M ) I/Os • In practice • M/B ≈ 105#passes =1 few mins Tuning depends on disk features • Large fan-out (M/B) decreases #passes • Compression would decrease the cost of a pass!

17. Keep attention... • If sorting needs to manage arbitrarily long strings Key observations: • Array A is an “array of pointers to objects” • For each object-to-object comparison A[i] vs A[j]: • 2 random accesses to 2 memory locations A[i] and A[j] • Q(n log n) random memory accesses (I/Os ??) Indirectsort Memory containing the strings A Again chaching helps, But it may be less effective than before

18. Sec. 4.3 SPIMI: Single-pass in-memory indexing • Key idea #1: Generate separate dictionaries for each block. • Key idea #2: Don’t sort. Accumulate postings in postings lists as they occur. • With these two ideas we can generate a complete inverted index for each block. • These separate indexes can then be merged into one big index. • Easy append with 1 file per posting (docID are increasing among blocks), otherwise emulate multi-way merging

19. Sec. 4.3 SPIMI-Invert Merging of blocks  BSBI (now dictionary-driven).

20. Sec. 4.3 SPIMI: Compression • Compression makes SPIMI even more efficient. • Compression of terms • Compression of postings • See next lectures

21. Sec. 4.4 Distributed indexing • For web-scale indexing: must use a distributed computing cluster • Individual machines are fault-prone • Can unpredictably slow down or fail • How do we exploit such a pool of machines?

22. Sec. 4.4 Google data centers • Google data centers mainly contain commodity machines. • Data centers are distributed around the world. • Estimate: Google installs 100,000 servers each quarter. • Based on expenditures of 200–250 million dollars per year • This is 10% of the computing capacity of the world !?!

23. Sec. 4.4 Distributed indexing • Maintain a master machine directing the indexing job – considered “safe”. • Break up indexing into sets of (parallel) tasks. • Master machine assigns tasks to idle machines

24. Sec. 4.4 Parallel tasks • We will use two sets of parallel tasks • Parsers • Inverters • Break the input document collection into splits • Each split is a subset of documents • Term-based partition • one machine handles a subrange of terms • Doc-based partition • one machine handles a subrange of documents

25. Sec. 4.4 Data flow: doc-based partitioning Master assign assign Postings Set1 Parser Inverter IL1 Set2 Parser Inverter IL2 Inverter splits ILk Parser Setk Query must be also distributed

26. Sec. 4.4 Data flow: term-based partitioning Master assign assign Postings Set1 Parser Inverter a-f g-p q-z a-f Set2 Parser a-f g-p q-z Inverter g-p Inverter splits q-z Parser a-f g-p q-z Setk Each query-term goes to one machine

27. Sec. 4.4 Parsers • Master assigns a split to an idle parser machine • Parser reads a document at a time and emits (term, doc) pairs • Parser writes pairs into j partitions • Each partition is for a range of terms’ first letters (e.g., a-f, g-p, q-z) – here j = 3.

28. Sec. 4.4 Inverters • An inverter collects all (term,doc) pairs (= postings) for one term-partition. • Sorts and writes to postings lists

29. Sec. 4.4 MapReduce • Index construction is an instance of MapReduce • a robust and conceptually simple framework for distributed computing • … without having to write code for the distribution part. • Google indexing system (ca. 2002) consists of a number of phases, each implemented in MapReduce.

30. Sec. 4.4 Data flow: term-based partitioning Master assign assign Postings Parser Inverter a-f g-p q-z a-f Parser a-f g-p q-z Inverter g-p Inverter splits q-z Parser a-f g-p q-z Map phase Reduce phase

31. Sec. 4.5 Dynamic indexing • Up to now, we have assumed static collections. • It was rare but now more frequently occurs that: • Documents come in over time and need to be inserted. • Documents are deleted and modified. • And this induces: • Postings updates for terms already in dictionary • New terms added/deleted to/from dictionary

32. Sec. 4.5 Simplest approach • Maintain “big” main index • New docs go into “small” auxiliary index • Search across both, and merge the results • Deletions • Invalidation bit-vector for deleted docs • Filter search results (i.e. docs) by the invalidation bit-vector • Periodically, re-index into one main index

33. Sec. 4.5 Issues with main and auxiliary indexes • Merges may touch a lot of stuff • Poor performance • Merging of the auxiliary index into the main index is efficient if we keep a separate file for each postings list [docIDs are greater]. • Merge is the same as a simple append. • But this needs a lot of files – inefficient for O/S. • Assumption: The index is one big file. • In reality: Use a scheme somewhere in between (e.g., split very large postings lists, collect postings lists of length 1 in one file etc.)

34. Sec. 4.5 Logarithmic merge • Maintain a series of indexes, each twice as large as the previous one. • Keep smallest (Z0) in memory • Larger ones (I0, I1, …) on disk • If Z0 gets too big (> n), write to disk as I0 or merge with I0 (if I0 already exists) as Z1 • Either write merge Z1 to disk as I1 (if no I1) or merge with I1 to form Z2 • etc.

35. Sec. 4.5 Logarithmic merge • Auxiliary and main index: index construction time is O(T2) as each posting is touched in each merge. • Logarithmic merge: Each posting is merged O(log T) times, so complexity is O(T log T) So logarithmic merge is much more efficient for index construction, but its query processing requires the merging of O(log T) indexes

36. Sec. 4.5 Dynamic indexing at search engines • Most search engines now support dynamic indexing • News items, blogs, new topical web pages • But (sometimes/typically) they also periodically reconstruct the index from scratch • Query processing is then switched to the new index, and the old index is then deleted

37. Sec. 4.5 Other sorts of indexes • Positional indexes • Same type of sorting problem … just larger • Character n-gram indexes: • For each n-gram, need pointers to all dictionary terms containing it – the “postings”. • Permuterm index: • Consider all cyclic rotations of T$, where T is any term