Today’s topics

1. Today’s topics • Wild-card queries • Positional/phrase/proximity queries • Index size • Index construction • time and strategies • Dynamic indices - updating • Additional indexing challenges from the real world

2. Wild-card and phrase queries

3. Wild-card queries: * • mon*: find all docs containing any word beginning “mon”. • Easy with binary tree (or B-Tree) lexicon: retrieve all words in range: mon ≤ w < moo • *mon: find words ending in “mon”: harder • Permuterm index: for word hello index under: • hello$, ello$h, llo$he, lo$hel, o$hell • Queries: • X lookup on X$ X* lookup on X*$ • *X lookup on X$**X* lookup on X* • X*Y lookup on Y$X*X*Y*Z??? Exercise!

4. Wild-card queries • Permuterm problem: ≈quadruples lexicon size • Another way: index all k-grams occurring in any word (any sequence of k chars) • e.g., from text “April is the cruelest month” we get the 2-grams (bigrams) • $ is a special word boundary symbol $a,ap,pr,ri,il,l$,$i,is,s$,$t,th,he,e$,$c,cr,ru, ue,el,le,es,st,t$, $m,mo,on,nt,h$

5. Processing n-gram wild-cards • Query mon* can now be run as • $m AND mo AND on • Fast, space efficient • But we’d get a match on moon. • Must post-filter these results against query. • Further wild-card refinements • Cut down on pointers by using blocks • Wild-card queries tend to have few bigrams • keep postings on disk • Exercise: given a trigram index, how do you process an arbitrary wild-card query?

6. Phrase search • Search for “to be or not to be” • No longer suffices to store only <term:docs> entries • But could just do this anyway, and then post-filter [i.e., grep] for phrase matches • Viable if phrase matches are uncommon • Alternatively, store, for each term, entries • <number of docs containing term; • doc1: position1, position2 … ; • doc2: position1, position2 … ; • etc.>

7. Positional index example • Can compress position values/offsets as we did with docs in the last lecture • Nevertheless, this expands postings list in size substantially <be: 993427; 1: 7, 18, 33, 72, 86, 231; 2: 3, 149; 4: 17, 191, 291, 430, 434; 5: 363, 367, …> Which of these docs could contain “to be or not to be”?

8. Processing a phrase query • Extract inverted index entries for each distinct term: to, be, or, not • Merge their doc:position lists to enumerate all positions where “to be or not to be” begins. • to: • 2:1,17,74,222,551;4:8,27,101,429,433;7:13,23,191; ... • be: • 1:17,19; 4:17,191,291,430,434;5:14,19,101; ... • Same general method for proximity searches

9. Example: WestLaw http://www.westlaw.com/ • Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992) • About 7 terabytes of data; 700,000 users • Majority of users still use boolean queries • Example query: • What is the statute of limitations in cases involving the federal tort claims act? • LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM • Long, precise queries; proximity operators; incrementally developed; not like web search

10. Index size and construction

11. Index size • In an earlier lecture we studied the space for postings • also considered stemming, case folding, stop words ... • Another lecture covered the dictionary • What is the effect of stemming, etc. on index size?

12. Index size • Stemming/case folding cut • number of terms by ~40% • number of pointers by 10-20% • total space by ~30% • Stop words • Rule of 30: ~30 words account for ~30% of all term occurrences in written text • Eliminating 150 commonest terms from indexing will cut almost 25% of space

13. Postings Positional postings 1000 1 1 100,000 1 100 Positional index size • Need an entry for each occurrence, not just once per document • Index size depends on average document size • Average web page has <1000 terms • SEC filings, books, even some epic poems … easily 100,000 terms • Consider a term with frequency 0.1% Why? Document size

14. Rules of thumb • Positional index size factor of 2-4 over non-positional index • Positional index size 35-50% of volume of original text • Caveat: all of this holds for “English-like” languages

15. Index construction • Thus far, considered index space • What about index construction time? • What strategies can we use with limited main memory?

16. Somewhat bigger corpus • Number of docs = n = 40M • Number of terms = m = 1M • Use Zipf to estimate number of postings entries: • n + n/2 + n/3 + …. + n/m ~ n ln m = 560M entries • No positional info yet Check for yourself

17. Recall index construction • Documents are parsed to extract words and these are saved with the Document ID. Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious

18. After all documents have been parsed the inverted file is sorted by terms Key step

19. Index construction • As we build up the index, cannot exploit compression tricks • parse docs one at a time, final postings entry for any term incomplete until the end • (actually you can exploit compression, but this becomes a lot more complex) • At 10-12 bytes per postings entry, demands several temporary gigabytes

20. System parameters for design • Disk seek ~ 1 millisecond • Block transfer from disk ~ 1 microsecond per byte (following a seek) • All other ops ~ 10 microseconds

21. Bottleneck • Parse and build postings entries one doc at a time • To now turn this into a term-wise view, must sort postings entries by term (then by doc within each term) • Doing this with random disk seeks would be too slow If every comparison took 1 disk seek, and n items could be sorted with nlog2n comparisons, how long would this take?

22. Sorting with fewer disk seeks • 12-byte (4+4+4) records (term, doc, freq). • These are generated as we parse docs. • Must now sort 560M such 12-byte records by term. • Define a Block = 10M such records • can “easily” fit a couple into memory. • Will sort within blocks first, then merge the blocks into one long sorted order.

23. Sorting 56 blocks of 10M records • First, read each block and sort within: • Quicksort takes about 2 x (10M ln 10M) steps • Exercise: estimate total time to read each block from disk and and quicksort it. • 56 times this estimate - gives us 56 sorted runs of 10M records each. • Need 2 copies of data on disk, throughout.

24. 1 3 2 4 Merging 56 sorted runs • Merge tree of log256 ~ 6 layers. • During each layer, read into memory runs in blocks of 10M, merge, write back. 2 1 3 4 Disk

25. Merging 56 runs • Time estimate for disk transfer: • 6 x (56 x 120M x 10-6) x 2 ~ 22 hours. disk block transfer time Work out how these transfers are staged, and the total time for merging.

26. Exercise - fill in this table Step Time 1 56 initial quicksorts of 10M records each 2 Read 2 sorted blocks for merging, write back 3 Merge 2 sorted blocks ? 4 Add (2) + (3) = time to read/merge/write 5 56 times (4) = total merge time

27. Large memory indexing • Suppose instead that we had 16GB of memory for the above indexing task. • Exercise: how much time to index? • Repeat with a couple of values of n, m. • In practice, spidering interlaced with indexing. • Spidering bottlenecked by WAN speed and many other factors - more on this later.

28. Improving on merge tree • Compressed temporary files • compress terms in temporary dictionary runs • Merge more than 2 runs at a time • maintain heap of candidates from each run …. 5 2 4 3 6 1 ….

29. Dynamic indexing • Docs come in over time • postings updates for terms already in dictionary • new terms added to dictionary • Docs get deleted

30. Simplest approach • Maintain “big” main index • New docs go into “small” auxiliary index • Search across both, merge results • Deletions • Invalidation bit-vector for deleted docs • Filter docs output on a search result by this invalidation bit-vector • Periodically, re-index into one main index

31. More complex approach • Fully dynamic updates • Only one index at all times • No big and small indices • Active management of a pool of space

32. Fully dynamic updates • Inserting a (variable-length) record • e.g., a typical postings entry • Maintain a pool of (say) 64KB chunks • Chunk header maintains metadata on records in chunk, and its free space Header Free space Record Record Record Record

33. Global tracking • In memory, maintain a global record address table that says, for each record, the chunk it’s in. • Define one chunk to be current. • Insertion • if current chunk has enough free space • extend record and update metadata. • else look in other chunks for enough space. • else open new chunk.

34. Changes to dictionary • New terms appear over time • cannot use a static perfect hash for dictionary • OK to use term character string w/pointers from postings as in lecture 2.

35. Index on disk vs. memory • Most retrieval systems keep the dictionary in memory and the postings on disk • Web search engines frequently keep both in memory • massive memory requirement • feasible for large web service installations, less so for standard usage where • query loads are lighter • users willing to wait 2 seconds for a response • More on this when discussing deployment models

36. Distributed indexing • Suppose we had several machines available to do the indexing • how do we exploit the parallelism • Two basic approaches • stripe by dictionary as index is built up • stripe by documents

37. Indexing in the real world • Typically, don’t have all documents sitting on a local file system • Documents need to be spidered • Could be dispersed over a WAN with varying connectivity • Must schedule distributed spiders/indexers • Could be (secure content) in • Databases • Content management applications • Email applications • http often not the most efficient way of fetching these documents - native API fetching

38. Indexing in the real world • Documents in a variety of formats • word processing formats (e.g., MS Word) • spreadsheets • presentations • publishing formats (e.g., pdf) • Generally handled using format-specific “filters” • convert format into text + meta-data • Documents in a variety of languages • automatically detect language(s) in a document • tokenization, stemming, are language-dependent

39. “Rich” documents • (How) Do we index images? • Researchers have devised Query Based on Image Content (QBIC) systems • “show me a picture similar to this orange circle” • watch for next week’s lecture on vector space retrieval • In practice, image search based on meta-data such as file name e.g., monalisa.jpg

40. Passage/sentence retrieval • Suppose we want to retrieve not an entire document matching a query, but only a passage/sentence - say, in a very long document • Can index passages/sentences as mini-documents • But then you lose the overall relevance assessment from the complete document - more on this as we study relevance ranking • More on this when discussing XML search

41. Digression: food for thought • What if a doc consisted of components • Each component has its own access control list. • Your search should get a doc only if your query meets one of its components that you have access to. • More generally: doc assembled from computations on components • e.g., in Lotus databases or in content management systems • Welcome to the real world … more later.

42. Ranking models in IR • Key idea: • We wish to return in order the documents most likely to be useful to the searcher • To do this, we want to know which documents best satisfy a query • An obvious idea is that if a document talks about a topic more then it is a better match • A query should then just specify terms that are relevant to the information need, without requiring that all of them must be present • Document relevant if it has a lot of the terms

43. Binary term presence matrices • Record whether a document contains a word: document is binary vector in {0,1}v • What we have mainly assumed so far • Idea: Query satisfaction = overlap measure:

44. Overlap matching • What are the problems with the overlap measure? • It doesn’t consider: • Term frequency in document • Term scarcity in collection (document mention frequency) • Length of documents • (And queries: score not normalized)

45. Overlap matching • One can normalize in various ways: • Jaccard coefficient: • Cosine measure: • What documents would score best using Jaccard against a typical query? • Does the cosine measure fix this problem?

46. Count term-document matrices • We haven’t considered frequency of a word • Count of a word in a document: • Bag of words model • Document is a vector in ℕv

47. Weighting term frequency: tf • What is the relative importance of • 0 vs. 1 occurrence of a term in a doc • 1 vs. 2 occurrences • 2 vs. 3 occurrences … • Unclear: but it seems that more is better, but a lot isn’t necessarily better than a few • Can just use raw score • Another option commonly used in practice:

48. Dot product matching • Match is dot product of query and document • [Note: 0 if orthogonal (no words in common)] • Rank by match • It still doesn’t consider: • Term scarcity in collection (document mention frequency) • Length of documents and queries • Not normalized

49. Weighting should depend on the term overall • Which of these tells you more about a doc? • 10 occurrences of hernia? • 10 occurrences of the? • Suggest looking at collection frequency (cf) • But document frequency (df) may be better: Word cf df try 10422 8760 insurance 10440 3997 • Document frequency weighting is only possible in known (static) collection.

50. tf x idf term weights • tf x idf measure combines: • term frequency (tf) • measure of term density in a doc • inverse document frequency (idf) • measure of informativeness of term: its rarity across the whole corpus • could just be raw count of number of documents the term occurs in (idfi = 1/dfi) • but by far the most commonly used version is: • See Kishore Papineni, NAACL 2, 2002 for theoretical justification