Information Retrieval

1. Information Retrieval

2. Topics • Information Retrieval (IR) • What is it • Inverted Indexes • Web search

3. Information Retrieval • Its not just searching for books in libraries anymore • Wikipedia definition: • Information retrieval (IR) is the area of study concerned with searching for documents, for information within documents, and for metadata about documents, as well as that of searching structured storage, relational databases, and the World Wide Web.

4. Databases Today – NoSQL, Big Data • Unstructured data • Data can be of any type, may have no format or sequence • cannot be represented by any type of schema • Web pages in HTML • Video, sound, images • Semi-structured • Data has certain structure, but not all items identical – NoSQL • Schema info may be mixed in with data values • Similar entities grouped together – may have different attributes • Self-describing data, e.g. XML, displayed as graph

5. DBs vs. IR

6. Some of the slides are from Information Retrieval with MapReduce Jimmy Lin The iSchool

7. Searching Documents:How do we represent text? • Remember: computers don’t “understand” anything! • “Bag of words” • Treat all the words in a document as index terms for that document • Assign a “weight” to each term based on “importance” • Disregard order, structure, meaning, etc. of the words • Simple, yet effective! • Assumptions • Term occurrence is independent • Document relevance is independent • “Words” are well-defined

8. What’s a word? 天主教教宗若望保祿二世因感冒再度住進醫院。這是他今年第二度因同樣的病因住院。 وقال مارك ريجيف - الناطق باسم الخارجية الإسرائيلية - إن شارون قبل الدعوة وسيقوم للمرة الأولى بزيارة تونس، التي كانت لفترة طويلة المقر الرسمي لمنظمة التحرير الفلسطينية بعد خروجها من لبنان عام 1982. Выступая в Мещанском суде Москвы экс-глава ЮКОСа заявил не совершал ничего противозаконного, в чем обвиняет его генпрокуратура России. भारत सरकार ने आर्थिक सर्वेक्षण में वित्तीय वर्ष 2005-06 में सात फ़ीसदी विकास दर हासिल करने का आकलन किया है और कर सुधार पर ज़ोर दिया है 日米連合で台頭中国に対処…アーミテージ前副長官提言 조재영기자= 서울시는 25일이명박시장이 `행정중심복합도시'' 건설안에대해 `군대라도동원해막고싶은심정''이라고말했다는일부언론의보도를부인했다.

9. Sample Document “Bag of Words” McDonald's slims down spuds Fast-food chain to reduce certain types of fat in its french fries with new cooking oil. NEW YORK (CNN/Money) - McDonald's Corp. is cutting the amount of "bad" fat in its french fries nearly in half, the fast-food chain said Tuesday as it moves to make all its fried menu items healthier. But does that mean the popular shoestring fries won't taste the same? The company says no. "It's a win-win for our customers because they are getting the same great french-fry taste along with an even healthier nutrition profile," said Mike Roberts, president of McDonald's USA. But others are not so sure. McDonald's will not specifically discuss the kind of oil it plans to use, but at least one nutrition expert says playing with the formula could mean a different taste. Shares of Oak Brook, Ill.-based McDonald's (MCD: down $0.54 to $23.22, Research, Estimates) were lower Tuesday afternoon. It was unclear Tuesday whether competitors Burger King and Wendy's International (WEN: down $0.80 to $34.91, Research, Estimates) would follow suit. Neither company could immediately be reached for comment. … 16 × said 14 × McDonalds 12 × fat 11 × fries 8 × new 6 × company, french, nutrition 5 × food, oil, percent, reduce, taste, Tuesday …

10. Text Preprocessing • Commonly used words expected to occur in 80% or more of documents in a collection • Most frequent 50 account for 40% of all text

11. Word Frequency in English Frequency of 50 most common words in English (sample of 19 million words)

12. Text Preprocessing - Stop Words • First 6 most frequent words account for 20% of all words • Many frequent words do not contribute much to relevance, so useless and remove them • Stop list (there is more than one such list): • http://www.lextek.com/manuals/onix/stopwords1.html

13. Text Preprocessing - Stemming • Stem – word obtained after trimming suffix and prefix • ‘compute’ is stem for ‘computing’, ‘computer’, ‘computation’ • Stemming algorithms to reduce any word to its stem

14. aid 0 1 all 0 1 back 1 0 brown 1 0 come 0 1 dog 1 0 fox 1 0 good 0 1 jump 1 0 lazy 1 0 men 0 1 now 0 1 over 1 0 party 0 1 quick 1 0 their 0 1 time 0 1 Representing Documents Document 1 Term Document 1 Document 2 The quick brown fox jumped over the lazy dog’s back. Stopword List for is of Document 2 the to Now is the time for all good men to come to the aid of their party.

15. Inverted Index • Inverted indexing is fundamental to all IR models, also used sometimes in DBs • Consists of postings lists, one with each term in the collection • Posting list – document id (d) and payload (p) • Payload can be term frequency or number of times occurs on document, position of occurrence, properties, etc. • Can be ordered by document id, page rank, etc. • Data structure necessary to map from document id to e.g. URL

16. Term Doc 2 Doc 3 Doc 4 Doc 1 Doc 5 Doc 6 Doc 7 Doc 8 aid 0 0 0 1 0 0 0 1 all 0 1 0 1 0 1 0 0 back 1 0 1 0 0 0 1 0 brown 1 0 1 0 1 0 1 0 come 0 1 0 1 0 1 0 1 dog 0 0 1 0 1 0 0 0 fox 0 0 1 0 1 0 1 0 good 0 1 0 1 0 1 0 1 jump 0 0 1 0 0 0 0 0 lazy 1 0 1 0 1 0 1 0 men 0 1 0 1 0 0 0 1 now 0 1 0 0 0 1 0 1 over 1 0 1 0 1 0 1 1 party 0 0 0 0 0 1 0 1 quick 1 0 1 0 0 0 0 0 their 1 0 0 0 1 0 1 0 time 0 1 0 1 0 1 0 0 Inverted Index Term Postings aid 4 8 all 2 4 6 back 1 3 7 brown 1 3 5 7 come 2 4 6 8 dog 3 5 fox 3 5 7 good 2 4 6 8 jump 3 lazy 1 3 5 7 men 2 4 8 now 2 6 8 over 1 3 5 7 8 party 6 8 quick 1 3 their 1 5 7 time 2 4 6

18. Process query - retrieval • Given a query: • fetch posting lists associated with query • traverse postings to compute result set • Top k documents extracted

19. In order to process the query • Construct inverted index (indexing) • Gather web content (crawling) • Ranking documents given a query (retrieval)

20. Crawling, Indexing, Querying • Crawling and indexing • share similar characteristics and requirements • Both are offline problems, no need for real-time • Tolerable for a few minutes delay before content searchable • OK to run smaller-scale index updates frequently • Querying • online problem • Demands sub-second response time • Low latency high throughput • Loads can very greatly

21. Architecture of IR Systems Documents Query online offline Representation Function Representation Function Query Representation Document Representation Index Comparison Function Hits

22. Web Crawler • To acquire the document collection over which indexes are built • Copies the document • Acquiring web content requires crawling • Traverse web by repeatedly following hyperlinks and storing downloaded pages • Start by populating a queue with seed pages

23. Web Crawler Issues • Shouldn’t overload web servers • Prioritize order in which unvisited pages downloaded • Avoid downloading page multiple times – coordination and load balancing • Robust when failures • Learn update patterns so content current • Identify near duplicates and select best for index • Identify dominant language on page

24. How much of the web is crawled? • Deep web - Tor • Google, Bing, Chrome do not go there

25. Indexing – Non-deep Web • Creating inverted index • Must be relatively fast, but need not be real time • For Web, incremental updates are important • How large is the inverted index?

26. Postings Size: Zipf’s Law • George Kingsley Zipf (1902-1950) observed the following relation between frequency and rank • In other words: • A few elements occur very frequently • Many elements occur very infrequently • Zipfian distributions: • English words • Library book checkout patterns • DB queries • Website popularity (almost anything on the Web) f = frequency r = rank c = constant or

27. Zipf

28. Vocabulary Size: Heaps’ Law Heaps' law means that as more instance text is gathered, there will be diminishing returns in terms of discovery of the full vocabulary from which the distinct terms are drawn. Typically, K is between 10 and 100,  is between 0.4 and 0.6 V is vocabulary size n is corpus size (number of documents) K and  are constants, determine empirically When adding new documents, the system is likely to have seen most terms already… but the postings keep growing

29. Inverted Index • How large is the inverted index? • Size of vocabulary • Size of postings • Well-optimized inverted index can be 1/10 of size of original document collection • Fundamentally, a large sorting problem • Terms usually fit in memory • Postings usually don’t

30. Process query • Given a query, fetch posting lists associated with query, traverse postings to compute result set • Optimization strategies to reduce # postings must examine • Query document scores must be computed • Partial scores stored in accumulators • Top k documents extracted

31. Retrieval • The retrieval problem • Must have sub-second response • For Web, only need relatively few results • Types of queries in IR systems • Keyword Queries • Boolean Queries • Phrase Queries • Proximity Queries • Wildcard Queries • Natural Language Queries

32. Ranked Retrieval • Order documents by how likely they are to be relevant to the information need • How do we estimate relevance? • What characteristics to use? • Google’s Page Rank Algorithm