Introduction to Information Retrieval (IR)

1. Introduction to Information Retrieval (IR) Mark Craven craven@cs.wisc.edu craven@biostat.wisc.edu 5730 Medical Sciences Center

2. Documents and Corpora • document: a passage of free text or hypertext • Usenet posting • Web page • newswire story • MEDLINE abstract • journal article • corpus (pl. corpora): a collection of documents • MEDLINE • Reuters stories from 1999 • the Web

3. The Ad-Hoc Retrieval Problem • given: • a document collection (corpus) • an arbitrary query • do: • return a list of relevant documents • this is the problem addressed by Web search engines

4. query processor spider document processor Typical IR System inverted index

5. The Index and Inverse Index • index: a relation mapping each document to the set of keywords it is about • inverse index • where do these come from?

6. aardvark anteater hungry zebra Inverted Index index corpus

7. aardvark anteater hungry zebra A Simple Boolean Query • to answer query “hungry AND zebra”, get intersection of documents pointed to by “hungry” and documents pointed to by “zebra”

8. Other Things to Consider • How wan we search on phrases? • Should we treat these queries differently? • “a hungry zebra” • “the hungry zebra” • “hungry as a zebra” • If we query on “laugh zebra” should we return documents containing the following? • “laughing zebra” • “laughable zebra” • Boolean queries are too coarse - return too many or too few relevant documents.

9. aardvark anteater hungry zebra Handling Phrases 95 40 • store position information in the inverted index • to answer query “hungry zebra”, look for documents having “hungry” at position i and “zebra” at position i + 1 25 38 26

10. Handling Phrases • but this is a primitive notion of phrase • we might want “zebras that are hungry” to be considered a match to the phrase “hungry zebra” • this requires doing sentence analysis; determining parts of speech for words, etc.

11. Stop Words • Should we treat these queries differently? • “a hungry zebra” • “the hungry zebra” • “hungry as a zebra” • Some systems employ a list of stop words (a.k.a. function words) that are probably not informative for most searches. • a, an, the, that, this, of, by, with, to … • stop words in a query are ignored • but might be handled differently in phrases

12. Stop Words a able about above according accordingly across actually after afterwards again against all allow allows almost alone along already also although always am among amongst an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate are around as aside ask asking associated at available away awfully b be became because become becomes becoming been before beforehand behind being believe below beside besides best better between beyond both brief but by ...

13. A Special Purpose Stop List Bos taurus Botrytis cinerea C. elegans Chicken Goat Gorilla Guinea pig Hamster Human Mouse Pig Rat Spinach unknown gene cDNA DNA clone BAC PAC cosmid clone genomic sequence potentially degraded

14. Stemming • If we query on “laugh zebra” should we return documents containing the following? • “laughing zebra” • “laughable zebra” • Some systems perform stemming on words; truncating related words to a common stem. • laugh laugh- • laughs laugh- • laughing laugh- • laughed laugh-

15. Stemming • the Lovins stemmer • 260 suffix patterns • iterative longest match procedure (.*)SSES $1SS (.*[AEIOU].*)ED $1 • the Porter stemmer • about 60 patterns grouped into sets • apply patterns in each set before moving to next

16. Stemming • May be helpful • reduces vocabulary 10-50% • may increase recall • May not be helpful • for some queries, the sense of a word is important • stemming algorithms are heuristic; may conflate semantically different words (e.g. gall and gallery) • As with stop words, might want to handle stemming differently in phrases

17. d1 d2 q d3 The Vector Space Model • Boolean queries are too coarse - return too many or too few relevant documents. • Most IR systems are based on the vector space model

18. The Vector Space Model • documents/queries represented by vectors in a high-dimensional space • each dimension corresponds to a word in the vocabulary • most relevant documents are those whose vectors are closest to query vector

19. Vector Similarity • one way to determine vector similarity is the cosine measure: • if the vectors are normalized, we can simply take their dot product

20. Determining Word Weights • lots of heuristics • one well established one is TFIDF (term frequency, inverse document frequency) weighting • numerator includes , number of occurrences of word in document • denominator includes , total number of occurrences of in corpus

21. TFIDF: One Form (N = total number of words in the corpus)

22. The Probability Ranking Principle • most IR systems are based on the premise that ranking documents in order of decreasing probability is the right thing to do • assumes documents are independent • does wrong thing with duplicates • doesn’t promote diversity in returned documents