1 / 56

Chapter 2 Information Retrieval

Chapter 2 Information Retrieval. Chapter2 in the textbook Sections: 2.1, 2.2 (2.2.1, 2.2.2 ), 2.3 (2.3.1, 2.3.2, 2.3.3), 2.4(2.4.1, 2,4,2). Modern Information Retrieval. Document representation Using keywords Relative weight of keywords Query representation Keywords

thai
Download Presentation

Chapter 2 Information Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 2Information Retrieval Chapter2 in the textbook Sections: 2.1, 2.2 (2.2.1, 2.2.2), 2.3 (2.3.1, 2.3.2, 2.3.3), 2.4(2.4.1, 2,4,2)

  2. Modern Information Retrieval • Document representation • Using keywords • Relative weight of keywords • Query representation • Keywords • Relative importance of keywords • Retrieval model • Similarity between document and query • Rank the documents • Performance evaluation of the retrieval process

  3. Document Representation Transforming a text document to a weighted list of keywords

  4. Stopwords Figure 2.2 A partial list of stopwords

  5. Sample Document Data Mining has emerged as one of the most exciting and dynamic fields in computing science. The driving force for data mining is the presence of petabyte-scale online archives that potentially contain valuable bits of information hidden in them. Commercial enterprises have been quick to recognize the value of this concept; consequently, within the span of a few years, the software market itself for data mining is expected to be in excess of $10 billion. Data mining refers to a family of techniques used to detect interesting nuggets of relationships/knowledge in data. While the theoretical underpinnings of the field have been around for quite some time (in the form of pattern recognition, statistics, data analysis and machine learning), the practice and use of these techniques have been largely ad-hoc. With the availability of large databases to store, manage and assimilate data, the new thrust of data mining lies at the intersection of database systems, artificial intelligence and algorithms that efficiently analyze data. The distributed nature of several databases, their size and the high complexity of many techniques present interesting computational challenges.

  6. List of words in d1 after deleting stopwords

  7. Stemming A given word may occur in a variety of syntactic forms • plurals • past tense • gerund forms (a noun derived from a verb) Example The word connect, may appear as • connector, connection, connections, connected, connecting, connects, preconnection, and postconnection.

  8. Stemming A stem is what is left after its affixes (prefixes and suffixes) are removed Suffixes • connector, connection, connections, connected, connecting, connects, Prefixes • preconnection, and postconnection. Stem • connect

  9. Porter’s Algorithm • Letters A, E, I, O, and U are vowels • A consonant in a word is a letter other than A, E, I, O, or U, with the exception of Y • The letter Y is a vowel if it is preceded by a consonant, otherwise it is a consonant • For example, Y in synopsis is a vowel, while in toy, it is a consonant • A consonant in the algorithm description is denoted by c, and a vowel by v

  10. Porter’s Algorithm • m is the measure of vcrepetition • m = 0 TR, EE, TREE, Y, BY • m = 1 TROUBLE, OATS, TREES, IVY • m = 2 TROUBLES, PRIVATE, OATEN, ORRERY • *S – the stem ends with S (Similarly for other letters) • *v* - the stem contains a vowel • *d – the stem ends with a double consonant (e.g., -TT) • *o – the stem ends cvc, where the seconds c is not W, X, or Y (e.g. -WIL)

  11. Porter’s algorithmStep 1 Step 1: plurals and past participles

  12. Porter’s algorithm - Step 2 Steps 2–4: straightforward stripping of suffixes

  13. Porter’s algorithmStep 3 Steps 2–4: straightforward stripping of suffixes

  14. Porter’s algorithmStep 4 Steps 2–4: straightforward stripping of suffixes

  15. Porter’s algorithmStep 5 Steps 5: tidying-up

  16. Example • generalizations • Step1: GENERALIZATION • Step2: GENERALIZE • Step3: GENERAL • Step4: GENER • OSCILLATORS • Step1: OSCILLATOR • Step2: OSCILLATE • Step4: OSCILL • Step5: OSCIL

  17. Porter’s algorithm Suffix stripping of a vocabulary of 10,000 words (http://www.tartarus.org/~martin/)

  18. Document Representation

  19. Term-Document Matrix • Term-document matrix (TDM) is a two-dimensional representation of a document collection. • Rows of the matrix represent various documents • Columns correspond to various index terms • Values in the matrix can be either the frequency or weight of the index term (identified by the column) in the document (identified by the row).

  20. Term-Document matrix

  21. Sparse Matrixes- triples

  22. Sparse Matrixes- Pairs

  23. Normalization • raw frequency values are not useful for a retrieval model • prefer normalized weights, usually between 0 and 1, for each term in a document • dividing all the keyword frequencies by the largest frequency in the document is a simple method of normalization:

  24. Normalized Term-Document Matrix

  25. Vector Representation of document d1 (word, frequency, normalized frequency)

  26. Retrieval models Retrieval models match query with documents to: • separate documents into relevantand non-relevant class • rank the documents according to the relevance

  27. Retrieval models • Boolean model • Vector space model (VSM) • Probabilistic models

  28. Boolean Retrieval Model

  29. Boolean Retrieval Model • One of the simplest and most efficient retrieval mechanisms • Based on set theory and Boolean algebra • Conventional numeric representations of false as 0 and true as 1 • Boolean model is interested only in the presence or absence of a term in a document • In the term-document matrix replace all the nonzero values with 1

  30. Boolean Term-document Matrix

  31. Example Document set • DocSet(K0) = {D1,D3,D5} • DocSet(K4)={D2,D3,D4,D6} Query • K0 and K4 • K0 or K4

  32. K0 or (not K3 and K5)

  33. Boolean Query • User Boolean queries are usually simple Boolean expressions • A Boolean query can be represented in a “disjunctive normal form” (DNF) • disjunction corresponds to or • conjunction refers to and • DNF consists of a disjunction of conjunctive Boolean expressions

  34. DNF form • K0 or (not K3 and K5) is in DNF • DNF query processing can be very efficient • If any one of the conjunctive expressions is true, the entire DNF will be true • Short-circuit the expression evaluation • Stop matching the expression with a document as soon as a conjunctive expression matches the document; label the document as relevant to the query

  35. Boolean ModelAdvantages • Simplicity and efficiency of implementation • Binary values can be stored using bits • reduced storage requirements • retrieval using bitwise operations is efficient • Boolean retrieval was adopted by many commercial bibliographic systems • Boolean queries are akin to database queries

  36. Boolean Model Disadvantages • A document is either relevant or non-relevant to the query • It is not possible to assign a degree of relevance • Complicated Boolean queries are difficult for users • Boolean queries retrieve too few or too many documents. • K0 and K4 retrieved only 1 out of 6 documents • K0 or K4 retrieved 5 out of a possible 6 documents

  37. Vector Space Model (VSM)

  38. Vector Space Model • Treats both the documents and queries as vectors • A weight based on the frequency in the document:

  39. Graphical representation of the VSM Model

  40. Computing the similarity

  41. Relevance Values and Ranking

  42. Variations of VSM • Variations of the normalized frequency • Inverse document frequency (idf) • N = no. of documents • nj = no. of documents containing jth term • Modified weights :

  43. Inverse Document Frequencies for Collection (normalized)

  44. TDM using idf

  45. VSM vs. Boolean • Queries are easier to express: allow users to attach relative weights to terms • A descriptive query can be transformed to a query vector similar to documents • Matching between a query and a document is not precise: document is allocated a degree of similarity • Documents are ranked based on their similarity scores instead of relevant/non-relevant classes • Users can go through the ranked list until their information needs are met.

  46. Evaluation of Retrieval Performance

  47. Evaluation of Retrieval Performance Evaluation should include: • Functionality • Response time • Storage requirement • Accuracy

  48. Accuracy Testing Early days: • Batch testing • Document collection such as cacm.all • Query collection such as query.text Present day: interactive tests are used • Difficult to conduct and time consuming • Batch testing still important

More Related