LIS618 lecture 3

1. LIS618 lecture 3 Thomas Krichel 2003-02-13

2. Structure of talk • Document Preprocessing • Basic ingredients of query languages • Retrieval performance evaluation

3. document preprocessing • There are some operations that may be done to the documents before indexing • lexical analysis • stemming of words • elimination of stop words • selection of index terms • construction of term categorization structures we will look at those in turn • in many cases, document preprocessing is not well documented by the provider. • but searchers need to be aware of them…

4. lexical analysis • divides a stream of characters into a stream of words • seems easy enough but…. • should we keep numbers? • hyphens. compare "state-of-the-art" with "b-52" • removal of punctuation, but "333B.C." • casing. compare "bank" and "Bank"

5. stemming • in general, users search for the occurrence of a term irrespective of grammar • plural, gerund forms, past tense can be subject to stemming • important algorithm by Porter • evidence about the effect of stemming on information retrieval is mixed • stemming is relatively rare these days.

6. elimination of stop words • some words carry no meaning and should be eliminated • in fact any word that appears in 80% of all documents is pretty much useless, but • consider a searcher for "to be or not to be". • It is better to reduce the index weight of terms that appear very frequently

7. index term selection • some engines try to capture nouns only • some nouns that appear heavily together can be considered to be one index term, such as "computer science" • Dialog deals with this through phrase indexing. • Most web engines, however, index all words, and all of the individually

8. thesauri • a list of words and for each word, a list of related words • synonyms • broader terms • narrower terms • used • to provide a consistent vocabulary for indexing and searching • to assist users with locating terms for query formulation • allow users to broaden or narrow query

9. use of thesauri • Thesauri are limited to experimental systems, or some high-quality systems, see http://www.sosig.ac.uk for an example. • It can be confusing to users. • Frequently the relationship between terms in the query is badly served by the relationships in the thesaurus. Thus thesaurus expansion of an initial query (if performed automatically) can lead to bad results.

10. simple queries • single-word queries • one word only • Hopefully some word combinations are understood as one word, e.g. on-line • Context queries • phrase queries (be aware of stop words) • proximity queries, generalize phrase queries • Boolean queries

11. simple pattern queries • prefix queries (e.g. "anal" for analogy) • suffix queries (e.g. "oral" for choral) • substring (e.g. "al" for talk) • ranges (e.g. form "held" to "hero") • within a distance, usually Levenshtein distance (i.e. the minimum number of insertions, deletions, and replacements) of query term

12. regular expressions • come from UNIX computing • build form strings where certain characters are metacharacters. • example: "pro(blem)|(tein)s?" matches problem, problem, protein and proteins. • example: New .*y matches "New Jersey" and "New York City", and "New Delhy". • great variety of dialects, usually very powerful. • Extremely important in digital libraries.

13. structured queries • make use of document structures • simplest example is when the documents are database records, we can search for terms is a certain field only. • if there is sufficient structure to field contents, the field can be interpreted as meaning something different than the word it contains. example: dates

14. query protocols • There are some standard languages • Z39.50 queries • CCL, "common command language" is a development of Z39.50 • CD-RDx "compact disk read only data exchange" is supported by US government agencies such as CIA and NASA • SFQL "structure full text query language" built on SQL

15. http://openlib.org/home/krichel Thank you for your attention!

16. retrieval performance evaluation • "Recall" and "Precision" are two classic measures to measure the performance of information retrieval in a single query. • Both assume that there is an answer set of documents that contain the answer to the query. • Performance is optimal if • the database returns all the documents in the answer set • the database returns only documents in the answer set • Recall is the fraction of the relevant documents that the query result has captured. • Precision is the fraction of the retrieved documents that is relevant.

17. recall and precision curves • Assume that all the retrieved documents arrive at once and are being examined. • During that process, the user discover more and more relevant documents. Recall increases. • During the same process, at least eventually, there will be less and less useful document. Precision declines (usually). • This can be represented as a curve.

18. Example • Let the answer set be {0,1,2,3,4,5,6,7,8,9} and non-relevant documents represented by letters. • A query reveals the following result: 7,a,3,b,c,9,n,j,l,5,r,o,s,e,4. • For the first document, (recall, precision) is (10%,100%), for the third, (20%,66%), for the sixth (30%,50%), for the tenth (40%,40%) etc.

19. recall/precision curves • Such curves can be formed for each query. • An average curve, for each recall level, can be calculated for several queries. • Recall and precision levels can also be used to calculate two single-valued summaries. • average precision at seen document • R-precision

20. average precision at seen document • To find it, sum all the precision level for each new relevant document discovered by the user and divide by the total number of relevant documents for the query. • In our example, it is 0.57 • This measure favors retrieval methods that get the relevant documents to the top.

21. R-precision • a more ad-hoc measure. • Let R be the size of the answer set. • Take the first R results of the query. • Find the number of relevant documents • Divide by R. • In our example, the R-precision is .4. • An average can be calculated for a number of queries.

22. critique of recall & precision • Recall has to be estimated by an expert • Recall is very difficult to estimate in a large collection • They focus on one query only. No serious user works like this. • There are some other measures, but that is more for an advanced course in IR.