Evaluation Measures in Information Retrieval: Enhancing System Performance

1. Modern Information Retrieval Chapter 3 Retrieval Evaluation

2. The most common measures of system performance are time and space • an inherent tradeoff • Data retrieval • time and space • indexing • Information retrieval • precision of the answer set also important

4. evaluation considerations • query with/without feedback • query interface design • real data/synthetic data • real life/laboratory environment • repeatability and scalability

5. recall and precision • recall: fraction of relevant documents which has been retrieved • precision: fraction of retrieved documents which is relevant

6. can we precisely compute precisions? can we precisely compute recalls?

7. precision versus recall curve: a standard evaluation strategy

9. interpolation procedure for generating the 11 standard recall levels • Rq={d3,d56,d129} where j is in {0,1,2,…,10} and P(r) is a known precision

11. to evaluate the retrieval strategy over all test queries, the precisions at each recall level are averaged

12. another approach: compute average precision at given relevant document cutoff values • advantages?

13. single value summary for each query • average precision at seen relevant documents • example in Figure 3.2 • favor systems which retrieve relevant documents quickly • can have a poor overall recall performance • R-precision • R: total number of relevant documents • examples in Figures 3.2 and 3.3

14. precision histogram

15. combining recall and precision • the harmonic mean • it assumes a high value only when both recall and precision are high

16. the E measure • b=1, complement of the harmonic mean • b>1, the user is more interested in precision • b<1, the user is more interested in recall

17. user-oriented measures

18. coverage ratio: fraction of the documents known to be relevant which has been retrieved • the system finds the relevant documents the user expected to see

19. novelty ratio: fraction of the relevant documents retrieved which was previously unknown to the user • the system reveals new relevant documents previously unknown to the user

20. relative recall: the ratio between the number of relevant documents found and the number of relevant documents the user expected to find • relative recall= • when the relative recall equals to 1 (the user finds enough relevant documents), the user stops searching

21. recall effort: the ratio between the number of relevant documents the user expected to find and the number of documents examined in an attempt to find the expected relevant documents • research in IR • lack a solid formal framework • lack robust and consistent testbeds and benchmarks • Text REtrieval Conference

22. retrieval techniques • methods using automatic thesauri • sophisticated term weighting • natural language techniques • relevance feedback • advanced pattern matching • document collection • over 1 million documents • newspaper, patents, etc. • topics • in natural language • conversion done by the system

23. relevant documents • the pooling method: for each topic, collect the top k documents generated by each participating system and decide their relevance by human assessors • the benchmark tasks • ad hoc task • filtering task • Chinese • cross languages • spoken document retrieval • high precision • very large collection

24. evaluation measures • summary table statistics: number of documents retrieved, number of relevant documents retrieved, number of relevant documents not retrieved, etc. • recall-precision averages • document level averages: average precision at seen relevant documents • average precision histogram