1 / 19

Relevance and Evaluation of IR Performance

Relevance and Evaluation of IR Performance. Dr. Bilal IS 530 Fall 2009. Searching for Information. Imprecise Incomplete Tentative Challenging. Why Some Items Are Not Retrieved?. Indexing errors Wrong search terms Wrong database Language variations Other (to be answered by students).

conley
Download Presentation

Relevance and Evaluation of IR Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Relevance and Evaluation of IR Performance Dr. Bilal IS 530 Fall 2009

  2. Searching for Information • Imprecise • Incomplete • Tentative • Challenging

  3. Why Some Items Are Not Retrieved? • Indexing errors • Wrong search terms • Wrong database • Language variations Other (to be answered by students)

  4. Boolean Operators • OR increases recall • AND increases precision • NOT increases precision by elimination

  5. Relevance • A match between a query and information retrieved • A judgment • Can be judged by anyone who is informed of the query and views the retrieved information

  6. Relevance • Judgment is dynamic • Documents can be ranked by likely relevance • In practice, not easy to measure • Not focused on user needs

  7. Pertinence • Based on information need rather than a match between a query and retrieved documents • Can only be judged by user • May differ from relevance judgment

  8. Pertinence • Transient, varies with many factors • Not often used in evaluation • May be used as a measure of satisfaction • User-based, as opposed to relevance

  9. Relevance Judgment • Users base it on: • Topicality • Aboutness • Utility • Novelty • Satisfaction

  10. IR Performance Recall Ratio = the number of relevant documents retrieved the total number of relevant documents

  11. Recall and Precision in Practice • Inversely related • Search strategies designed for high precision or high recall (or medium) • Needs of users dictate search strategy towards recall or precision • Practice helps changing queries to favor recall or precision

  12. Recall and Precision 1.0 Recall 1.0 Precision

  13. IR Performance Precision Ratio = the number of relevant documents retrieved the total number of documents retrieved

  14. High Precision Search • Use these strategies, as appropriate: • Controlled vocabulary • Limit feature (e.g., specific fields, major descriptors, date(s), language, as appropriate) • AND operator • Proximity operators carefully • Truncation carefully

  15. High Recall Search • Use these strategies, as appropriate: • OR logic • Keyword searching • No or minimal limit to specific field(s) • Truncate • Broader terms

  16. Improving IR Performance • Good mediation of search topic before searching • User presence during search, if possible • Preliminary search judged by user • Evaluation during search (by searcher or by searcher and user)

  17. Improving IR Performance • Refinement of search strategies • Searcher to evaluate final results • User to evaluate final results

  18. Improving IR Performance • Better system design • Better indexing and word parsing • Better structure of thesauri • Better user interface (e.g., more effective help feature) • Better error recovery feedback • User-centered design

  19. Relevance in Information Science • Dr. Tefko Saracevic talk at SIS on: “Relevance in information science” • To watch the streaming video and the slide show, click on • http://mediabeast.ites.utk.edu/mediasite4/Viewer/?peid=fb8f84cb-9f82-499f-b12c-9a56ab5cf5ba

More Related