1 / 18

Evaluation of IR Performance

Evaluation of IR Performance. Dr. Bilal IS 530 Fall 2005. Searching for Information . Imprecise Incomplete Tentative Challenging. IR Performance . Precision Ratio = the number of relevant documents retrieved the total number of documents retrieved. IR Performancel. Recall Ratio =

weylin
Download Presentation

Evaluation of IR Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of IR Performance Dr. Bilal IS 530 Fall 2005

  2. Searching for Information • Imprecise • Incomplete • Tentative • Challenging

  3. IR Performance Precision Ratio = the number of relevant documents retrieved the total number of documents retrieved

  4. IR Performancel Recall Ratio = the number of relevant documents retrieved the total number of relevant documents

  5. Why Do We Miss Items? • Indexing errors • Wrong search terms • Wrong database • Language variations • Other???

  6. Why Do We Get Unwanted Items? • Indexing errors • Wrong search terms • Homographs • Incorrect term relations • Other???

  7. Boolean Operators • OR increases recall • AND increases precision • NOT increases precision

  8. Recall and Precision in Practice • Inversely related • Search strategies designed for high precision or high recall (or medium) • Needs of users dictate search strategy towards recall or precision • Practice helps changing queries to favor recall or precision

  9. Recall and Precision 1.0 Recall 1.0 Precision

  10. Problems with Relevance, Recall, and Precision • Yes or no decision • Things are more or less relevant • In practice not easy to measure • Not focused on user needs

  11. Relevance • A match between a query and information retrieved • Is a judgment • Can be judged by anyone who is informed of the query and views the retrieved information

  12. Relevance (cont.) • Judgments may differ • Is the base for information retrieval evaluation methods (recall and precision) • Documents can be ranked by likely relevance

  13. Pertinence • Based on information need rather than request and documents • Can only be judged by user • May differ from relevance judgments

  14. Pertinence (cont.) • Transient, varies with many factors • Not often used in evaluation • May be used as a measure of satisfaction

  15. High Precision Searching • Controlled vocabulary • Limits: Specific fields, major descriptors, Date, language, etc. • AND operator • Proximity • Careful with truncation

  16. High Recall Searching • OR logic • Keyword searching • No limits • Truncate • Broader terms

  17. Related Concepts • Topicality • Aboutness • Utility • Pertinence • Satisfaction

  18. Hints for Improving Performance • Good interview • User presence, if possible • Preliminary search and user response • Evaluation during search (you or you and user) • User feedback • Search refinement as you progress

More Related