1 / 19

What Does the User Really Want ?

What Does the User Really Want ?. Relevance, Precision and Recall. Overview. What is relevance ? Define Precision Define Recall. Basic Web Retrieval Process. Results. Need. KeywordQuery. Relevance. Good results are relevant to the need ! BUT

aminia
Download Presentation

What Does the User Really Want ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What Does the User Really Want ? Relevance, Precision and Recall

  2. Overview • What is relevance ? • Define Precision • Define Recall

  3. Basic Web Retrieval Process Results Need KeywordQuery

  4. Relevance • Good results are relevant to the need ! • BUT • How do we know what the user needs and wants

  5. Three Definition of Relevance • Pages which match the user need • Expert centred • Pages which the user thinks match their need • User centred • Pages which match the user query • System centred

  6. Expert Centred • Agree a definition of what the query means • Trawl through results identifying each page returned as relevant or irrelevant • Disadvantage • Takes no account of users current knowledge or needs • static

  7. User Centred • Get user to trawl through pages judging them as relevant or irrelevant • Problem • If they knew the answer would they be looking ? • Can we get the same result twice ? • expensive

  8. System Centred • Assumes keywords adequately express need • Highly reproducible • Really assessing accuracy of indexing and keyword matching

  9. Formal evaluations • Almost always use an Expert Centred or System Centred Approach

  10. Formal Definition of relevance A relevant page is a page which meets the user’s need and can be recognised as such by the search result description provided by the search engine by a sufficiently expert user

  11. Precision

  12. Precision Simply Defined The proportion of documents returned which are actually relevant to the query or need

  13. Precision Issues • Needs to be averaged across a set of queries to be meaningful • Needs to take account of rank order in results set

  14. Recall

  15. Recall Simply Defined The proportion of available relevant documents (anywhere on the web) which are actually returned in the results set

  16. Retrieved versus Relevant Docs High Recall Retrieval Retrieved Relevant High Precision Retrieval

  17. Recall Problems • Need to find all the pages on the web relevant to the need • Theoretically get a team of experts to assess every page on the web for relevance • In practice the results of a high recall search of a fixed web trawl is assessed

  18. Different sorts of retrieval • Today’s Sunderland result • High precision • Legal Cases • Has anyone ever been acquitted of murder and subsequently used their presence at the murder as an alibi ? • High recall

  19. Key Points • Definition of Relevance • and its problems • Definition of Precision • Definition of Recall

More Related