1 / 31

Resource Selection in Distributed Information Retrieval – an Experimental Study

Resource Selection in Distributed Information Retrieval – an Experimental Study. Hans Friedrich Witschel (formerly) University of Leipzig (now) SAP Research CEC Karlsruhe. Overview. Motivation Problem definition Solutions to be explored Experimental setup Results Conclusions.

nuncio
Download Presentation

Resource Selection in Distributed Information Retrieval – an Experimental Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Resource Selection in Distributed Information Retrieval – an Experimental Study Hans Friedrich Witschel (formerly) University of Leipzig (now) SAP Research CEC Karlsruhe

  2. Overview • Motivation • Problem definition • Solutions to be explored • Experimental setup • Results • Conclusions

  3. Motivation

  4. combine harvester cattle crops tractor agricultural acres statics building project landscape anti-seismic design cubature treatment surgery radiation oncology diagnostic bone marrow urology client server servent p2p terms algorithm ranking Motivation Resource selection Whom could I ask about „information retrieval“??

  5. Motivation Resource selection • Reason for selecting only a subset of all available resources/peers: cost reduction • Distributed IR (DIR): time and load on databases • Peer-to-peer IR (P2PIR): amount of messages, we will concentrate on P2PIR here • Basic approach: treat peers/resources as giant documents, use existing (slightly modified) retrieval functions to rank them, visit top-ranked ones…

  6. Problem definition

  7. Problem definition Assumptions • Peers have profiles = lists of terms with weights (unigram language models) • Two options: • Represent peers by what they have → extract terms from a peer‘s shared documents • Represent peers by queries for which they provide relevant documents • Profiles have to be compact in order to reduce communication overhead • absolute size of profiles dictated by available (network) resources

  8. Problem definition Research questions • How much will profile pruning degrade the quality of resource selection? That is, how many terms can we prune from a profile and still have acceptable results? • What can be done to improve peer selection? • Improve queries → Query Expansion? • Improve profiles → Profile adaptation?

  9. Solutions to be explored

  10. Solutions to be explored Preliminaries • Profiles: • use CORI for weighting terms t in the collection of peer p, rank by P(t|p) • Compression: apply simple thresholding • Profile sizes: 10,20,40,80,160,320,640,unpruned • Global term weights (I component of CORI) • Use external reference corpus for estimating idf values • Local retrieval function at each peer: BM25 • Uses the same idf estimations as above=> document scores comparable across all peers=> can concentrate on resource selection process, results not blurred by result merging effects

  11. Solutions to be explored Baselines • Random: Rank peers in random order • By-size: rank peers by the number of documents they hold, independent of offered content • Base CORI: rank peers by the sum of CORI weights of terms contained in both the query and the peer‘s profile

  12. Solutions to be explored Query expansion • All methods use Local Context analysis • Input passages are taken from: • The web: top 10 results snippets returned by Yahoo! API for the query • Local documents: best 10 documents returned by highest-ranked peer (local pseudo feedback) • For comparison („upper QE baseline“): use global view on collection (global pseudo feedback)

  13. Solutions to be explored Profile adaptation • Idea: • Boost weight of term t in peer p‘s profile if p has successfully answered a query containing t • Aim: profile allows the peer to answer popular queries for which it has many relevant documents • Can be done using a query log • Extensions: collaborative tagging approach, allow user interaction etc. (hard to evaluate)

  14. Solutions to be explored Profile adaptation • Update formula for term i in profile of peer p • Dp = documents returned by p • Do = documents returned by all peers contacted • AVGRP = average relative precision (RP) over all peers the query has reached • Update is only executed if ratio > 1, i.e. if p‘s results are „better“ than the average • For evaluation purposes: split a query log into query and test set, use training set for updating profiles

  15. Experimental Setup

  16. Experimental Setup Simplifying it… • Evaluate distributed IR only, instead of running full P2PIR simulation • Decouple query routing from other aspects (overlay topology etc.) • Considerably reduces number of free parameters • Underlying assumption: a resource selection algorithm A that works better than algorithm B for DIR, will also be better for P2PIR (i.e. when only a subset of all resources is visible) • A DIR scenario corresponds to a fully connected P2P overlay (e.g. PlanetP)

  17. Experimental Setup Parameterising it… • DIR evaluation, but: use parameters typical of P2PIR settings: • Pruned profiles • >> 1000 Peers • Peer collections: small and semantically (relatively) homogeneous • All this as opposed to DIR

  18. Experimental Setup Applying it… • Basic evaluation procedure: • Obtain a ranking R of all peers w.r.t. query q • Visit the top 100 peers in the order implied by R • After visiting each peer: merge documents found so far into a ranking S, judge quality of R by the quality of S using e.g. relevance judgments for documents

  19. Experimental Setup Test collections • Digital library scenario: peers = topics • Ohsumed: medical abstracts, annotated with Medical Subject Headings (MeSHs) • GIRT: German sociology abstracts, annotated with terms from a thesaurus • For both collections, queries and relevance judgments are available • Individuals sharing publications: • Citeseer abstracts with peers = (co-)authors • Query log available, but no relevance judgments

  20. Experimental Setup Evaluation measures • Missing relevance judgements: introduce new measure relative precision (RP) • Idea: compare a given ranking D with ranking C of a reference retrieval system (here: centralised system) • Probability of relevance of a document estimated as inverse rank in reference ranking • RP@k = average probability of relevance among first k documents of ranking D C = [K,L,M,N,O,P] D = [L,M,O]

  21. Results

  22. Results Profile pruning, CiteSeer

  23. Results Profile pruning, GIRT

  24. Results Profile pruning, space savings

  25. Results Qualitative analysis

  26. Results Query expansion M=intervals where QE runs significantly better than baseline M‘=intervals where QE significantly worse

  27. Results Profile adaptation

  28. Results Profile adaptation, delayed updates

  29. Conclusions

  30. Conclusions • Profile pruning: • Pruning profiles hurts performance less than expected • Whether or not pruning to a predefined size hurts, does not necessarily depend on the original profile size • In the experiments, it was always safe to prune for (total) space savings of 90% • „Advanced“ techniques: • Query expansion: more often hurts than improves performance • Profile adaptation: • Stable improvement of over 10% among the first 15 peers visited • Especially high improvement for the highest ranked peer • delayed updates do not hurt effectiveness (weak locality)

  31. Questions?

More Related