1 / 50

Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web

Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web. MS Thesis Defense Manishkumar Jha Committee Members Dr. Subbarao Kambhampati Dr. Huan Liu Dr. Hasan Davulcu. Deep Web Integration Scenario.

emera
Download Presentation

Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Topic-Sensitive SourceRank: Extending SourceRank for Performing Context-Sensitive Search over Deep-Web MS Thesis Defense ManishkumarJha Committee Members Dr. SubbaraoKambhampati Dr. Huan Liu Dr. HasanDavulcu

  2. Deep Web Integration Scenario Millions of sources containing structured tuples Autonomous Uncontrolled collection Contains information spanning multiple topics Mediator Access is limited to query-forms ←answer tuples ←query query→ answer tuples→ answer tuples→ ←query answertuples→ ←answer tuples query→ ←query Web DB Web DB Web DB Web DB Web DB Web DB Web DB Deep Web

  3. Source quality and SourceRank Source quality SourceRank SourceRank[1] provides a measure for assessing source quality based on source trustworthiness and result importance • Deep-Web is adversarial • Source quality is a major issue over deep-web [1] SourceRank:Relevance and Trust Assessment for Deep Web Sources Based on Inter-Source Agreement, WWW, 2011

  4. … But Source quality is topic-sensitive • Sources might have data corresponding to multiple topics. Importance may vary across topics • Example: Barnes & Noble might be quite good as a book source but not be as good a movie source • SourceRank will fail to capture this fact • Issues were noted for surface-web. But are much more critical for deep-web as sources are even more likely to cross topics book movie

  5. Deep Web Integration Scenario Mediator ←answer tuples ` ←query query→ answer tuples→ answer tuples→ ←query answertuples→ ←answer tuples Movie query→ ←query Web DB Web DB Web DB Music Web DB Web DB Web DB Web DB Deep Web Camera Books

  6. Problem Definition Problem Definition: Performing effective multi-topic source selection sensitive to trustworthiness for deep-web

  7. Our solution – Topic sensitive-SourceRank • Compute multiple topic-sensitive SourceRanks • At query-time, using query-topic combine these rankings into composite importance ranking • Challenges • Computing topic-sensitive SourceRanks • Identifying query-topic • Combining topic-sensitive SourceRanks

  8. Agenda • SourceRank • Topic-sensitive SourceRank • Experiments and Results • Conclusion

  9. Agenda • SourceRank • Topic-sensitive SourceRank • Experiments and Results • Conclusion

  10. SourceRank Computation • Assesses source quality based on trustworthiness and result importance • Introduces a domain-agnostic agreement based technique for implicitly creating an endorsement structure between deep-web sources • Agreement of answer sets returned in response to same queries manifests as a form of implicit endorsement

  11. SourceRank Computation contd. Endorsement is modeled as directed weighted agreement graph Nodes represent sources Edge weights represent agreement between the sources SourceRank of a source is computed as the stationary visit probability of a Markov random walk performed on this agreement graph

  12. Agenda • SourceRank • Topic-sensitive SourceRank • Experiments and Results • Conclusion

  13. Trust-based measure for multi-topic deep-web • Issues with SourceRank for multi-topic deep-web • Single importance ranking • Is query-agnostic • We propose Topic-sensitive SourceRank, TSR for effectively performing multi-topic selection sensitive to trustworthiness • TSR overcomes the drawbacks of SourceRank

  14. Topic-sensitive SourceRankOverview • Instead of creating a single importance ranking, multiple importance rankings are created • Each importance ranking is biased towards a particular topic • At query-time, using query information and individual topic-specific importance rankings, compute a composite importance ranking biased towards the query

  15. Challenges for TSR • Computing topic-specific importance rankings is not trivial • Inferring query information • Identifying query-topic • Computing composite importance ranking

  16. Computing topic-specific SourceRank • For a deep-web source, its SourceRank score for a topic, will depend on the answers to queries of same topic • Using topic-specific sampling queries for a topic, will result in an endorsement structure, biased towards the same topic • Example: If movie-related sampling queries are used, then movie sources are more likely to agree on the answer sets than other topic sources. This will result in the endorsement structure biased towards the movie-topic

  17. Computing topic-specific SourceRank contd. • SourceRank computed on biased agreement graph for a topic will capture topic-specific source importance ranking for the same topic

  18. Topic-specific sampling queries • Publicly available online directories such as ODP, Yahoo Directory provide hand-constructed topic hierarchies • These directories along with the links posted under each topic are a good source for obtaining topic-specific sampling queries

  19. Computing Topic-specific SourceRanks • Partial topic-specific sampling queries are used for obtaining source crawls • Topic-specific source crawls are used for computing biased agreement graphs • Topic-specific SourceRanks, TSR’s are obtained by performing a weighted random walk on the biased agreement graphs

  20. Query Processing • Query processing involves • Computing query-topic • Computing query-topic sensitive importance scores • Source selection

  21. Computing query-topic • Query-topic • Likelihood of the query belonging to topics • Soft classification problem: For user query q and a set of topics ci C, goal is to find fractional topic membership of q with each topic ci topic query-topic For Query=“godfather”

  22. Computing query-topic – Training Data • Training data • Description of topics • Use complete topic-specific sampling queries to obtain topic-specific source crawls • Topic descriptions are treated as bag of words

  23. Computing query-topic – Classifier • Classifier • Naïve Bayes Classifier (NBC) is used with parameters set to maximum likelihood estimates • For a user query q, NBC uses topic-description to estimate topic probability conditioned on q i.e. for topic ci, NBC uses topic-description for cito estimate P(ci|q)

  24. Computing query-topic – Classifier contd. Computing P(ci|q) where qjis the jth term of query q P(ci) can be set based on domain knowledge, but for our computations, we use uniform probabilities for topics

  25. Computing query-topic sensitive importance scores • Query-topic sensitive importance scores • Topic-specific SourceRanks are linearly combined using query-topic as weights • Query-topic sensitive or composite SourceRank score for source skis computed as where TSRki is the topic-specific SourceRank score of source sk for topic ci

  26. Source selection • Linearly combines relevance-scores with importance scores • Overall score of a source sk is computed as where Rk: relevancy score of sk CSRk: query-topic sensitive score of sk

  27. Agenda • SourceRank • Topic-sensitive SourceRank • Experiments and Results • Conclusion

  28. Experimental setup • Experiments were conducted on a multi-topic deep-web environment consisting of four-representative topics – camera, book, movie and music • Source DataSet • Sources were collected via Google Base • Google Base was probed with 40 queries containing a mix of camera names, book, movie and music album titles • Total of 1440 sources were collected: 276 camera, 556 book, 572 movie and 281 music sources

  29. Sampling queries • Generated using publicly available online listings • Used 200 titles or names in each topic • Randomly selected cameras from pbase.com, book from New York Times best sellers, movies from ODP and music albums from Wikipedia’s top-100, 1986-2010

  30. Test queries • Contained a mix of queries from all four topics • Do not overlap with the sampling queries • Generated by randomly removing words from camera names, book, movie and music album titles with 0.5 probability • Number of test queries varied for different topics to obtain the required (0.95) statistical significance

  31. Query similarity based measure- CORI • CORI • Source statistics were collected using highest document frequency terms • Sources were selected using the same parameters as found optimal in CORI paper

  32. Query similarity based measure- Google Base • Google Base • Two-versions of Google Base were used • Gbase on dataset: Google Base search is restricted to our crawled sources • Gbase: Google Base search with no restrictions i.e. considers all sources in Google Base

  33. Agreement based measures - USR • Undifferentiated SourceRank, USR • SourceRank extended to multi-topic deep-web • Single agreement graph is computed using entire set of sampling queries • USR of sources is computed based on a random walk on this graph

  34. Agreement based measures - DSR • Oracular source selection, DSR • Assumes a perfect classification of sources and user queries are available i.e. each source and test query is manually labeled with its domain association • Creates agreement graphs and SourceRanks for a domain including only sources in that domain • For each test query, sources ranking high in the domain corresponding to the test query are used

  35. Result merging, ranking and relevance evaluation • Top-k sources are selected • Google Base is made to query only on these to top-k sources • Experimented with different values of k and found k=10 to be optimal • Google Base’s tuple ranking was used for ranking resulting tuples and return top-5 results in response to test queries

  36. Result merging, ranking and relevance evaluation contd. • Top-5 results returned were manually classified as relevant or irrelevant • Result classification was rule based • Example- if the test query is “pirates caribbean chest” and original movie name is “Pirates of Caribbean and Dead Man’s Chest”, then if the result entity refers to the same movie (dvd, blue-ray etc.) then the result is classified as relevant and otherwise irrelevant • To avoid author bias, results from different source selection methods were merged in a single file so that the evaluator does not know which method each result came from while he does the classification

  37. Results • TSR was compared with the baseline source selection methods • Agreement based measures (TSR, USR and DSR) were combined with query-similarity based CORI measure. The combination is represented by agreement based measure name and the weight assigned to agreement based measure, 1- • Example: TSR(0.1) represents 0.9xCORI + 0.1xTSR • We experimented with different values of  and found that =0.9 gives best precision for TSR-based source selection i.e. TSR(0.1) • Higher weightage of CORI compared to TSR is to compensate the fact that TSR scores have higher dispersion compared to CORI scores

  38. TSR precision exceeds that of similarity-based measures by 85% Comparison of top-5 precision of TSR(0.1) and query similarity based methods: CORI and Google Base

  39. TSR significantly out-performs all query-similarity based measures for all topics Comparison of topic-wise top-5 precision of TSR(0.1) and query similarity based methods: CORI and Google Base

  40. TSR precision exceeds USR(0.1) by 18% and USR(1.0) by 40% Comparison of top-5 precision of TSR(0.1) and agreement based methods: USR(0.1) and USR(1.0)

  41. For three out of the four topics, TSR(0.1) out-performs USR(0.1) and USR(1.0) with confidence levels 0.95 or more Comparison of topic-wise top-5 precision of TSR(0.1) and agreement based methods: USR(0.1) and USR(1.0)

  42. TSR(0.1) is able to match DSR(0.1)’s performance Comparison of top-5 precision of TSR(0.1) and oracular DSR(0.1)

  43. TSR(0.1) matches DSR(0.1) performance across all topics indicating its effectiveness in identifying important sources across all topics Comparison of topic-wise top-5 precision of TSR(0.1) and oracular DSR(0.1)

  44. Agenda • SourceRank • Topic-sensitive SourceRank • Experiments and Results • Conclusion

  45. Conclusion • We attempted multi-topic source selection sensitive to trustworthiness and importance for the deep-web • We introduced topic-sensitive SourceRank (TSR) • Our experiments on more than a thousand deep-web sources show that a TSR-based approach is highly effective in extending SourceRank to multi-topic deep-web

  46. Conclusion contd. • TSR out-performs query-similarity based measures by around 85% in precision • TSR results in statistically significant precision improvements over other baseline agreement-based methods • Comparison with oracular DSR approach reveals effectiveness of TSR for topic-specific query and source classification and subsequent source selection

  47. Paper submitted to Comad’11

  48. Questions?

More Related