Finding Authoritative Pages in Hyperlinked Environment

1. Authoritative Sources in Hyperlinked EnvironmentJon M. Kleinberg JACM 1999 Presented By Raman Adaikkalavan Feb 23, 2005, CSE 6392 Instructor: Dr. Gautam Das

2. Overview • Problem – in general • Query Types • Problems of Answering Queries • Authoritative Pages – Broad-topic queries • Iterate Method/Algorithm • Similar Page Queries • Multiple Sets of Hubs and Authorities • Diffusion and Generalization • Evaluation • Comparison – ? • Conclusion

3. Problem – in general • Searching on the www for discovering pages that are relevant to a given query • Improving Quality of search

4. Query Types • Does Netscape support the JDK 1.1 code signing API • Specific queries • Find information about the Java programming language • Broad-topic queries • Find pages ‘similar’ to java.sun.com • Similar-page queries

5. Problems with Answering Queries • Specific queries • Scarcity problem: Very few pages that contain required information • Difficult to determine the identity of the pages • Broad-topic queries • Abundance problem: Number of pages that could reasonably be returned as relevant is far too large for a human user to digest • Select a small set of the most “authoritative” or “definitive” ones – pages that are most relevant

6. Authoritative Pages – Central focus • Given a query how to get the small set of authoritative pages corresponding to that query • How to accurately model authority in the context of a particularquery topic • Text-based searching/ranking – Sufficient ? Many prominent pages are not sufficiently self-descriptive. • “harvard” – www.harvard.edu • “search engines” – Yahoo, AltaVista, …? • “automobile manufacturers” – Honda, Toyota, …?

7. Analysis of the Link Structure • Hyperlinks encode a considerable amount of latent human judgment – used for authority ? • e.g., the creator of page p, by including a link to page q, has in some measure conferred authority on q • a large number of links are created primarily for navigational purposes, back • Links to paid advertisements • relevance and popularity • Find pages using #inlinks – this would consider highly popular pages as authoritative – Yahoo.com

8. Conferral of Authority • Model that consistently identifies relevant, authoritative www pages for broad search topics • Based on the relationship between ‘authorities’ and ‘hubs’ • Authorities: Pages that have relevant information about a given topic • Hubs: Pages that link to many related authorities

9. Till Now WWW • Authoritativepages • Not only based on text • Usinglinkanalysis Information about Java PL (Broad Topic Queries)

10. Can We Operate Over Entire WWW ? • Specific to a query; i.e., not predefined • Computational costs – should be reduced • Analysis of the link structure; which subgraph www should be operated on ? • All pages containing query string • May be over million pages - computation • Some or most of the best authorities may not belong to this set

11. Finding Authoritative Pages • Steps • 1: Construct a focused subgraph (S) of the www; such that • S is relatively small • S is rich in relevant pages • S contains most (or many) of the strongest authorities • 2: Compute Hubs and Authorities from the focused subgraph

12. Expanded Set Pages S t highest-ranked pages Rootset R Topic Search Engine At most d pages Forward link pages Backward link pages Construction of Focused Subgraph

13. Offsetting Navigational Links • G[S] subgraph induced on the pages in S • Types of links • Transverse: if between pages with different domain names • Intrinsic: is between pages within the same domain name • Delete Intrinsic Links from G[S]; resulting in a graph G • Collusion: large # of pages from a single domain all point to a single page p. “This site is designated to…” Eliminate by a parameter m (approx 4 – 8)

14. Finding Authoritative Pages • Steps • 1: Construct a focused subgraph (S) of the www • S is relatively small • S is rich in relevant pages • S contains most (or many) of the strongest authorities • 2: Compute Hubs and Authorities from the focused subgraph

15. Computing Hubs & Authorities • Goal: Given a query find: • Good sources of content (authorities) • Good sources of links (hubs) FROM: Monika Henzinger, Hyperlink Analysis on the Web

16. Intuition • Authoritycomes from in-edges. Being a goodhubcomes from out-edges. • Better authoritycomes from in-edges from good hubs. Being a better hubcomes from out-edges to good authorities. FROM: Monika Henzinger, Hyperlink Analysis on the Web

17. Hubs and Authorities • An iterative algorithm • with each page p, we associate • a non-negative authority weight x<p> • a non-negative hub weight y<p> • weights of each type are normalized so their squares sum to 1 • pS(x<p>)2 = 1 pS(y<p>)2 = 1 • The pages with larger x and yvalues have “better” authorities and hubs respectively.

18. Hubs and Authorities • If ppoints to many pages with large x-values, then it should receive a large y-value • If p is pointed to by many pages with large y-values, then it should receive a large x-value • Inlinks I: • Outlinks O:

19. Hubs and Authorities • As one applies Iterate with arbitrary large k, the {xk} and {yk}converge to fixed points x* and y* • Let G = (V, E), with V = {p1, p2,…, pn}, and let A denote the adjacency matrix of the graph G: the (i, j)th entry of A is 1 if (pi, pj) is an edge of G, and is 0 otherwise. • x* is the principal eigenvector of ATA, and y* is the principal eigenvector of AAT • The convergence of Iterate is quite rapid (k=20 is sufficient)

20. X X Y Y Z Z é é ù ù 1 1 1 0 1 1 X X ê ê ú ú ê ê ú ú = = M T M 0 1 0 0 1 1 Y Y ê ê ú ú ê ê ú ú 1 1 1 1 0 0 Z Z ê ê ú ú ë ë û û T = H H M M * i - i 1 T = A M M A * * - i i 1 Mini Web (Modified) Forward links Backward links HUBS AUTHORITIES X = H M A * - i i 1 T = A M H Z Y * - i i 1 SOURCE: Vagelis H, Random Walks Presentation

21. T = A M M A * * - i i 1 X is the best hub Z is the most authoritative Mini Web (Modified) T = H H M M * i - i 1 é ù é ù 2 2 1 3 1 2 ê ú ê ú ê ú ê ú = = 2 2 1 1 1 0 T T M M M M ê ú ê ú ê ú ê ú 1 1 2 2 0 2 ê ú ê ú ë û ë û ¥ Iteration 1 2 3 … X Z Y SOURCE: Vagelis H, Random Walks Presentation

22. Basic Results – Broad Topic Search

23. Observations • Just “pure” analysis of link structure • i.e., text-based search is just an initial set • Pages legitimately considered as authoritative in the context of www without access to large-scale index of the www • i.e., global analysis of the full www link structure can be replaced by local method over small focused subgraph

24. Overview • Problem – in general • Query Types • Problems of Answering Queries • Authoritative Pages – Broad-topic queries • Iterate Method/Algorithm • Similar Page Queries • Multiple Sets of Hubs and Authorities • Diffusion and Generalization • Evaluation • Comparison • Conclusion

25. Similar-Page Queries • E.g., Find pages ‘similar’ to honda.com • Using links analysis to infer a notion of “similarity” among pages • We have found a page p that is of interest and it’s an authoritative page on a topic. • What do users of the WWW consider to be related to p when they create pages and links ? • If p is highly referenced ? – abundance problem

26. Similar-Page Queries • In the local region of the link structure nearp, what are the strongest authorities • Can be a potential broad-topic summary of pages related to p • Normal Search; a query string  - “Find t pages containing ” as R and then get subgraph S • a page p -- “Find t pages pointing to p” as R and then get subgraph S

27. Results – Similar Page Queries

28. Multiple Sets of Hubs and Authorities • Broad-topic queries: most densely linked collection of hubs and authorities • Can we can find several densely linked collections of hubs and authorities among the same set S of pages. • Each collection could potentially be relevant to the query topic, but they could well-separated from one another in the graph G: • The query string  may have several very different meanings. E.g. “jaguar”, “java”. • The string may arise as a term in the context of multiple technical communities. E.g. “randomized algorithms”. • The string may refer to a highly polarized issue, involving groups that are not likely to link to one another. E.g. “abortion”.

29. Multiple Sets of Hubs and Authorities • Relevant documents can be grouped in to several clusters • For Broad-topic Queries: x* is the principal eigenvector of ATA, and y* is the principal eigenvector of AAT • Can we use the non-principal eigenvectors to extract additional densely linked collections of hubs and authorities • Positive and Negative

30. Results – Multiple Sets of H & A

31. Results – Multiple Sets of H & A

32. Results – Multiple Sets of H & A

33. Diffusion and Generalization • Diffusion happens • if the  specifies a topic that is not sufficiently broad, there will be not enough relevant pages in G • the most relevant collection in G is not the “densest” one • as a result the I and O operations will find the diffused collection of authority corresponding to the “broader” topics • Limits the algorithm • The broader topic that supplants the original, too-specific query  very often represents a natural generalization of  • It provides a simple way of abstracting a specific query topic to a broader related one.

34. Results – Diffusion & Generalization

35. Results – Diffusion & Generalization • The use of non-principal eigenvectors, combined with basic term-matching, can be a simple way to extract collections of authoritative pages that are more relevant to a specific query topic

36. Evaluation • 26 broad search topics, 37 users • For each topic, took the top 10 pages from AltaVista, the top five hubs and five authorities from Clever, and a random set of 10 pages from Yahoo • The results • For 31% of the topics, Yahoo and Clever received evaluations equivalent to each other • For 50%, Clever received a higher evaluation • For 19%, Yahoo received the higher evaluation

37. Summary • Answering Broad-topic queries • Finding Authoritative Pages using the good hubs and good authorities • Answering similar-page queries by starting with a different root set • Finding Multiple Hubs and Authorities using non-principle eigenvectors • Overcoming Diffusion and Generalization by using non-principal eigenvectors and basic term matching

38. PageRank vs. HITS • Computation: • Requires computation for each query • Query-dependent • Relatively easy to spam • Quality depends on quality of start set • Gives hubs as well as authorities • Computation: • Once for all documents and queries (offline) • Query-independent – requires combination with query-dependent criteria • Hard to spam FROM: Monika Henzinger, Hyperlink Analysis on the Web

39. [Lempel] Not rank-stable: O(1) changes in graph can change O(N2) order-relations [Ng,Zheng, Jordan01] “Value”-Stable: change in k nodes (with PR values p1,…pk) results in p* s.t. PageRank vs. HITS • Not rank-stable • “value”-stablility depends on gap g between largest and second largest eigenvector: change of O(g) nodes results in p* s.t. FROM: Monika Henzinger, Hyperlink Analysis on the Web

40. References/Slide Sources • Authoritative Sources in Hyperlinked EnvironmentJon M. Kleinberg JACM 1999 • Monika Henzinger “Hyperlink Analysis on the Web”. • Original Mini-web example http://www.cs.fiu.edu/~vagelis/presentations/RandomWalks.ppt • “Authoritative sources in a hyperlinked environment” Presentation By Vivek B. Tawde.

41. Conclusion • Influential paper • Citeseer – 457 Citings • ACM – 115 Citings • Same time period as the Google page-rank algorithm

42. Thank You