Authoritative Sources in a Hyperlinked environment Jon M. Kleinberg

1. Authoritative Sources in aHyperlinked environmentJon M. Kleinberg Presented By:- Vikrant KhoslaSridhar Kameswara Nemani

2. Outline • Search on WWW – Problem in general • Overview of the authoritative approach proposed by this paper • Constructing a focused Subgraph • Computing Hubs and Authorities • Similar page Queries • Multiple Sets of Hubs and Authorities • Diffusion and generalization • Evaluation • Conclusion

3. General Problem • How to improve quality of search on WWW? • Quality of search requires human evaluation due to the subjectivity inherent in notions such as relevance. • The WWW is a hypertext corpus of enormous complexity and information. • This paper aims to create link based model that consistently identifies relevant, authoritative WWW pages for broad search topics.

4. Understand Query Types • There is more than one type of query and the handling of each may require different techniques. • Type of queries: • Specific queries E.g. “Does Netscape support the JDK 1.1 code-signing API?” • Broad-topic queries E.g. “Find information about the Java programming language.” • Similar page queries Example: Find pages ‘similar ’ to honda.com

5. Difficulty in Handling query • Specific queries: • Scarcity Problem- There are few pages containing those information and it is difficult to determine the identity of those pages. • Broad topic queries: • Abundance problem- The number of pages that could reasonably be returned as relevant is far too large for a human user to digest. • Select a small set of the most “authoritative” or “definitive” ones from a huge collection of pages that are most relevant

6. Authoritative Pages • Given a particular page, how do we tell whether it is authoritative? • Problem is related to limitations of text based analysis. • Text based ranking function • E.g. For the “harvard”, www.harvard.edu is proper authoritative page but there may be lots of other web pages containing “harvard” more often. • Most popular Pages are not sufficiently self descriptive • Usually the term “search engine” doesn’t appear on search engine home web pages of Yahoo, AltaVista, Excite etc. • Honda or Toyota home pages hardly contain the term “automobile manufacturer”.

7. Analysis of link structure • Hyperlinks encode a latent human judgment which can be used to formulate a notion of authority. • Creation of a link represents a concrete indication of the following type of judgment • The creator of page p, by including a link to page q, has in some measure conferred authority on q. • Opportunity for the user to find potential authorities purely through the pages that point to them. • Potential Pitfalls of above concept • Most links are created for navigational purposes.(eg: main-menu, paid-adds) • Difficult to balance between appropriate relevance and popularity(eg: Yahoo)

8. Authorities and Hubs • Authorities are pages that are recognized as providing significant, trustworthy, and useful information on a topic. • Hubs are index pages that provide lots of useful links to relevant content pages (topic authorities). • In-degree - Number of pointers to a page and is one simple measure of authority. • Out-degree - Number of pointers from a page to other pages.

9. Can we operate over entire WWW ? • Local approaches- deals with intranet and amount of data is much smaller as compared to WWW as a whole. • Clustering approach- dissects a heterogeneous population into subpopulations that in some way more cohesive, but underlying problem of filtering vast number of pages is still the same. • Authoritative approach- global nature • Perform search on text based WWW search engine • Distil broad topic from these pages via the discovery of authority.

10. Overview of search steps

11. Overview • Search string • Text Search Engine • Authoritative Approach • Constructing focused subgraph • Computing Hub & Authorities • Better quality search result

12. Constructing Subgraph • The collection V of hyperlinked pages can be viewed as a directed graph G=(V,E):nodes correspond to pages, and a directed edge (p,q) ε E indicates the presence of a link from p to q. • Construct a focused subgraph (S  ) of the WWW with the following properties:- • S  is relatively small (so that computation is affordable) • S  is rich in relevant pages (so that its easier to find good authority) • S  contains most (or many) of the strongest authorities

13. How to find S • Set Q-set of all pages containing query string. • Root set R  - t highest ranked pages for the query  got from a text-based search engine. It satisfy property 1 & 2. • Problems with R  : • R is a subset of collection Q and Q does not satisfy property 3. • There are extremely few links between pages in R, rendering it essentially “structureless”. • Strong authority for query is quite likely to be pointed to by at least one page in R . • Construct Base set S by extend root set R  by including :- • All pages linked to by pages in R • All pages that link to a page in R at most d

14. Subgraph algorithm

15. Observation & Heuristics • Heuristic 1: Delete all intrinsic links & keep all transverse links • Intrinsic links: • if the link is between pages with the same domain name. • Generally these are for navigation purposes. • Less informative and often contain repetitive information. • Transverse: if it is between pages with different domain names. • Heuristic 2: Delete pages having collusion or keep 4 to 8 • Large number of pages from a single domain all point to a single page p. • Generally used for mass endorsement, advertisement etc.

16. Overview • Search string • Text Search Engine • Authoritative Approach • Constructing focused subgraph • Computing Hub & Authorities • Better quality search result

17. Computing Hubs & Authorities • Simplest approach would be to order pages by in-degree • Problem: • Nodes with highest in-degree in base set:- • might not necessarily be authorities & lack any thematic unity. • might simply be universally popular pages like yahoo, google, etc.

18. Computing Hubs & Authorities • Observation: • Good sources of content (authorities) • Good sources of links (hubs) • True authority pages are pointed by a number of good hubs. • Mutually reinforcing relationship: • Hubs point to lots of authorities. • Authorities are pointed to by lots of hubs • We will use the iterative algorithm to break this circularity. • Terms : • Good hub: page that points to many good authorities. • Good authority: page pointed to by many good hubs.

19. Overview of Algorithm

20. Iterative Algorithm • An iterative algorithm • with each page p , we associate • a non-negative authority weight x<p> • a non-negative hub weight y<p> • weights of each type are normalized so their squares sum to 1 • The pages with larger x and y values have “better” authorities and hubs respectively.

21. Iterative Algorithm • If p points to many pages with large x-values, then it should receive a large y-value • If p is pointed to by many pages with large y-values, then it should receive a large x-value • Inlinks Operation I: • Outlinks Operation O:

22. Algorithm

23. Matrices Basics

24. Observations • As one applies Iterate with arbitrary large k , the vectors • Let G = (V , E ), with V = {p1 , p2 ,…, pn }, and let A denote the adjacency matrix of the graph G : the (i , j )th entry of A is 1 if (pi , pj ) is an edge of G , and is 0 otherwise. • x* is the principal eigenvector of ATA , and y* is the principal eigenvector of AAT • The convergence of Iterate is quite rapid (k =20 is sufficient)

25. Observations • Any eigenvector algorithm can be used to compute the fixed points X* and Y* • Emphasizes the underlying motivation of the approach by reinforcing I and O operations • Do not require to iterate I and O to convergence • Can start from initial vector X0 and Y0 and computer using a fixed bound of I and O operations

26. Example: Mini Web

27. Example: Mini Web (Cont..)

28. Basic Results

29. Observations • Just “pure ” analysis of link structure • We ignored the text in searching for authoritative pages. • i.e., text-based search is just an initial set • Pages legitimately considered as authoritative in the context of www without access to large- scale index of the www • i.e., global analysis of the full www link structure can be replaced by local method over small focused subgraph • This approach can replace local approaches used in intranet

30. Similar page queries • Example: Find pages ‘similar ’ to honda.com • Using link structure to infer a notion of “similarity” among pages • We have found a page p that is of interest and it’s an authoritative page on a topic. • Can this help in finding similar pages? • What do users of the WWW consider to be related to p when they create pages and links ?

31. Similar page queries • Previously our request to search engine was: • “Find t pages containing the string . • Now our request to search engine is: • “Find t pages pointing to p” • Rp root set • Sp base set • Gp focused subgraph • Strongest authorities in the local region of the link structure near p are the potential broad-topic summary of pages related to p.

32. Results- Similar page queries

33. Multiple Sets of Hubs & Authorities • Several densely linked collections of hubs and authorities within the same set. • Example: • “jaguar” – has several different meanings. • “randomized algorithms” – arises multiple technical communities. • “abortion” - -involves groups that may not be linked to each other. • Clustering in presence of Abundance problem is needed.

34. Multiple Sets of Hubs & Authorities • The non-principal Eigenvectors provide us a way to extract additional densely linked collections of hubs and authorities. • Non-principal eigenvectors will have both positive and negative entries. • Often, the highly positive entries will correspond to a cluster of pages and negative entries to a different cluster. • Typically the two clusters will not be tightly intertwined. intertwined.

35. Jaguar Example • Authority principal eigenvector is primarily about the Atari product. • In the positive end of the 2nd non-principal eigenvector, the pages are primarily about the Jacksonville Jaguars. • In the positive end of the 3rd non-principal eigenvector, the pages are primarily about the car.

36. Randomized Algorithms Example • The first non-principal eigenvector, positive end returned home pages of theoretical computer scientists. • First non-principal eigenvector’s, negative end returns compendia of mathematical software. • In the negative end of the fourth non-principal eigenvector, the pages are primarily about wavelets.

37. Diffusion and Generealization • The query may not be sufficiently “broad.” • In this case there will not be enough highly relevant pages in the base set to extract a sufficiently dense sub-graph of relevant hubs and authorities. • When this occurs, the collection will often represent a broader topic, and the results will reflect a diffused version of the initial query. • Example: “WWW conferences” -> WWW resource pages. resource pages.

38. In studies conducted in 1998 over 26 queries and 37 volunteers, Clever reported better authorities than Yahoo!, which in turn was better than Alta Vista.

39. Conclusion • Need a way to distill a broad topic, for which there may be millions of relevant pages • Provides a high quality results in context of what is available on the www globally • Operate without maintaining an index of the www or its link structure • It identifies the complex pattern of social organization on the www.

40. References • http://crystal.uta.edu/~gdas/Courses/websitepages/spring07DBIR.htm • An Intro to Information Retrieval by Manning and Raghaban • Information Retrieval Data Structures Algorithms - William B. Frakes • Random Walks in Ranking Query Results in Semistructured Databases slides by Vagelis Hristidis

41. Thank You