1 / 52

Search Engines

Search Engines. Indexing Page Ranking. The W W W. WebSite1. Page 1. Page 1. WebSite2. Page 2. Page 3. Page 3. Page 2. Page 5. Page 1. WebSite5. Page 4. Page 1. Page 6. Page 1. Page 2. WebSite4. WebSite3. The Web Search Problem. Query: set of key words or phrase. Search

avital
Download Presentation

Search Engines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Search Engines Indexing Page Ranking

  2. The W W W WebSite1 Page 1 Page 1 WebSite2 Page 2 Page 3 Page 3 Page 2 Page 5 Page 1 WebSite5 Page 4 Page 1 Page 6 Page 1 Page 2 WebSite4 WebSite3

  3. The Web Search Problem Query: set of key words or phrase Search Engine • Response: list of documents (pages) containing • the key words or phrase • Important requirements: • Response must be quick • Documents must be relevant

  4. Discover documents around the WWW Search keywords in documents Filter/rank documents according to their relevance Tasks of a Search Engine WebCrawlers (spiders, bots, wanderers, etc) Based on graph searching algorithms (BFS or DFS ?) For obvious performance reasons, this cannot be done by string searching after every query ! Solution: Indexing;Web Search Engine Architectures

  5. Web Search Engine Architecture WebCrawler Page Repository Query Text Analysis Link Analysis Text Index PageRank Ranker

  6. Outline • Data structures and algorithms for indexing the web • The PageRank algorithm

  7. Outline • Data structures and algorithms for indexing the web • The PageRank algorithm

  8. Indexing the web • Once a crawl has collected pages, their text is compressed and stored in a repository • Each URL mapped to a unique ID • A lexicon (sorted list of all words) is created • A hit list (“Inverted index”) is created for every word in the lexicon • Terminology: • Forward index: Document -> list of contained words • Inverted index: Word -> list of containing documents

  9. Simple Inverted IndexingWords -> PageIDs

  10. Using Simple Inverted Indexes for Queries • Simple indexes help searching for keywords or sets of keywords • Example: • Search “cat” => found in pages 1 and 3 • Search “cat” AND “dog” => found in page 3 • Simple indexes cannot help performing phrase queries: • Example: • Search “cat sat” => found in pages 1 and 3, but actually only page 1 contains the phrase “cat sat” • Solution: indexing contains also the in-page location

  11. Fully Inverted Indexing Words -> PageID’s + in-page locations

  12. Using Fully Inverted Indexing for Queries • Performing queries for phrases: • Search “cat sat” • “cat” found at 1-2, 3-2 • “sat” found at 1-3, 3-7 • “cat” AND “sat”: • in page 1, at 1-2 AND 1-3 => distance 1 between words • in page 3, at 3-2 AND 3-7 => distance 5 between words • Using the distance between words, only page 1 matches the search phrase

  13. Using Metainformation If the searched word is part of a title, the document is probably more relevant for the query

  14. Indexing the web • Once a crawl has collected pages, their text is compressed and stored in a repository • Each URL (document) mapped to a unique ID • A lexicon (sorted list of all words) is created • A hit list (“Inverted index”) is created for every word in the lexicon • Occurrences of a word in a particular document, including position, font, capitalization, metainformation (part of titles)

  15. Google’s Indexing – Step 1 • Each document is parsed an transformed into a collection of “hit lists” that are put into “barrels”, sorted by docID. • Hit: <wordID, position in doc, font info, hit type> • Hit type: Plain or fancy. • Fancy hit: Occurs in URL, title, anchor text, metatag.

  16. Google’s Forward Barrels Forward Barrels Docid Wordid #hits Hit, hit Wordid #hits Hit, hit, hit, hit, hit Wordid #hits Hit Wordid #hits Hit, hit Docid Wordid #hits Hit Wordid #hits Hit, hit, hit Wordid #hits Hit, hit Barrel i Docid Wordid #hits Hit, hit, hit Barrel i+1 Wordid #hits Hit, hit

  17. Google’s Indexing – Step 2 • Each barrel is then sorted by wordID to create the inverted index. This sorting also creates the lexicon file. • Lexicon: <wordID, offset into inverted index> • Lexicon is mostly cached in-memory

  18. wordid wordid wordid #docs #docs #docs Docid #hits Hit, hit, hit, hit, hit Docid #hits Hit Docid #hits Hit, hit Docid #hits Hit Docid #hits Hit, hit, hit Google’s Inverted Index Lexicon (in-memory) Postings (“Inverted Barrels”, on disk) Barrel i Sorted by wordid Barrel i+1

  19. Outline • Data structures and algorithms for indexing the web • The PageRank algorithm

  20. Motivation • Efficient matching: Indexing helps finding pages that contain the search phrase, giving priority to the pages that contain it in titles or other privileged positions. Still there can be a huge number of such matches ! • Also needed for an effective search: a measure of importance of the pages that matched the search criteria • Problem: Assessing the importance of web pages without human evaluation of the content • First solution: the PageRank algorithm

  21. PageRank History • History: • Proposed by 2 PhD students, Sergey Brin and Lawrence Page in 1998 at Stanford. • “The Anatomy of a Large-Scale Hypertextual Web Search Engine”. • “The PageRank citation ranking: Bringing order to the web ”, http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf • Algorithm of the first generation of Google Search Engine.

  22. PageRank Principles • Measure the importance of Web page based on the link structure alone. • The importance of a page is given by the number of pages linking to it (number of “votes” received) as well as their importance(the importance of the voters) • If a page contains links to a number of l pages, its contribution to the importance of each page is a fraction1/l of its own importance (it “splits” its votes)

  23. PageRank Principles - Example Importance(P1)=100 Outdegree(P1)=2 Importance(P3)=53 P3 P1 50 50 3 Importance(P2)=9 Outdegree(P2)=3 Importance(P4)=3 P4 P2 3 3

  24. Issues with Computing PageRank • The simplified PageRank computation principles presented before cannot be directly applied: • Pages without inlinks: which should be their PR value? (it cannot be zero, otherwise nothing gets propagated) • Cycles in page graphs: we cannot go forever round the cycle, always increasing the scores • The solution to this problem can be formulated from one of the possible viewpoints on PageRank: • Algebraic point of view • Probabilistic point of view

  25. PageRank – The Probabilistic Point of View • The Random Surfer Model • Since the importance of a web page P is measured by its popularity (how many incoming links it has) we can view the importance of the page as the probability that a random surfer that starts browsing the net at any page arrives at the page P following hyperlinks. • If the random surfer is at a page having k outlinks, he has 1/k probability to go next to any of the k pages

  26. The Random Surfer Model • Initial data: • The page graph contains N pages Pi, i=1..N • We denote by Bi the set of all pages Pjthat have links to Pi • We denote by lj the outdegree of page Pj (the number of its outgoing links) • Initially, each page Pihas 1/N probability to be choosen as a start page. This is the initial probability (at moment 0) of the page to be reached, PR(i, 0)

  27. The Random Surfer Model • Updating probabilities: • At a moment t, each page Pi has a probability PR(i, t) • At next moment t’, the probability of page Pi is PR(i, t’) and it is the weighted sum of the probabilities of its incoming pages, weighted by their outdegrees:

  28. The Random Surfer Model • Updating probabilities: PR(i, t’) PR(j, t) Outdegree(Pj)=lj Pi Pj PR(j, t)/lj

  29. The Random Surfer Model • Convergence: • The values PR(i, t), when t→∞, converge to PR(i) • The fact that PR converges to a unique probabilistic vector (the stationary distribution) can be mathematically proved (see: stochastic matrices, eigenvectors, the power method for finding eigenvector)

  30. N=4 l1=3, l2=2, l3=1, l4=2 Initially (t=0): PR(1,0)=1/4 PR(2,0)=1/4 PR(3,0)=1/4 PR(4,0)=1/4 Example PR(1,0)=1/4 PR(3,0)=1/4 1 P1 P3 1/3 1/3 1/2 1/3 1/2 1/2 P2 P4 1/2 PR(2,0)=1/4 PR(4,0)=1/4

  31. t=1; PR(1,1)=1*PR(3,0)+1/2*PR(4,0) = 1 * 0.25 + 1/2 * 0.25 = 0.37 PR(2,1)=1/3*PR(1,0)= 1/3 * 0.25 = 0.08 PR(3,1)=1/3*PR(1,0)+1/2*PR(2,0)+1/2*PR(4,0) = 1/3 * 0.25 + 1/2 * 0.25 + 1/2 * 0.25 = 0.33 PR(4,1)=1/3*PR(1,0)+1/2*PR(2,0)= 1/3 * 0.25 + 1/2 * 0.25 = 0.20 Example (cont) PR(1,0)=0.25 PR(3,0)=0.25 1 P1 P3 1/3 1/3 1/2 1/3 1/2 1/2 P2 P4 1/2 PR(2,0)=0.25 PR(4,0)=0.25

  32. t=2; PR(1,2)=1*PR(3,1)+1/2*PR(4,1) = 1 * 0.33 + 1/2 * 0.20 = 0.43 PR(2,2)=1/3*PR(1,1)= 1/3 * 0.37 = 0.12 PR(3,2)=1/3*PR(1,1)+1/2*PR(2,1)+1/2*PR(4,1) = 1/3 * 0.37 + 1/2 * 0.08 + 1/2 * 0.20 = 0.27 PR(4,2)=1/3*PR(1,1)+1/2*PR(2,1)= 1/3 * 0.37 + 1/2 * 0.08 = 0.16 Example (cont) PR(1,1)=0.37 PR(3,1)=0.33 1 P1 P3 1/3 1/3 1/2 1/3 1/2 1/2 P2 P4 1/2 PR(2,1)=0.08 PR(4,1)=0.20

  33. t=3; PR(1,3)=1*PR(3,2)+1/2*PR(4,2) = 1 * 0.27 + 1/2 * 0.16 = 0.35 PR(2,3)=1/3*PR(1,2)= 1/3 * 0.43 = 0.14 PR(3,3)=1/3*PR(1,2)+1/2*PR(2,2)+1/2*PR(4,2) = 1/3 * 0.43 + 1/2 * 0.12 + 1/2 * 0.16 = 0.29 PR(4,3)=1/3*PR(1,2)+1/2*PR(2,2)= 1/3 * 0.43 + 1/2 * 0.12 = 0.20 Example (cont) PR(1,2)=0.43 PR(3,2)=0.27 1 P1 P3 1/3 1/3 1/2 1/3 1/2 1/2 P2 P4 1/2 PR(2,2)=0.12 PR(4,2)=0.16

  34. The values of PR calculated until now: t=0: [0.25, 0.25, 0.25, 0.25] t=1: [0.37, 0.08, 0.33, 0.20] t=2: [0.43, 0.12, 0.27, 0.16] t=3: [0.35, 0.14, 0.29, 0.20] We can continue the iterations, and get: t=4: [0.39, 0.11, 0.29, 0.19] t=5: [0.39, 0.13, 0.28, 0.19] t=6: [0.38, 0.13, 0.29, 0.19] t=7: [0.38, 0.12, 0.29, 0.19] t=8: [0.38, 0.12, 0.29, 0.19] PR(1)=0.38 PR(2)=0.12 PR(3)=0.29 PR(4)=0.19 Example (cont)

  35. Dangling Nodes and Disconnected Components • Problems with the initial Random Surfer Model: • If the random web surfer arrives at a page Pj that has no outlinks (a dangling node), he has nowhere to go. The accumulated importance of Pj “gets lost”, since it is not transferred further to any other pages • If the web is formed by several connected components, the random web surfer will never reach pages that are in a different connected component than the initial random node

  36. N=3 l1=2, l2=2, l3=0 Initially (t=0): PR(1,0)=1/3 PR(2,0)=1/3 PR(3,0)=1/3 Update rules: PR(1,t’)=1/2 *PR(2,t) PR(2,t’)=1/2*PR(1,t) PR(3,t’)=1/2*PR(1,t)+1/2*PR(2,t) Example – The Dangling Node Problem P1 1/2 1/2 1/2 P3 1/2 P2

  37. Applying the update rules we get: t=0: [1/3, 1/3. 1/3] t=1: [1/6, 1/6, 1/3] t=2: [1/12, 1/12, 1/6] t=3: [1/24, 1/24, 1/12] …. Result: PR(1)=PR(2)=PR(3)=0 ! Example – The Dangling Node Problem (cont) P1 1/2 1/2 1/2 P3 1/2 P2 This result has no meaning as a ranking -> a solution must be found for dangling nodes

  38. Solution for dangling nodes and disconnected components • The PageRank Random Surfer model is updated as follows: • Most of the time (a percentage d) a surfer will follow links from a page, as in the model before. If a page has no outlinks, he will continue after it with a random page (a page with no outlinks will be considered to have N outlinks to any other page). • A smaller, but positive percentage of time (the rest of the percentage 1-d) the surfer will dump the current page and choose arbitrarily a different page from the web and “teleport” there

  39. The probability of reaching a page Pi The probability of arriving from a page Pj that has no outlinks The probability of arriving from a page Pj that has a link to Pi The probability of arriving through teleporting at a random time Computing PageRank d=dumping factor, heuristic

  40. The dumping factor • Dumping factor (d) can have values in [0,1] • If d=0: all the web surfer moves are random jumps (teleports), no links are followed • If d=1: the web surfer makes no teleports, he only follows links, except for the case of dangling nodes • The value of d also influences how fast the vector converges to the stationary distribution (the number of needed iterations) • Usual value (proposed by Brin and Page): d=0.85 • Convergence is reached in less than 100 iterations

  41. public Map<Vertex, Double> computePageRank(Digraph<Vertex> g) { double d=0.85; int iterations=100; int N=g.getNumberOfNodes(); List<Vertex> nodes= g.getAllNodes(); List<Vertex> nodesWithoutOutlinks = g.getNodesWithoutOutlinks(); Map<Vertex, Double> opr = new HashMap<Vertex, Double>(); // old pageranks Map<Vertex, Double> npr = new HashMap<Vertex, Double>(); // new pageranks for (Vertex n:nodes) npr.put(n, 1.0/N); // init pageranks with 1/N for (Vertex n:nodes) opr.put(n, 1.0/N); while (iterations>0) { double dp=0; for (Vertex p:nodesWithoutOutlinks) dp=dp+opr.get(p)/N; for (Vertex p:nodes) { double nprp; nprp=dp+(1-d)/N; for (Vertex ip: g.inboundNeighbors(p)) nprp=nprp+d*opr.get(ip)/g.outDegree(ip); npr.put(p,nprp); } Map<Vertex, Double> temp; temp=opr; opr=npr; npr=temp; iterations=iterations-1; } return npr; }

  42. PageRank – the Algebraic Point of View • Initial data: • The page graph contains N pages Pi, i=1..N • We denote by Bi the set of all pages Pjthat have links to Pi • We denote by lj the outdegree of page Pj (the number of its outgoing links) • The Hyperlink matrix A: a square matrix with the rows and column corresponding to web pages, where A[i,j] = 1/lj if there is a link from j to i and A[i,j] = 0 if not.

  43. Example –The Hyperlink Matrix 1 2 3 1 4 P1 P3 1 1/3 2 1/3 1/2 1/3 1/2 1/2 3 P2 P4 1/2 4

  44. Properties of the Hyperlink Matrix • Properties of the Hyperlink Matrix • All entries are nonnegative • The sum of the entries in a column j is 1, if j has outgoing links . • All elements of a column j are 0 if j has no outgoing links (j is a dangling node) • If the web has no dangling nodes, the Hyperlink matrix is stochastic

  45. Stochastic Matrices • A column stochastic matrix (probability matrix, Markov matrix) is a square matrix of nonnegative real numbers, with each column summing to 1.

  46. Stochastic Matrices • The Perron-Frobenius Theorem: Every positive column stochastic matrix A has a unique stationary column vector X (an eigenvector with eigenvalue 1): A*X=X • The Power Method Convergence Theorem: Let A be a positive column stochastic matrix of size n*n and X its stationary column vector. Then X can be calculated by following procedure: Initialize the column vector Z with all entries equal to 1/n. Then the sequence Z, A*Z, A2*Z ….,Ak*Z converges to the vector X.

  47. The Google Matrix • A= Transition matrix • S= a matrix obtained from A, by setting the elements of the columns where all elements of the column are 0, to 1/N • G= the Google matrix: • G[i,j]=d*S[i,j]+(1-d)/N • Property: the Google matrix is a stochastic matrix • The stationary vector of G contains the PageRank values

  48. PageRank and the History of Search Engines • PageRank (1998) was the first algorithm to introduce the concept of “importance of a webpage” and calculate it without relying on external information • crucial factor in Google ascension • Drawbacks: • PageRank can be manipulated • SEO (“Search Engine Optimisation”)

  49. PageRank and the Future of Search Engines • 2011: Google Panda: • introduce filters that prevent low quality sites and/or pages from ranking well in the search results, identifying • use human feedback and machine learning algorithms • 2012: Google Penguin: • decrease ranking of sites identified as using “black-hat SEO techniques”  • 2013: Google Hummingbird • Judge the context of a query - thereby judging the intent of a person carrying out a search, to determine what they are trying to find out

  50. Other Uses of PageRank • Ranking scientific articles according to their citations • Ranking streets for predicting human movement and street congestion • Automatic summarization – extracting the most relevant sentences from a text

More Related