1 / 25

Google Search Engine* CS461 Lecture Department of Computer Science Iowa State University

Google Search Engine* CS461 Lecture Department of Computer Science Iowa State University. “ The Anatomy of a Large-Scale Hypertextual Web Search Engine ”, S. Brin and L. Page, in Proceeding of WWW’98

tasha-vang
Download Presentation

Google Search Engine* CS461 Lecture Department of Computer Science Iowa State University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Google Search Engine* CS461 Lecture Department of Computer Science Iowa State University • “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, S. Brin and L. Page, in Proceeding of WWW’98 • “The pagerank citation ranking: Bringing order to the Web “, L. Page, S. Brin, R. Motwani, and T. Winograd, Technical Report, Stanford University, 1998

  2. What to cover today • PageRank • Google Architecture

  3. Problem Statement • Ultimate version • Find what I want • In most cases, I don’t know exactly or cannot expressed clearly what I want • “What-I-want” can be estimated using a set of keywords • Simplified version • Find the files that are most related to a set of keywords

  4. Naïve Solution • How it works • Download the entire Internet to a local machine • Search and return all files containing the set of keywords • Problems: all files are treated equally importance • Could return tons of files, but most of them are not what I want • Since most users simply check out the first few files, this scheme actually cannot find much useful things

  5. Ranking Based on Hit Rate • How it works • A file is ranked higher if it is visited more frequently • Problems • Could be affected by faked hits • A file will be ranked higher and higher

  6. Ranking based on Citation • Basic idea • A paper is important if it is cited by many papers • Each paper has a set of references that link to the related work • A pioneering paper typically has a high citation • An HTML page is more important if it is linked by many other page • Each page may link to other pages • Problems • Publish of academic papers is well-controlled • Many are peer-reviewed • Chronically ordered • Internet files could be anything

  7. Proposed: PageRank • Basic idea • A page with many links to it is more likely to be useful than one with few links to it • Just like citation • The links from a page that itself is the target of many links are likely to be particularly important • This is something new

  8. Proposed: PageRank • Basic idea • A page with many links to it is more likely to be useful than one with few links to it • Just like citation • The links from a page that itself is the target of many links are likely to be particularly important • This is something new back links forward link Each link has different weight

  9. Proposed: PageRank • How it works • Each page is ranked using a value called PageRank (PR) • A page’s PR depends on the PRs of its back link pages PR(A)=(1-d) + d*[PR(T1)/C(T1)+…+ PR(Tn)/C(Tn)] d: damping factor, normally this is set to 0.85 T1, … Tn: pages point to page A PR(A): PageRank of page A PR(Ti): PageRank of page Ti pointing to page A C(Ti): the number of links going out of page Ti

  10. Proposed: PageRank • Properties of PageRank formula • PageRanks form a probability distribution over web pages, so the normalized sum of all web pages' PageRanks will be one • Challenge of calculating PageRanks • The links could be circulated, e.g., ABA Page A Page B

  11. Page A Page B PageRank Calculation • Assign each page an initial rank value • Could be any number (seed) • Repeat calculations until the rank of each page does not change much Seed = 1 PR(A)= 0.15 + 0.85 * 1 = 1 PR(B)= 0.15 + 0.85 * 1 = 1 d= 0.85 PR(A)= (1 – d) + d(PR(B)/1) PR(B)= (1 – d) + d(PR(A)/1)

  12. Page A Page B PageRank Calculation • Assign each page an initial rank value • Could be any number (seed) • Repeat calculations until the rank of each page does not change much Seed = 0 1) PR(A)= 0.15 + 0.85 * 0 = 0.15  PR(B)= 0.15 + 0.85 * 0.15 = 0.2775 2) PR(A)= 0.15 + 0.85 * 0.2775 = 0.385875 PR(B)= 0.15 + 0.85 * 0.385875 = 0.47799375 3) PR(A)= 0.15 + 0.85 * 0.47799375 = 0.5562946875 PR(B)= 0.15 + 0.85 * 0.5562946875 = 0.622850484375 d= 0.85 PR(A)= (1 – d) + d(PR(B)/1) PR(B)= (1 – d) + d(PR(A)/1)

  13. Page A Page B PageRank Calculation • Assign each page an initial rank value • Could be any number (seed) • Repeat calculations until the rank of each page does not change much Seed = 40 1) PR(A)= 0.15 + 0.85 * 40 = 34.25 PR(B)= 0.15 + 0.85 * 0.385875 = 29.1775 2) PR(A)= 0.15 + 0.85 * 29.1775 = 24.950875 PR(B)= 0.15 + 0.85 * 24.950875 = 21.35824375 3) ...... d= 0.85 PR(A)= (1 – d) + d(PR(B)/1) PR(B)= (1 – d) + d(PR(A)/1)

  14. Page A Page B PageRank Calculation • Assign each page an initial rank value • Could be any number (seed) • Repeat calculations until the rank of each page does not change much Seed = 40 1) PR(A)= 0.15 + 0.85 * 40 = 34.25 PR(B)= 0.15 + 0.85 * 0.385875 = 29.1775 2) PR(A)= 0.15 + 0.85 * 29.1775 = 24.950875 PR(B)= 0.15 + 0.85 * 24.950875 = 21.35824375 3) …… Observation: It doesn’t matter what the seed value you use, once the PageRank calculations settle down, the “normalized probability distribution” (the average PageRank for all pages) will be 1.0

  15. Example of Calculation (0) Page A Page B Page C Page D

  16. Example of Calculation (1) Page A 1 Page B 1 Page C 1 Page D 1

  17. Example of Calculation (2) Page A 1 Page B 1 1*0.85/2 1*0.85/2 1*0.85 1*0.85 Page C 1 Page D 1 1*0.85

  18. Each page has not passed on 0.15, so we get: Page A: 0.85 (from Page C) + 0.15 (not transferred) = 1 Page B: 0.425 (from Page A) + 0.15 (not transferred) = 0.575 Page C: 0.85 (from Page D) + 0.85 (from Page B) + 0.425 (from Page A) + 0.15 (not transferred) = 2.275 Page D: receives none, but has not transferred 0.15 = 0.15 Page A 1 Page B 0.575 Page C 2.275 Page D 0.15

  19. Example of Calculation (3) Page A 1 Page B 0.575 Page C 2.275 Page D 0.15

  20. Page A: 2.275*0.85 (from Page C) + 0.15 (not transferred) = 2.08375 Page B: 1*0.85/2 (from Page A) + 0.15 (not transferred) = 0.575 Page C: 0.15*0.85 (from Page D) + 0.575*0.85(from Page B) + 1*0.85/2 (from Page A) +0.15 (not transferred) = 1.19125 Page D: receives none, but has not transferred, remains at 0.15 Page A 2.03875 Page B 0.575 Page C 1.1925 Page D 0.15

  21. Example of calculation (4) • After 20 iterations, we get Page A 1.490 Page B 0.783 Page C 1.577 Page D 0.15 In reality: a PageRank for 26,000,000web pages can be computed in a few hours on a medium size workstation. (1998)

  22. Result • Page C has the highest PageRank, and page A has the next highest: page C has a highest importance in this page links! • More iterations lead to a stability PageRank of the resulting page for keyword research.

  23. PageRank Summary • PageRank is a citation importance ranking • Approximated measure of importance or quality • Number of citations or backlinks • The pages with high PageRanks are those that are linked to by many pages and/or by important pages (e.g., Yahoo!)

  24. PageRank Summary • PageRank is a citation importance ranking • Approximated measure of importance or quality • Number of citations or backlinks • Each citation has different weight • The pages with high PageRanks are those that are linked to by many pages and/or by important pages (e.g., Yahoo!) • Questions: how to improve the ranking of your web pages? • Creating dummy sites to link to their main sites? • Increasing internal links and/or decreasing external links?

  25. Google Architecture

More Related