Uniform Sampling from the Web via Random Walks

1. Uniform Sampling from the Web via Random Walks Ziv Bar-Yossef Alexander Berg Steve Chien Jittat Fakcharoenphol Dror Weitz University of California at Berkeley

2. Main goal: Develop a cheap method to sample uniformly from the Web Use a random sample of web pages to approximate: search engine coverage domain name distribution (.com, .org, .edu) percentage of porn pages average number of links in a page average page length Note: A web page is a static html page Motivation: Web Measurements

3. The Structure of the Web (Broder et al., 2000) large strongly connected component 1/4 left side right side 1/4 1/4 indexable web 1/4 tendrils & isolated regions

4. Obvious solution: sample from an index of all pages Maintaining an index of Web pages is difficult Requires extensive resources (storage, bandwidth) Hard to implement There is no consistent index of all Web pages Difficult to get complete coverage Month to crawl/index most of the Web Web is changing every minute Why is Web Sampling Hard?

5. Our Approach: Random Walks for Random Sampling • Random walk on a graph provides a sample of nodes • Graph is undirected and regular sample is uniform • Problems: The Web is neither undirected nor regular • Our solution • Incrementally create an undirected regular graph with the same nodes as the Web • Perform the walk on this graph

6. Related Work • Monika Henzinger, et al. (2000) • Random walk produces pages distributed by Google’s page rank. • Weight these pages to produce a nearly uniform sample. • Krishna Bharat & Andrei Broder (1998) • Measured relative size and overlap of search engines using random queries. • Steve Lawrence & Lee Giles (1998, 1999) • Size of the web by probing IP addresses and crawling servers. • Search engine coverage in response to certain queries.

7. v u v u From node v pick any outgoing edge with equal probability. Go to u. Random Walks: Definitions probability distribution qt qt(v) = prob.vis visited at stept Transition matrix A qt+1 = qtA Stationary distribution Limit as t grows of qt if it exists and is independent of q0 Markov processThe probability of a transition depends only on the current state. Mixing time # of steps required to approach the stationary distribution

8. Straightforward Random Walk on the Web amazon.com netscape.com Follow a random out-link at each step 4 7 1 6 9 3 5 8 2 www.cs.berkeley.edu/~zivi • Gets stuck in sinks and in dense Web communities • Biased towards popular pages • Converges slowly, if at all

9. WebWalker: Undirected Regular Random Walk on the Web 3 5 amazon.com Follow arandom out-link or a random in-linkat each step Useweighted self loopsto even out pages’ degrees 3 2 3 0 4 netscape.com 0 1 4 3 3 2 1 1 3 2 2 2 w(v) = degmax - deg(v) 4 www.cs.berkeley.edu/~zivi Fact: A random walk on a connected undirected regular graph converges to a uniform stationary distribution.

10. WebWalker: Mixing Time • Theorem [Markov chain folklore]: • A random walk’s mixing time is at most log(N)/(1 - 2) • where N = size of the graph • 1 -2 = eigenvalue gap of the transition matrix • Experiment (using an extensive Alexa crawl of the web from 1996) • WebWalker’s eigenvalue gap: 1 - 210-5 • Result: Webwalker’s mixing time is 3.1 million steps • Self loop steps are free • Only 1 in 30,000 steps is not a self loop step (degmax 3x105, degavg= 10) • Result: Webwalker’s actual mixing time is only 100 steps!

11. WebWalker: Mixing Time (cont.) • Mixing time on the current Web may be similar • Some evidence that the structure of the Web today is similar to the structure in 1996 (Kumar et al., 1999, Broder et al., 2000)

12. WebWalker: Realization (1) • Webwalker(v): • Spend expected degmax/deg(v) steps at v • Pick a random link incident to v (either v  u or u  v) • Webwalker(u) • Problems • The in-links of v are not available • deg(v) is not available • Partial sources of in-links: • Previously visited nodes • Reverse link services of search engines

13. WebWalker: Realization (2) • WebWalker uses only available links: • out-links • in-links from previously visited pages • first r in-links returned from the search engines • WebWalker walks on a sub-graph of the Web • sub-graph induced by available links • to ensure consistency: as soon as a page is visited its incident edge list is fixed for the rest of the walk

14. WebWalker: Example WebWalker’s Induced Sub-Graph Web Graph v6 v5 v6 v5 v1 v2 v1 2 1 v3 v2 v3 1 0 v4 v4 1 w 1 covered by search engines not covered by search engines available link non-available link

15. WebWalker: Bad News • WebWalker becomes a true random walk only after its induced sub-graph “stabilizes” • Induced sub-graph is random • Induced sub-graph misses some of the nodes • Eigenvalue gap analysis does not hold anymore

16. WebWalker: Good News • WebWalker eventually converges to a uniform distribution on the nodes of its induced sub-graph • WebWalker is a “close approximation” of a random walk much before the sub-graph stabilizes • Theorem: WebWalker’s induced sub-graph is guaranteed to eventually cover the whole indexable Web. • Corollary: WebWalker can produce uniform samples from the indexable Web.

17. Evaluation of WebWalker’s Performance Questions to address in experiments: • Structure of induced sub-graphs • Mixing time • Potential bias in early stages of the walk: • towards high degree pages • towards the search engines • towards the starting page’s neighborhood

18. WebWalker: Evaluation Experiments • Run WebWalker on the 1996 copy of the Web • 37.5 million pages • 15 million indexable pages • degavg= 7.15 • degmax= 300,000 • Designate a fraction p of the pages as the search engine index • Use WebWalker to generate a sample of 100,000 pages • Check the resulting sample against the actual values

19. Evaluation: Bias towards High Degree Nodes Percent of nodes from walk Low Degree High Degree Deciles of nodes ordered by degree

20. Evaluation: Bias towards the Search Engines Estimate of search engine size 30% 50% Search engine size

21. Evaluation: Bias towards the Starting Node’s Neighborhood Percent of nodes from walk Far from Starting Node Close to Starting Node Deciles of nodes by distance from starting node

22. WebWalker: Experiments on the Web • Run WebWalker on the actual Web • Two runs of 34,000 pages each • Dates: July 8, 2000 - July 15, 2000 • Used four search engines for reversed links: • AltaVista, HotBot, Lycos, Go

23. Domain Name Distribution

24. Search Engine Coverage

25. Web Page Parameters • Average page size: 8,390 Bytes • Average # of images on a page: 9.3 Images • Average # of hyperlinks on a page: 15.6 Links

26. Conclusions • Uniform sampling of Web pages by random walks • Good news: • walk provably converges to a uniform distribution • easy to implement and run with few resources • encouraging experimental results • Bad news: • no theoretical guarantees on the walk’s mixing time • some biases towards high degree nodes and the search engines • Future work: • obtain a better theoretical analysis • eliminate biases • deal with dynamic content

27. Thank You!