1 / 53

Web Search

Web Search. Crawling. Start from some root site e.g., Yahoo directories. Traverse the HREF links. Consequence: If there isn’t an HREF path from some Yahoo like directory, then your page probably isn’t indexed by any search engine. Information Retrieval (IR).

erwin
Download Presentation

Web Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web Search

  2. Crawling • Start from some root site e.g., Yahoo directories. • Traverse the HREF links. • Consequence: If there isn’t an HREF path from some Yahoo like directory, then your page probably isn’t indexed by any search engine.

  3. Information Retrieval (IR) • Conceptually, IR is the study of finding needed information. • IR helps users find information that matches their information needs expressed as queries • Historically, IR is about document retrieval, emphasizing document as the basic unit. • Finding documents relevant to user queries • Web search also has its root in IR. From: Bing Liu. Web Data Mining. 2007

  4. IR architecture From: Bing Liu. Web Data Mining. 2007

  5. IR queries • Keyword queries • Boolean queries (using AND, OR, NOT) • Phrase queries • Proximity queries • Full document queries • Natural language questions From: Bing Liu. Web Data Mining. 2007

  6. Information retrieval models • An IR model governs how a document and a query are represented and how the relevance of a document to a user query is defined. • Main models: • Boolean model • Vector space model • Statistical language model • etc From: Bing Liu. Web Data Mining. 2007

  7. Boolean model • Each document or query is treated as a “bag” of words or terms. Word sequence is not considered. • Given a collection of documents D, let V = {t1, t2, ..., t|V|} be the set of distinctive words/terms in the collection. V is called the vocabulary. • A weight wij> 0 is associated with each term tiof a document djin D. dj= (w1j, w2j, ..., w|V|j) • For a term that does not appear in document dj, wij= 0. From: Bing Liu. Web Data Mining. 2007

  8. Boolean model (cont’d) • Query terms are combined logically using the Boolean operators AND, OR, and NOT. • E.g., ((data AND mining) AND (NOT text)) • Retrieval • Given a Boolean query, the system retrieves every document that makes the query logically true. • Called exact match. • The retrieval results are usually quite poor because term frequency is not considered. From: Bing Liu. Web Data Mining. 2007

  9. Vector space model • Documents are also treated as a “bag” of words or terms. • Each document is represented as a vector. • However, the term weights are no longer 0 or 1. Each term weight is computed based on some variations of TF or TF-IDF scheme. • Term Frequency (TF) Scheme:The weight of a term tiin document djis the number of times that tiappears in dj, denoted by fij. Normalization may also be applied. • Shortcoming of the TF scheme is that it doesn’t consider the situation where a term appears in many documents of the collection. • Such a term may not be discriminative. From: Bing Liu. Web Data Mining. 2007

  10. TF-IDF term weighting scheme • The most well known weighting scheme • TF: (normalized) term frequency • IDF: inverse document frequency. N: total number of docs dfi: the number of docs that ti appears. • The final TF-IDF term weight is: From: Bing Liu. Web Data Mining. 2007

  11. Retrieval in vector space model • Query q is represented in the same way as a document. • The term wiq of each term ti in q can also computed in the same way as in normal document. • Relevance of dj to q: Compare the similarity of query q and document dj. • For this, use cosine similarity (the cosine of the angle between the two vectors) From: Bing Liu. Web Data Mining. 2007

  12. An Example • Suppose a document space is defined by three terms (words): • hardware, software, users • the vocabulary • A set of documents are defined as: • A1=(1, 0, 0), A2=(0, 1, 0), A3=(0, 0, 1) • A4=(1, 1, 0), A5=(1, 0, 1), A6=(0, 1, 1) • A7=(1, 1, 1) A8=(1, 0, 1). A9=(0, 1, 1) • If the Query is “hardware and software” • what documents should be retrieved? From: Bing Liu. Web Data Mining. 2007

  13. An Example (cont.) • In Boolean query matching: • document A4, A7 will be retrieved (“AND”) • retrieved: A1, A2, A4, A5, A6, A7, A8, A9 (“OR”) • In similarity matching (cosine): • q=(1, 1, 0) • S(q, A1)=0.71, S(q, A2)=0.71, S(q, A3)=0 • S(q, A4)=1, S(q, A5)=0.5, S(q, A6)=0.5 • S(q, A7)=0.82, S(q, A8)=0.5, S(q, A9)=0.5 • Document retrieved set (with ranking)= • {A4, A7, A1, A2, A5, A6, A8, A9} From: Bing Liu. Web Data Mining. 2007

  14. Text pre-processing • Word (term) extraction: easy • Stopwords removal • Stemming • Frequency counts and computing TF-IDF term weights. From: Bing Liu. Web Data Mining. 2007

  15. Stopwords removal • Some of the most frequently used words aren’t useful in IR and text mining – these words are called stop words. • the, of, and, to, …. • Typically about 400 to 500 such words • For an application, an additional domain specific stopwords list may be constructed • Why do we need to remove stopwords? • Reduce indexing (or data) file size • stopwords accounts 20-30% of total word counts. • Improve efficiency and effectiveness • stopwords are not useful for searching or text mining • they may also confuse the retrieval system. From: Bing Liu. Web Data Mining. 2007

  16. Stemming • Techniques used to find out the root/stem of a word. E.g., • user engineering • users engineered • used engineer • using • stem: use engineer Usefulness: • improving effectiveness of IR and text mining • matching similar words • Mainly improve recall • reducing indexing size • combing words with same roots may reduce indexing size as much as 40-50%. From: Bing Liu. Web Data Mining. 2007

  17. Basic stemming methods Using a set of rules. E.g., • remove ending • if a word ends with a consonant other than s, followed by an s, then delete s. • if a word ends in es, drop the s. • if a word ends in ing, delete the ing unless the remaining word consists only of one letter or of th. • If a word ends with ed, preceded by a consonant, delete the ed unless this leaves only a single letter. • …... • transform words • if a word ends with “ies” but not “eies” or “aies” then “ies --> y.” From: Bing Liu. Web Data Mining. 2007

  18. Precision and Recall In information retrieval (search engines) community, system evaluation revolves around the notion of relevantand notrelevantdocuments. Precisionis the fraction of retrieved documents that are relevant Recallis the fraction of relevant documents that are retrieved

  19. In terms of confusion matrix…

  20. Why having two numbers? • The advantage of having the two numbers for precision and recall is that one is more important than the other in many circumstances. • Typical web surfers: • would like every result on the first page to be relevant (high precision), but have not the slightest interest in knowing let alone looking at every document that is relevant. • Professional searchers such as paralegals and intelligence analysts: • are very concerned with trying to get as high recall as possible, and will tolerate fairly low precision results in order to get it.

  21. What about a single number? • The combined measure which is standardly used is called the F measure, which is the weighted harmonic mean of precision and recall: where • The default is to equally weight precision and recall, giving a balanced F measure. • This corresponds to making = 1/2 or =1. • Commonly written as F1, which is short for F=1

  22. Precision at k • The above measures precision at all recall levels. • What matters is rather how many good results there are on the first page or the first three pages. • This leads to measures of precision at fixed low levels of retrieved results, such as 10 or 30 documents. • This is referred to as “Precision at k”, for example “Precision at 10.”

  23. Web Search as a huge IR system • A Web crawler (robot) crawls the Web to collect all the pages. • Servers establish a huge inverted indexing database and other indexing databases • At query (search) time, search engines conduct different types of vector query matching. • There is an Information Retrieval score coming out of this. • The documents have HREF links as well. They are used to compute a reputation score. • The two scores are combined together in order to produce a ranking of the returned documents.

  24. Inverted Indexes • The idea behind an inverted index is simple: Start with a set of documents containing words, and you want to "invert" that, to create a bunch of words each of which lists all the documents that contain that word. • Usually used with “buckets.”

  25. Additional Information in Buckets • We can extend bucket to include role, position of word, e.g. Type Position

  26. Google Page Ranking “The Anatomy of a Large-Scale Hypertextual Web Search Engine” by Sergey Brin and Lawrence Page http://www-db.stanford.edu/~backrub/google.html

  27. Outline • Page rank, for discovering the most “important” pages on the Web, as used in Google. • Hubs and authorities, a more detailed evaluation of the importance of Web pages using a variant of the eigenvector calculation used for Page rank.

  28. Page Rank (PR) Intuitively, we solve the recursive definition of “importance”: A page is important if important pages link to it. • Page rank is the estimated page importance. • In short PageRank is a “vote”, by all the other pages on the Web, about how important a page is. • A link to a page counts as a vote of support. • If there’s no link there’s no support (but it’s an abstention from voting rather than a vote against the page).

  29. Page Rank Formula PR(A) = PR(T1)/C(T1) +…+ PR(Tn)/C(Tn) • PR(Tn) - Each page has a notion of its own self-importance, which is say 1 initially. • C(Tn) – Count of outgoing links from page Tn. • Each page spreads its vote out evenly amongst all of it’s outgoing links. • PR(Tn)/C(Tn) – • Each page spreads its vote out evenly amongst all of it’s outgoing links. • So if our page (say page A) has a back link from page “n” the share of the vote page A will get from page “n” is “PR(Tn)/C(Tn).”

  30. How is Page Rank Calculated? • This is where it gets tricky. The page rank (PR) of each page depends on the PR of the pages pointing to it. • But we won’t know what PR those pages have until the pages pointing to them have their PR calculated and so on… • And when you consider that page links can form circles it seems impossible to do this calculation! • But actually it’s not that bad. Google paper says: • PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds to the principal eigenvector of the link matrix of the web. • Just go ahead and calculate a page’s PR without knowing the final value of the PR of the other pages. • Each time we run the calculation we’re getting a closer estimate of the final value. • Repeat the calculations lots of times until the numbers stop changing much.

  31. Web Matrix • Capture the formula by the web matrix WebM, that is: • Each page i corresponds to row i and column i of the matrix. • If page j has n successors (links), then: • the ij th entry is 1/n if page i is one of these n successors of page j, and • 0 otherwise. Then, the importance vector containing the rank of each page is calculated by: Ranknew = WebM • Rankold

  32. Example • In 1839, the Web consisted on only three pages: Netscape, Microsoft, and Amazon. For example, the first column of the Web matrix reflects the fact that Netscape divides its importance between itself and Amazon. The second column indicates that Microsoft gives all its importance to Amazon. Start with n = m = a = 1, then do rounds of improvements. Based on Jeff Ullman’s notes

  33. Example • The first four iterations give the following estimates: n = 1 m = 1 a = 1 1 1/2 3/2 5/4 3/4 1 9/8 1/2 11/8 5/4 11/16 17/16 • In the limit, the solution is n = a = 6/5; m = 3/5. • That is, Netscape and Amazon each have the same importance, and twice the importance of Microsoft (well this was 1839). Based on Jeff Ullman’s notes

  34. Problems With Real Web Graphs Dead ends: a page that has no successors has nowhere to send its importance. Eventually, all importance will “leak out of” the Web. Example: Suppose Microsoft tries to claim that it is a monopoly by removing all links from its site. The new Web, and the rank vectors for the first 4 iterations are shown. Eventually, each of n, m, and a become 0; i.e., all the importance leaked out. n = 1 1 3/4 5/8 1/2 m = 1 1/2 1/4 1/4 3/16 a = 1 1/2 1/2 3/8 5/16 Based on Jeff Ullman’s notes

  35. Problems With Real Web Graphs Spider traps: a group of one or more pages that have no links out of the group will eventually accumulate all the importance of the Web. Example: Angered by the decision, Microsoft decides it will link only to itself from now on. Now, Microsoft has become a spider trap. The new Web, and the rank vectors for the first 4 iterations are shown. n = 1 1 3/4 5/8 1/2 m = 1 3/2 7/4 2 35/16 a = 1 1/2 1/2 3/8 5/16 Now, m converges to 3, and n = a = 0. Based on Jeff Ullman’s notes

  36. Google Solution to Dead Ends and Spider Traps Stop the other pages having too much influence. This total vote is “damped down” by multiplying it by a factor. Example: If we use a 20% damp-down, the equation of previous example becomes: The solution to this equation is n = 7/11; m = 21/11; a = 5/11. Based on Jeff Ullman’s notes

  37. Hubs and Authorities • Intuitively, we define “hub” and “authority” in a mutually recursive way: • a hub links to many authorities, and • an authority is linked to by many hubs. • Authorities turn out to be pages that offer information about a topic, e.g., http://www.bctransit.com • Hubs are pages that don't provide the information, but tell you where to find the information, e.g., http://yahoo.com

  38. Matrix formulation • Use a matrix formulation similar to that of PageRank, but without the stochastic restriction. • We count each link as 1, regardless of how many successors or predecessors a page has. • Namely, define a matrix A whose rows and columns correspond to Web pages, with entry Aij = 1 if page i links to page j, and 0 if not. • Notice that AT , the transpose of A, looks like the matrix used for computing Page rank, but AT has 1's where the Page-rank matrix has fractions.

  39. Authority and Hubbiness Vectors • Let a and h be vectors, whose i th component corresponds to the degrees of authority and hubbiness of the i th page. • Let and  be suitable scaling factors. Then we can state: • h =  A a That is, the hubbiness of each page is the sum of the authorities of all the pages it links to, scaled by . • a = AT h That is, the authority of each page is the sum of the hubbiness of all the pages that link to it, scaled by . Based on Jeff Ullman’s notes

  40. Simple substitutions • We can derive from (1) and (2), using simple substitution, two equations that relate vectors a and h only to themselves. • a =  ATA a • h =  A ATh • As a result, we can compute h and a by relaxation, giving us the principal eigenvectors of the matrices AAT and ATA, respectively. Based on Jeff Ullman’s notes

  41. Example If we use  =  = 1 and assume that the vectors h = [hn, hm, ha] = [1, 1, 1], and a = [an, am, aa] = [1, 1, 1], the first three iterations of the equations for a and h are: Based on Jeff Ullman’s notes

  42. Web Spam: raison d’etre • E-commerce is rapidly growing • Projected to $329 billion by 2010 • More traffic  more money • Large fraction of traffic from Search Engines • Increase Search Engine referrals: • Place ads  • Provide genuinely better content  • Create Web spam …  From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  43. Web Spam (you know it when you see it)

  44. Defining Web Spam • Spam Web page:A page created for the sole purpose of attracting search engine referrals (to this page or some other “target” page) • Ultimately a judgment call • Some web pages are borderline cases From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  45. Why Web Spam is Bad • Bad for users • Makes it harder to satisfy information need • Leads to frustrating search experience • Bad for search engines • Wastes bandwidth, CPU cycles, storage space • Pollutes corpus (infinite number of spam pages!) • Distorts ranking of results From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  46. How pervasive is Web Spam? • Real-Web data from the MSNBot crawler • Collected during August 2004 • Processed only MIME types • text/html • text/plain • 105,484,446 Web pages in total From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  47. Spam per Top-level Domain 95% confidence From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  48. Spam per Language 95% confidence From: Ntoulas, Najork, Manasse, Fetterly. Detecting Spam Web Pages through Content Analysis. 2006

  49. Content Spamming • Most search engines use variations of TF-IDF based measures to assess the relevance of a page to a user query. • Content-based spamming methods tailor the contents of the text fields in HTML pages to make spam pages more relevant to some queries. • Content spamming can be placed in any text field: • Title • Since search engines usually give higher weights to terms in the title of a page due to the importance of the title to a page, it is thus common to spam the title. • Body • Anchor Text • anchor text of a hyperlink is considered very important by search engines. • It is indexed for the page containing it and also for the page that it points to, so anchor text spam affects the ranking of both pages.

  50. Link Spamming • Hyperlinks play an important role in determining the reputation score of a page. Thus spammers also spam on hyperlinks. • Out-Link Spamming: • It is quite easy to add out-links in one's pages pointing to some authoritative pages to boost the hub cores of one's pages. • A page is a hub page if it points to many authoritative (or quality) pages. • To create massive out-links, spammers may use a technique called directory cloning. • There are many directories, e.g., Yahoo!, DMOZ Open Directory. Spammers simply replicate a large portion of a directory in the spam page to create a massive out-link structure quickly.

More Related