3/23

1. HW 3 due on Thu 3/25 Midterm on Tu 3/30 Project 2 due on 4/6 3/23 Agenda: Engineering Issues (Crawling; Connection Server; Distributed Indexing; Map-Reduce) Text Clustering (followed by Text Classification)

2. Engineering Issues • Crawling • Distributed Index Generation • Connectivity Serving • Compressing everything..

6. SPIDER CASE STUDY

8. Mercator’s way of maintaining URL frontier Extracted URLs enter front queue Each URL goes into a front queue based on its Priority. (priority assigned Based on page importance and Change rate) URLs are shifted from Front to back queues. Each Back queue corresponds To a single host. Each queue Has time teat which the host Can be hit again URLs removed from back Queue when crawler wants A page to crawl

10. Robot (4) • How to extract URLs from a web page? Need to identify all possible tags and attributes that hold URLs. • Anchor tag: <a href=“URL” … > … </a> • Option tag: <option value=“URL”…> … </option> • Map: <area href=“URL” …> • Frame: <frame src=“URL” …> • Link to an image: <imgsrc=“URL” …> • Relative path vs. absolute path: <base href= …> “Path Ascending Crawlers” – ascend up the path of the URL to see if there is anything else higher up the URL

14. Focused Crawling • Classifier: Is crawled page P relevant to the topic? • Algorithm that maps page to relevant/irrelevant • Semi-automatic • Based on page vicinity.. • Distiller:is crawled page P likely to lead to relevant pages? • Algorithm that maps page to likely/unlikely • Could be just A/H computation, and taking HUBS • Distiller determines the priority of following links off of P

16. Connectivity Server.. • All the link-analysis techniques need information on who is pointing to who • In particular, need the back-link information • Connectivity server provides this. It can be seen as an inverted index • Forward: Page id  id’s of forward links • Inverted: Page id  id’s of pages linking to it

17. Large Scale Indexing

20. What is the best way to exploit all these machines? • What kind of parallelism? • Can’t be fine-grained • Can’t depend on shared-memory (which could fail) • Worker machines should be largely allowed to do their work independently • We may not even know how many (and which) machines may be available…

21. 3/25 3 Choices for Midterm Tuesday 3/30 Thursday 4/1 Deem & Pass

22. Map-Reduce Parallelism • Named after lisp constructs map and reduce • (reduce #’fn2 (map #’fn1 list)) • Run function fn1 on every item of the list, and reduce the resulting list using fn2 • (reduce #’* (map #’1+ ‘(4 5 6 7 8 9))) • (reduce #’* ‘(5 6 7 8 9 10)) • 151200 (=5*6*7*89*10) • (reduce #’+ (map #’primality-test ‘(num1 num2…))) • So where is the parallelism? • All the map operations can be done in parallel (e.g. you can test the primality of each of the numbers in parallel). • The overall reduce operation has to be done after the map operation (but can also be parallelized; e.g. assuming the primality-test returns a 0 or 1, the reduce operation can partition the list into k smaller lists and add the elements of each of the lists in parallel (and add the results) • Note that the parallelism in both the above examples depends on the length of input (the larger the input list the more parallel operations you can do in theory). • Map-reduce on clusters of computers involve writing your task in a map-reduce form • The cluster computing infrastructure will then “parallelize” the map and reduce parts using the available pool of machines (you don’t need to think—while writing the program—as to how many machines and which specific machines are used to do the parallel tasks) • An open source environment that provides such an infrastructure is Hadoop • http://hadoop.apache.org/core/ Qn: Can we bring map-reduce parallelism to indexing?

23. [From Lin & Dyer book]

24. Partition the set of documents into “blocks” construct index for each block separately merge the indexes

29. Other references on Map-Reduce http://www.umiacs.umd.edu/~jimmylin/book.html

30. Clustering

31. Idea and Applications • Clustering is the process of grouping a set of physical or abstract objects into classes of similar objects. • It is also called unsupervised learning. • It is a common and important task that finds many applications. • Applications in Search engines: • Structuring search results • Suggesting related pages • Automatic directory construction/update • Finding near identical/duplicate pages Improves recall Allows disambiguation Recovers missing details

33. An idea for getting cluster descriptions • Just as search results need snippets, clusters also need descriptors • One idea is to look for most frequently occurring terms in the cluster • A better idea is to consider most frequently occurring terms that are least common across clusters. • Each cluster is a set of document bags • Cluster doc is just the union of these bags • Find tf/idf over these cluster bags

35. Clustering can be done at: Indexing time At query time Applied to documents Applied to snippets Clustering can be based on: URL source Put pages from the same server together Text Content -Polysemy (“bat”, “banks”) -Multiple aspects of a single topic Links -Look at the connected components in the link graph (A/H analysis can do it) -look at co-citation similarity (e.g. as in collab filtering) (Text Clustering)When & From What

36. Clustering issues --Hard vs. Soft clusters --Distance measures cosine or Jaccard or.. --Cluster quality: Internal measures --intra-cluster tightness --inter-cluster separation External measures --How many points are put in wrong clusters. [From Mooney]

37. Cluster Evaluation • “Clusters can be evaluated with “internal” as well as “external” measures • Internal measures are related to the inter/intra cluster distance • A good clustering is one where • (Intra-cluster distance) the sum of distances between objects in the same cluster are minimized, • (Inter-cluster distance) while the distances between different clusters are maximized • Objective to minimize: F(Intra,Inter) • External measures are related to how representative are the current clusters to “true” classes. Measured in terms of purity, entropy or F-measure

38. Intra-cluster distance/tightness (Sum/Min/Max/Avg) the (absolute/squared) distance between All pairs of points in the cluster OR “diameter”—two farthest points Between the centroid /medoitand all points in the cluster OR Inter-cluster distance Sum the (squared) distance between all pairs of clusters Where distance between two clusters is defined as: distance between their centroids/medoids Distance between farthest pair of points (complete link) Distance between the closest pair of points belonging to the clusters (single link) Inter/Intra Cluster Distances Red: Single-link Black: complete-link

39. Cluster Evaluation • “Clusters can be evaluated with “internal” as well as “external” measures • Internal measures are related to the inter/intra cluster distance • A good clustering is one where • (Intra-cluster distance) the sum of distances between objects in the same cluster are minimized, • (Inter-cluster distance) while the distances between different clusters are maximized • Objective to minimize: F(Intra,Inter) • External measures are related to how representative are the current clusters to “true” classes. Measured in terms of purity, entropy or F-measure

40. Purity example          Cluster I Cluster II Cluster III Overall Purity = weighted purity Cluster I: Purity = 1/6 (max(5, 1, 0)) = 5/6 Cluster II: Purity = 1/6 (max(1, 4, 1)) = 4/6 Cluster III: Purity = 1/5 (max(2, 0, 3)) = 3/5

41. 3/30 Today’s agenda:  Text Clustering continued; K-means; hierachical clustering… Mid-term on Thu (4/1—ha ha ) Closed book and notes You are allowed one 8.5x11 sheet of hand written notes

42. Rand-Index: Precision/Recall based The following table classifies all pairs of entities (of which there are n choose 2) into One of four classes Is the cluster putting non-class items in? Is the cluster missing any in-class items?

43. How hard is clustering? • One idea is to consider all possible clusterings, and pick the one that has best inter and intra cluster distance properties • Suppose we are given n points, and would like to cluster them into k-clusters • How many possible clusterings? • Too hard to do it brute force or optimally • Solution: Iterative optimization algorithms • Start with a clustering, iteratively improve it (eg. K-means)

44. Classical clustering methods • Partitioning methods • k-Means (and EM), k-Medoids • Hierarchical methods • agglomerative, divisive, BIRCH • Model-based clustering methods

45. K-means • Works when we know k, the number of clusters we want to find • Idea: • Randomly pick k points as the “centroids” of the k clusters • Loop: • For each point, put the point in the cluster to whose centroid it is closest • Recompute the cluster centroids • Repeat loop (until there is no change in clusters between two consecutive iterations.) Iterative improvement of the objective function: Sum of the squared distance from each point to the centroid of its cluster (Notice that since K is fixed, maximizing tightness also maximizes inter-cluster distance)

46. Pick seeds Reassign clusters Compute centroids Reasssign clusters x x x Compute centroids x x x K Means Example(K=2) Reassign clusters Converged! [From Mooney]

47. What is K-Means Optimizing? • Define goodness measure of cluster k as sum of squared distances from cluster centroid: • Gk = Σi (di – ck)2 (sum over all di in cluster k) • G = Σk Gk • Reassignment monotonically decreases G since each vector is assigned to the closest centroid. • Is it global optimum? No.. because each node independently decides whether or not to shift clusters; sometimes there may be a better clustering but you need a set of nodes to simultaneously shift clusters to reach that.. (Mass Democrats moving en block to AZ example). • What cluster shapes will have the lowest sum of squared distances to the centroid? • Is it global optimum? No.. because each node independently decides whether or not to shift clusters; sometimes there may be a better clustering but you need a set of nodes to simultaneously shift clusters to reach that.. (Mass Democrats moving en block to AZ example). Spheres… But what if the real data doesn’t have spherical clusters? (We will still find them!)

48. K-means Example • For simplicity, 1-dimension objects and k=2. • Numerical difference is used as the distance • Objects: 1, 2, 5, 6,7 • K-means: • Randomly select 5 and 6 as centroids; • => Two clusters {1,2,5} and {6,7}; meanC1=8/3, meanC2=6.5 • => {1,2}, {5,6,7}; meanC1=1.5, meanC2=6 • => no change. • Aggregate dissimilarity • (sum of squares of distanceeach point of each cluster from its cluster center--(intra-cluster distance) • = 0.52+ 0.52+ 12+ 02+12 = 2.5 |1-1.5|2

49. Example of K-means in operation [From Hand et. Al.]

50. Problems with K-means Why not the minimum value? Example showing sensitivity to seeds • Need to know k in advance • Could try out several k? • Cluster tightness increases with increasing K. • Look for a kink in the tightness vs. K curve • Tends to go to local minima that are sensitive to the starting centroids • Try out multiple starting points • Disjoint and exhaustive • Doesn’t have a notion of “outliers” • Outlier problem can be handled by K-medoid or neighborhood-based algorithms • Assumes clusters are spherical in vector space • Sensitive to coordinate changes, weighting etc. In the above, if you start with B and E as centroids you converge to {A,B,C} and {D,E,F} If you start with D and F you converge to {A,B,D,E} {C,F}