Web Crawl Retrieval System for Researchers

1. CS 430 / INFO 430Information Retrieval Lecture 26 Thesauruses and Cluster Analysis 2

2. Course Administration CS 490 and CS 790 Independent Research Projects • Web Research Infrastructure -- Build a system to bring complete crawls of the Web from the Internet Archive to the Cornell Theory Center and make them available for researchers through a standard API. (Continues planning work carried out this semester.) • There will not be an independent research project in information retrieval.

3. Course Administration Final Examination • The final examination is on Monday, December 13, between 12:00 and 1:30. • There appears to be little interest in a make-up examination on another date. If, however, you have real problems with the scheduled date, send an email message to Anat Nidar-Levi (anat@cs.cornell.edu).

4. Cluster Analysis Cluster Analysis Methods that divide a set of n objects into m non- overlapping subsets. For information discovery, cluster analysis is applied to • terms for thesaurus construction • documents to divide into categories (sometimes called automatic classification, but classification usually requires a pre-determined set of categories).

5. Cluster Analysis Metrics Documents clustered on the basis of a similarity measure calculated from the terms that they contain. Documents clustered on the basis of co-occurring citations. Terms clustered on the basis of the documents in which they co-occur.

6. Non-hierarchical and Hierarchical Methods Non-hierarchicalmethods Elements are divided into m non-overlapping sets where m is predetermined. Hierarchical methods m is varied progressively to create a hierarchy of solutions. Agglomerative methods m is initially equal to n, the total number of elements, where every element is considered to be a cluster with one element. The hierarchy is produced by incrementally combining clusters.

7. Simple methods: Single Link Concept x x x x x x x x x x x x Similarity between clusters is similarity between most similar elements

8. Simple methods: Single Link Single Link A simple agglomerative method. Initially, each element is its own cluster with one element. At each step, calculate the similarity between each pair of clusters as the most similar pair of elements that are not yet in the same cluster. Merge the two clusters that are most similar. May lead to long, straggling clusters (chaining). Very simple computation.

9. Similarities: Incidence array D1: alpha bravo charlie delta echo foxtrot golf D2: golf golf golf delta alpha D3: bravo charlie bravo echo foxtrot bravo D4: foxtrot alpha alpha golf golf delta alpha bravo charlie delta echo foxtrot golf D1 1 1 1 1 1 1 1 D2 1 1 1 D3 1 1 1 1 D4 1 1 1 1 n 3 2 2 3 2 3 3

10. Term similarity matrix alpha bravo charlie delta echo foxtrot golf alpha 0.2 0.2 0.5 0.2 0.33 0.5 bravo 0.5 0.2 0.5 0.4 0.2 charlie 0.2 0.5 0.4 0.2 delta 0.2 0.33 0.5 echo 0.4 0.2 foxtrot 0.33 golf Using incidence matrix and dice weighting

11. Example -- single link 1 alpha delta golf bravo echo charlie foxtrot Agglomerative: step 1

12. Example -- single link 2 1 alpha delta golf bravo echo charlie foxtrot Agglomerative: step 2

13. Example -- single link 3 2 1 alpha delta golf bravo echo charlie foxtrot Agglomerative: step 3

14. Example -- single link 6 5 4 3 2 1 alpha delta golf bravo echo charlie foxtrot This style of diagram is called a dendrogram.

15. Simple methods: Complete Linkage Concept x x x x x x x x x x x x Similarity between clusters is similarity between least similar elements

16. Simple methods: complete linkage Complete linkage A simple agglomerative method. Initially, each element is its own cluster with one element. At each step, calculate the similarity between each pair of clusters as the similarity between the least similar pair of elements in the two clusters. Merge the two clusters that are most similar. Generates small, tightly bound clusters

17. Term similarity matrix alpha bravo charlie delta echo foxtrot golf alpha 0.2 0.2 0.5 0.2 0.33 0.5 bravo 0.5 0.2 0.5 0.4 0.2 charlie 0.2 0.5 0.4 0.2 delta 0.2 0.33 0.5 echo 0.4 0.2 foxtrot 0.33 golf Using incidence matrix and dice weighting

18. Example – complete linkage Cluster a b c d e f g elements Least similar pair / distance a - ab/.2 ac/.2 ad/.5 ae/.2 af/.33 ag/.5 b - bc/.5 bd/.2 be/.5 bf/.4 bg/.2 c - cd/.2 ce/.5 cf/.4 cg/.2 d - de/.2 df/.33 dg/.5 e - ef/.4 eg/.2 f - fg/.33 g - Step 1. Merge clusters {a} and {d}

19. Example – complete linkage Cluster a,d b c e f g elements Least similar pair / distance a,d - ab/.2 ac/.2 ae/.2 df/.33 ag/.5 b - bc/.5 be/.5 bf/.4 bg/.2 c - ce/.5 cf/.4 cg/.2 e - ef/.4 eg/.2 f - fg/.33 g - Step 2. Merge clusters {a,d} and {g}

20. Example – complete linkage Cluster a,d,g b c e f elements Least similar pair / distance a,d,g - ab/.2 ac/.2 ae/.2 af/.33 b - bc/.5 be/.5 bf/.4 c - ce/.5 cf/.4 e - ef/.4 f - Step 3. Merge clusters {b} and {c}

21. Example – complete linkage Cluster a,d,g b,c e f elements Least similar pair / distance a,d,g - ab/.2 ae/.2 af/.33 b,c - be/.5 bf/.4 e - ef/.4 f - Step 4. Merge clusters {b,c} and {e}

22. Example -- complete linkage Step 6 Step 5 Step 4 Step 3 Step 2 Step 1 alpha delta golf bravo charlie echo foxtrot

23. Problems with cluster analysis Selection of attributes on which items are clustered  Choice of similarity measure and algorithm  Computational resources  Assessing validity and stability of clusters  Updating clusters as data changes  Method for searching the clusters

24. Example 1: Concept Spaces for Scientific Terms Large-scale searches can only match terms specified by the user to terms appearing in documents. Cluster analysis can be used to provide information retrieval by concepts, rather than by terms. Bruce Schatz, William H. Mischo, Timothy W. Cole, Joseph B. Hardin, Ann P. Bishop (University of Illinois), Hsinchun Chen (University of Arizona), Federating Diverse Collections of Scientific Literature, IEEE Computer, May 1996. Federating Diverse Collections of Scientific Literature

25. Concept Spaces: Methodology Concept space: A similarity matrix based on co-occurrence of terms. Approach: Use cluster analysis to generate "concept spaces" automatically, i.e., clusters of terms that embrace a single semantic concept. Arrange concepts in a hierarchical classification.

26. Concept Spaces: INSPEC Data Data set 1: All terms in 400,000 records from INSPEC, containing 270,000 terms with 4,000,000 links. [24.5 hours of CPU on 16-node Silicon Graphics supercomputer.] computer-aided instruction see also education UF teaching machines BT educational computing TT computer applications RT education RT teaching

27. Concept Space: Compendex Data Data set 2: (a) 4,000,000 abstracts from the Compendex database covering all of engineering as the collection, partitioned along classification code lines into some 600 community repositories. [ Four days of CPU on 64-processor Convex Exemplar.] (b) In the largest experiment, 10,000,000 abstracts, were divided into sets of 100,000 and the concept space for each set generated separately. The sets were selected by the existing classification scheme.

28. Objectives • Semantic retrieval (using concept spaces for term suggestion) • Semantic interoperability (vocabulary switching across subject domains) • Semantic indexing (concept identification of document content) • Information representation (information units for uniform manipulation)

29. Use of Concept Space: Term Suggestion

30. Future Use of Concept Space: Vocabulary Switching "I'm a civil engineer who designs bridges. I'm interested in using fluid dynamics to compute the structural effects of wind currents on long structures. Ocean engineers who design undersea cables probably do similar computations for the structural effects of water currents on long structures. I want you [the system] to change my civil engineering fluid dynamics terms into the ocean engineering terms and search the undersea cable literature."

31. Example 2: Visual thesaurus for geographic images Methodology: • Divide images into small regions. • Create a similarity measure based on properties of these images. • Use cluster analysis tools to generate clusters of similar images. • Provide alternative representations of clusters. Marshall Ramsey, Hsinchun Chen, Bin Zhu, A Collection of Visual Thesauri for Browsing Large Collections of Geographic Images, May 1997. http://ai.bpa.arizona.edu/~mramsey/papers/visualThesaurus/visual Thesaurus.html

33. Example 3: Cluster Analysis of Social Science Journal In the social sciences, subject boundaries are unclear. Can citation patterns be used to develop criteria for matching information services to the interests of users? W. Y. Arms and C. R. Arms, Cluster analysis used on social science citations, Journal of Documentation, 34 (1) pp 1-11, March 1978.

34. Methodology Assumption: Two journals are close to each other if they are cited by the same source journals, with similar relative frequencies. Sources of citations:Select a sample of n social science journals. Citation matrix:Construct an m x n matrix in which the ijth element is the number of citations to journal i from journal j. Normalization:Alldata was normalized so that the sum of the elements in each row is 1.

35. Data Pilot study: 5,000 citations from the 1970 volumes of 17 major journals from across the social sciences. Criminology citations: Every fifth citation from a set of criminology journals (3 sets of data for 1950, 1960, 1970). Main file (52,000 citations): (a) Every citation from the 1970 volumes of the 48 most cited source journals in the pilot study. (b) Every citation from the 1970 volumes of 47 randomly selected journals.

36. Sample sizes Sample Source journals Target journals Pilot 17 115 Criminology: 1950 10 18 1960 13 49 1970 27 108 Main file: ranked 48 495 random 47 254 Excludes journals that are cited by only one source. These were assumed to cluster with the source.

37. Algorithms Main analysis used a non-hierarchical method of E. M. L. Beale and M. G. Kendal based on Euclidean distance. For comparison, 36 psychology journals clustered using: single-linkage complete-linkage van Rijsbergen's algorithm Beale/Kendal algorithm and complete-linkage produced similar results. Single-linkage suffered from chaining. Van Rijsbergen algorithm seeks very clear-cut clusters, which were not found in the data.

38. Non-hierarchical clusters Economics clusters in the pilot study

39. Non-hierarchical dendrogram Part of a dendrogram showing non-hierarchical structure

40. Conclusion "The overall conclusion must be that cluster analysis is not a practical method of designing secondary services in the social sciences." • Because of skewed distributions very large amounts of data are required. • Results are complex and difficult to interpret. • Overlap between social sciences leads to results that are sensitive to the precise data and algorithms chosen.

41. The End Return objects Return hits Browse content Search index