Lecture 7: Clustering and Decomposition of a Document to Authorial Components

1. Lecture 7: Clustering and Decomposition of a Document to Authorial Components Moshe Koppel and NavotAkiva

2. Clustering • The idea is to define “natural” clusters • Position in space depends on feature set • Need to have some notion of what you want in order to choose right feature types • e.g., cluster by topic: content words • e.g., cluster by author: style features • With many features, clusters aren’t so elegant

3. How Many Clusters? • Some methods decide in advance • Some methods try to optimize k based on data • e.g., in previous example, would want 3 clusters • Caution: some optimization criteria biased towards one giant cluster or many tiny ones

4. Flat Clustering: K-Means Algorithm Select K random docs {s1, s2,… sK} as seeds. Until clustering converges or other stopping criterion: For each doc di: Assign di to the cluster cjsuch that dist(xi, sj) is minimal. For each cluster cj: sj= (cj)(Update the seeds to the centroid of each cluster)

5. Pick seeds Reassign clusters Compute centroids Reassign clusters x x Compute centroids x x x x K-Means Example (K=2) Reassign clusters Converged!

6. Termination conditions • Several possibilities, e.g., • A fixed number of iterations. • Doc partition unchanged. • Centroid positions don’t change.

7. Convergence • Why should the K-means algorithm ever reach a fixed point? • A state in which clusters don’t change. • K-means is a special case of a general procedure known as the Expectation Maximization (EM) algorithm. • EM is known to converge. • Number of iterations could be large.

8. Convergence of K-Means • Define goodness measure of cluster k as sum of squared distances from cluster centroid: Gk = Σi (di – ck)2 (sum over all di in cluster k) G = Σk Gk • Reassignment monotonically decreases G since each vector is assigned to the closest centroid.

9. Dependence on Seed Choice • Results depend on random seed selection. • Some tricks for avoiding bad seeds: • Choose seed far from any already chosen • Try out multiple seed sets If you start with B and E as centroids you converge to {A,B,C} and {D,E,F} If you start with D and F you converge to {A,B,D,E} and {C,F}

10. A Problem… • The methods we’ve discussed so far don’t always give very satisfying results. • Have a look at this example….

11. Ng et al On Spectral clustering: analysis and algorithm

12. Spectral Methods • Because of examples like that, spectral methods have been developed. • Here’s the idea: Imagine weighted edges between points, where weight reflects similarity. Zero out all edges except k nearest neighbors with high weights. • Now make the “cheapest” cuts that result in separate components.

13. Ng et al On Spectral clustering: analysis and algorithm

14. Spectral Clustering Properly formulated, the min Ncut problem is NP-hard. But there are some nice linear algebra tricks we can use to get reasonable approximate solutions: • Pre-processing • Construct a matrix representation of the dataset. • Decomposition • Compute eigenvalues and eigenvectors of the matrix. • Map each point to a lower-dimensional representation based on one or more eigenvectors. • Grouping • Assign points to two or more clusters, based on the new representation.

15. New Problem: Separating Document Components • Often documents consist of multiple authorial components. • Our object is to tease apart the components of a composite document.

16. Basic Idea • Divide the document into natural chunks (e.g., chapters, paragraphs) • Vectorize chunks using some feature set • Cluster the vectors

17. Classic Example: The Bible • Great historic and cultural interest • Much prior research on components • Has been manually tagged in every conceivable way

18. Test Case Let’s throw the chapters of Ezekiel and Jeremiah into a hat and see if we can separate them out. • Each is presumably the work of a single (primary) author. • There’s no reason to think it’s the same author. • They are thematically quite similar, though far from identical.

19. Clustering Jeremiah+Ezekiel • Chunks = chapters (no sequence info) • Features = bag of words • Cluster method = Ncut • # clusters = 2

20. Results using all words • Not too exciting. We must be picking up thematic or genre-related differences that cross books. • Let’s try using only function words.

21. Results using function words • Not any better. • Let’s try a new approach.

22. A Better Idea • Exploit the fact that different authors use different synonyms for the same idea (e.g., makel/mateh). • It would be really convenient if it turned out that Jeremiah and Ezekiel made consistently different choices for various synsets. • Note that words aren’t synonyms, rather word senses are synonyms. (For example mateh=staff is a synonym of makel, but mateh=tribe is not.)

23. Automatically Finding Synonyms • There are various clever methods for identifying synsets, but most are not exact enough for our purpose. • Conveniently, for the Bible, we have many useful tools, including careful translations and manual sense tagging. • We identify as synonyms word senses that are translated into the same English word (e.g., makel=staff and mateh=staff). (We even get sense disambiguation for free.) • Due to polysemy (in English), this method overshoots. We manually delete mistakes. (This is the only manual intervention we will ever do.)

24. Synonym Method • The usual similarity measures (e.g., cosine, inverse Euclidean distance) aren’t quite right. • If two docs use the same synonym, they are similar. • If two docs use opposite synonyms, they are different. • If one of the docs uses one of the synonyms, but the other doesn’t, cosine would regard them as different. But are they? • For measuring similarity, we only consider synsets represented in both docs.

25. Results using synonyms • Now we’re getting somewhere. • But we’re not done yet…

26. Core of a Cluster • Some chapters are in a cluster because they really belong there; some just have to be somewhere. • Let’s consider only chapters near one centroid and far from the other. These are the “cores” of the respective clusters. 36 2 12 3 36 7 4

27. Cluster cores

28. Cluster cores Ezekiel 1, 10

29. Expanding the Core • Now that we have a core, we can use supervised methods (e.g., SVM) to learn a boundary. • In fact, we can use function words as our features. • Using synonyms will just get us back where we started. • And besides, FW are generally very reliable for supervised authorship attribution.

30. SVM expansion of core • Two training examples are “misclassed” by SVM. • Incredibly, these are Ezekiel 1 and 10, which were part of Jeremiah core, but are on Ezekiel side of optimal SVM boundary. • The only exception is Ezekiel 42, a non-core chapter which lies in the SVM margin.

31. Decomposing Unsegmented Text • Until now, we’ve assumed that our units (chapters) were all pure Ezekiel or pure Jeremiah. That’s a major unrealistic assumption. • Let’s create a munged “Jer-iel” book by alternately taking a random number of verses from each book until we run out of verses. • Can we tease that apart?

32. Chunks Might Be Mixed xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

33. Chunks Might Be Mixed xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

34. Decomposing Unsegmented Text • Start with some arbitrary chunking of the munged text. (Some of these chunks will be mixed.) • Now proceed exactly as above, until you have cluster cores. • With a bit of luck, the chunks in the core will be the “pure” chunks.

35. A Munged Document xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

36. Decomposing Unsegmented Text • As above, use SVM to learn a classifier, using the core chunks as training examples. • But at the last step, classify individual verses instead of chapters. • Some smoothing techniques are necessary to: • Deal with the sparseness of features in individual verses • Take advantage of serial correlation

37. A Munged Document xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

38. A Munged Document xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

39. A Munged Document xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

40. Results on Munged Books

41. Multiple Books • The same method works if we create a munged book out of k>2 constituent books. • BUT, we need to be given the k. • So we don’t yet know how many components there are, only how to best split into a given number of components.

42. Open Question • Can we extend this method so it can be used also for texts that don’t have canonical translations like the Bible? • Can we use this method to identify plagiarized sections of a text?