Unsupervised Learning of Natural Language Morphology using MDL

1. Unsupervised Learning of Natural Language Morphology using MDL John Goldsmith November 9, 2001

2. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

3. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

4. Unsupervised learning • Input: untagged text in orthographic or phonetic form • with spaces (or punctuation) separating words. • But no tagging or text preparation.

5. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

6. Overview of program and output • Linguistica: a C++ Windows-based program available for download at http://humanities.uchicago.edu/faculty/goldsmith/Linguistica2000 • Technical discussion in Computational Linguistics (June 2001) • Good results with 5,000 words, very fine-grained results with 500,000 words (corpus length, not lexicon count).

7. Output • List of stems, suffixes, and prefixes • List of signatures. • A signature: a list of all suffixes (prefixes) appearing in a given corpus with a given stem. • Hence, a stem in a corpus has a unique signature. • A signature has a unique set of stems associated with it • …

8. (example of signature in English) • NULL.ed.ing.s ask call point = ask asked asking asks call called calling calls point pointed pointing points

9. …output • Roots (“stems of stems”) and the inner structure of stems • Regular allomorphy of stems: e.g., learn “delete stem-final –e in English before –ing and –ed”

11. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

12. Minimum Description Length (MDL) • Jorma Rissanen: Stochastic Complexity in Statistical Inquiry (1989) • Work by Michael Brent and Carl de Marcken on word-discovery using MDL

13. Essence of MDL We are given • a corpus, and • a probabilistic morphology, which technically means that we are given a distribution over certain strings of stems and affixes. (“Given”? Given by who? We’ll get back to that.) (Remember: a distribution is a set of non-negative numbers summing to 1.0.)

14. The higher the probability is that the morphology assigns to the (observed) corpus, the better that morphology is as a model of that data. • Better said: -1 * log probability (corpus) is a measure of how well the morphology models the data: the smaller that number is, the better the morphology models the data. This is known as the optimal compressed length of the data, given the model. Using base 2 logs, this number is a measure in information theoretic bits.

15. Essence of MDL… • The goodness of the morphology is also measured by how compact the morphology is. • We can measure the compactness of a morphology in information theoretic bits.

16. How can we measure the compactness of a morphology? • Let’s consider a naïve version of description length: count the number of letters. • This naïve version is nonetheless helpful in seeing the intuition involved.

17. Naive Minimum Description Length Corpus: jump, jumps, jumping laugh, laughed, laughing sing, sang, singing the, dog, dogs total: 62 letters Analysis: Stems: jump laugh sing sang dog (20 letters) Suffixes: s ing ed (6 letters) Unanalyzed: the (3 letters) total: 29 letters. Notice that the description length goes UP if we analyze sing into s+ing

18. Essence of MDL… The best overall theory of a corpus is the one for which the sum of • log prob (corpus) + • length of the morphology (that’s the description length) is the smallest.

19. Essence of MDL…

20. Overall logic • Search through morphology space for the morphology which provides the smallest description length.

21. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

22. Corpus Pick a large corpus from a language -- 5,000 to 1,000,000 words.

23. Corpus Feed it into the “bootstrapping” heuristic... Bootstrap heuristic

24. Corpus Bootstrap heuristic Out of which comes a preliminary morphology, which need not be superb. Morphology

25. Corpus Bootstrap heuristic Feed it to the incremental heuristics... Morphology incremental heuristics

26. Corpus Out comes a modified morphology. Bootstrap heuristic Morphology modified morphology incremental heuristics

27. Corpus Is the modification an improvement? Ask MDL! Bootstrap heuristic Morphology modified morphology incremental heuristics

28. Corpus If it is an improvement, replace the morphology... Bootstrap heuristic modified morphology Morphology Garbage

29. Corpus Send it back to the incremental heuristics again... Bootstrap heuristic modified morphology incremental heuristics

30. Continue until there are no improvements to try. Morphology modified morphology incremental heuristics

31. 1. Bootstrap heuristic • A function that takes words as inputs and gives an initial hypothesis regarding what are stems and what are affixes. • In theory, the search space is enormous: each word w of length |w| has at least |w| analyses, so search space has at least members.

32. Better bootstrap heuristics Heuristic, not perfection! Several good heuristics. Best is a modification of a good idea of Zellig Harris (1955): Current variant: Cut words at certain peaks of successor frequency. Problems: can over-cut; can under-cut; and can put cuts too far to the right (“aborti-” problem). [Not a problem!]

33. Successor frequency n g o v e r Empirically, only one letter follows “gover”: “n”

34. Successor frequency e i m g o v e r n o s # Empirically, 6 letters follows “govern”: “n”

35. Successor frequency g o v e r n m e Empirically, 1 letter follows “governm”: “e” g o v e r 1 n 6 m 1 e peak of successor frequency

36. Lots of errors… 9 18 11 6 4 1 2 1 1 2 1 1 c o n s e r v a t i v e s wrong right wrong

37. Even so… We set conditions: Accept cuts with stems at least 5 letters in length; Demand that successor frequency be a clear peak: 1… N … 1 (e.g. govern-ment) Then for each stem, collect all of its suffixes into a signature; and accept only signatures with at least 5 stems to it.

38. 2. Incremental heuristics Course-grained to fine-grained • 1. Stems and suffixes to split: • Accept any analysis of a word if it consists of a known stem and a known suffix. • 2. Loose fit: suffixes and signatures to split: Collect any string that precedes a known suffix. • Find all of its apparent suffixes, and use MDL to decide if it’s worth it to do the analysis. We’ll return to this in a moment.

39. Incremental heuristic • 3.Slide stem-suffix boundary to the left: Again, use MDL to decide. How do we use MDL to decide?

40. Using MDL to judge a potential stem act, acted, action, acts. We have the suffixes NULL, ed, ion, and s, but no signature NULL.ed.ion.s Let’s compute cost versus savings of signature NULL.ed.ion.s Savings: Stem savings: 3 copies of the stem act: that’s 3 x 4 = 12 letters = almost 60 bits.

41. Cost of NULL.ed.ing.s • A pointer to each suffix: To give a feel for this: Total cost of suffix list: about 30 bits. Cost of pointer to signature: total cost is -- all the stems using it chip in to pay for its cost, though.

42. Cost of signature: about 45 bits • Savings: about 60 bits so MDL says: Do it! Analyze the words as stem + suffix. Notice that the cost of the analysis would have been higher if one or more of the suffixes had not already “existed”.

43. Today’s presentation • The task: unsupervised learning • Overview of program and output • Overview of Minimum Description Length framework • Application of MDL to iterative search of morphology-space, with successively finer-grained descriptions • Mathematical model • Current capabilities • Current challenges

44. Model • A model to give us a probability of each word in the corpus (hence, its optimal compressed length); and • A morphology whose length we can measure.

45. Frequency of analyzed word W is analyzed as belonging to Signature s,stem T and suffix F. [x] means the count of x’s in the corpus (token count) Where [W] is the total number of words. Actually what we care about is the log of this:

47. Next, let’s see how to measurethe length of a morphology A morphology is a set of 3 things: • A list of stems; • A list of suffixes; • A list of signatures with the associated stems. We’ll make an effort to make our grammars consist primarily of lists, whose length is conceptually simple.

48. Length of a list • A header telling us how long the list is, of length (roughly) log2 N, where N is the length. • N entries. What’s in an entry? • Raw lists: a list of strings of letters, where the length of each letter is log2 (26) – the information content of a letter (we can use a more accurate conditional probability). • Pointer lists:

49. Raw suffix list: ed s ing ion able … Signature 1: Suffixes: pointer to “ing” pointer to “ed” Signature 2: Suffixes pointer to “ing” pointer to “ion” Lists The length of each pointer is -- usually cheaper than the letters themselves

50. The fact that a pointer to a symbol has a length that is inversely proportional to its frequency is the key: • We want the shortest overall grammar; so • That means maximizing the re-use of units (stems, affixes, signatures, etc.)