CMU-Statistical Language Modeling & SRILM Toolkits

1. CMU-Statistical Language Modeling & SRILM Toolkits Ashwin Acharya Ece 5527 Search and Decoding Instructor: Dr. v.Këpuska

2. Objective Use the SRILM and CMU (Carnegie Mellon University) toolkits to built differing language models. Use the four Discounting Strategies to build the language models. Perform perplexity tests with both LM toolkits to study each ones performance.

3. CMU-SLM Toolkit • The CMU SLM Toolkit is a set of Unix software tools created to aid statistical language modeling. • The key tools provided are used to process textual data into: • Vocabularies and word frequency lists • Bigram and trigram counts • N-gram related statistics • Various N-gram language model • Once the language models are created they can be used to compute: • Perplexity • Out-of-vocabulary rate • N-gram hit ratios • Back off distributions

4. SRILM Toolkit • SRILM toolkit is used for building and applying statistical language models (LMs), primarily for use in speech recognition, statistical tagging and segmentation, and machine translation • SRILM consists of the following components: • A set of C++ class libraries implementing language models, supporting data structures and miscellaneous utility functions. • A set of executable programs built on top of these libraries to perform standard tasks such as training LMs and testing them on data, tagging or segmenting text. • A collection of miscellaneous scripts facilitating minor related tasks. • SRILM runs on UNIX platforms

5. Toolkit Environment The two toolkits run in a Unix Environment like Linux However, Cygwin works well and can be used to simulate this Unix environment on a windows platform. Download free Cygwin form following link: http://www.cygwin.com/

6. Cygwin Install Download the Cygwin installation file Execute setup.exe Choose “Install from Internet” Select root install directory “ C:\cygwin” Choose a download site “mirrors”

7. Cygwin Install • Make sure all of the following packages are selected during the install process: • gcc: the compiler • GNU make: utility, which determines automatically which pieces of a large program need to be recompiled, and issues the commands to recompile them. • TCL toolkit: tool command language • Tcsh: TENEX C shell • gzip: to read/write compressed file • GNU awk(gawk): to interpret many of the utility script • Binults: GNU assembler, linker and binary utilities

8. SRILM Install Download SRILM toolkit, srilm.tgz Run Cygwin Unzip srilm.tgz by following commands: $ cd /cygdrive/c/cygwin/srilm $ tarzxvfsrilm.tgz Add following lines to setup the direction in the makefile: SRILM=/cygdrive/c/cygwin/srilm MACHINE_TYPE=cygwin Run Cygwin and type following command to install SRILM: $ Make World

9. CMU-SLM Install Download CMU toolkit,CMU-Cam_Toolkit_v2.tgz Run Cygwin Unzip srilm.tgz by following commands: $ cd /cygdrive/c/cygwin/CMU $tarzxvf CMU-Cam_Toolkit_v2.tgz Run Cygwin and type following command to install CMU: $ Make All

10. Training Corpus Use the corpus provided by National Language Toolkit website. http://en.sourceforge.jp/projects/sfnet_nltk/downloads/nltk-lite/0.9/nltk-data-0.9.zip/ Containing a lexicon and two different corporafrom the Australian Broadcasting Commission. The first is a Rural News based corpora The second is a Science News based corpora

11. SRILM Procedure Generate 3-gram count file by using following commands: $./ngram-count-vocababc/en.vocab -textabc/rural.txt -order 3 -writeabc/rural.count -unk

12. Count File ngram-count count N-grams and estimate language models -vocab file Read a vocabulary from file. Subsequently, out-of-vocabulary words in both counts or text are replaced with the unknown-word token. If this option is not specified all words found are implicitly added to the vocabulary. -texttextfile Generate N-gram counts from text file. textfileshould contain one sentence unit per line. Begin/end sentence tokens are added if not already present. Empty lines are ignored. -order n Set the maximal order (length) of N-grams to count. This also determines the order of the estimated LM, if any. The default order is 3. -write file Write total counts to file. -unk Build an “open vocabulary” LM, i.e., one that contains the unknown-word token as a regular word. The default is to remove the unknown word.

13. SRILM Procedure Creating the language model : $ ./ngram-count -readabc/rural.count -order 3 -lmabc/rural.lm -readcountsfile Read N-gram counts from a file. ASCII count files contain one N-gram of words per line, followed by an integer count, all separated by whitespace. Repeated counts for the same N-gram are added. -lmlmfile Estimate a backoff N-gram model from the total counts, and write it to lmfile.

14. Discounting Good-Turing Discount $ ./ngram-count -readabc/rural.count -order 3 -lmabc/gtrural.lm -gt1min 1 -gt1max 3 -gt2min 1 -gt2max 3 -gt3min 1 -gt3max 3 -gtnmin count where n is 1, 2, 3, 4, 5, 6, 7, 8, or 9. Set the minimal count of N-grams of order n that will be included in the LM. All N-grams with frequency lower than that will effectively be discounted to 0. If n is omitted the parameter for N-grams of order > 9 is set. NOTE: This option affects not only the default Good-Turing discounting but the alternative discounting methods described below as well. -gtnmax count where n is 1, 2, 3, 4, 5, 6, 7, 8, or 9. Set the maximal count of N-grams of order n that are discounted under Good-Turing. All N-grams more frequent than that will receive maximum likelihood estimates. Discounting can be effectively disabled by setting this to 0. If n is omitted the parameter for N-grams of order > 9 is set.

15. Discounting Absolute Discounting $ ./ngram-count -readabc/test1.txt -order 3 -lmabc/ruralcd.lm -cdiscount1 0.5 -cdiscount2 0.5 -cdiscount3 0.5 -cdiscountn discount where n is 1, 2, 3, 4, 5, 6, 7, 8, or 9. Use Ney's absolute discounting for N-grams of order n, using discount as the constant to subtract. • Witten-Bell $ ./ngram-count -readabc/test1.txt -order 3 -lmabc/ruralwb.lm -wbdiscount1 -wbdiscount2 -wbdiscount3 • -wbdiscountn where n is 1, 2, 3, 4, 5, 6, 7, 8, or 9. Use Witten-Bell discounting for N-grams of order n. (This is the estimator where the first occurrence of each word is taken to be a sample for the “unseen” event.)

16. Discounting Kneser-Ney Discounting $ ./ngram-count -read project/count.txt -order 3 -lm knlm.txt -kndiscount1 -kndiscount2 -kndiscount3 -kndiscountn where n is 1, 2, 3, 4, 5, 6, 7, 8, or 9. Use Chen and Goodman's modified Kneser-Ney discounting for N-grams of order n.

17. Perplexity Perplexity is the common evaluation metric for N-gram language models. It is relative cheaper than the end-to-end evaluation which requires to test the recognition results of language models. The perplexity is defined as following: The better language model will assign a higher probability to the test data which lower the perplexity. As the result, the lower the perplexity means the higher probability of the test set according to that specific language model. Normalized by number of words. probability of the test set

18. Test Perplexity Randomly choose three articles of news from Internet for test data. Commands for four different 3-gram language Models: $ ./ngram-pplabc/test1.txt -order 3 -lmproject/gtrural.lm $ ./ngram-pplabc/test1.txt -order 3 -lmproject/cdrural.lm $ ./ngram-pplabc/test1.txt -order 3 -lmproject/wbrural.lm $ ./ngram-pplabc/test1.txt -order 3 -lmproject/knrural.lm Repeat these tests for all three test data with the other language models

19. Results of Perplexity Test SRILM Perplexity test with test2.txt with language model gtrural.lm ( rural corpora with Good-Turing Discounting)

20. Building LM with CMU SLM toolkit The following is a simple flow chart which explains the steps to build a language model using CMU language model toolkit.

21. Building LM with CMU SLM toolkit • text2wfreq: compute the word unigram counts. • wfreq2vocab: convert the word unigram counts into a vocabulary consisting of the preset number of common words. • text2idngram: Convert the idngram into a binary format language model. • idngram2lm: Build the language model. • evallm: compute the perplexity of the test corpus.

22. CMU SLM Procedure • Given a large corpus of text in a file a.text, but no specified vocabulary • Compute the word unigram counts cat a.text | text2wfreq > a.wfreq • Convert the word unigram counts into a vocabulary consisting of the 20,000 most common words cat a.wfreq | wfreq2vocab -top 20000 > a.vocab • Generate a binary id 3-gram of the training text, based on this vocabularycat a.text | text2idngram -vocab a.vocab > a.idngram • Convert the idngram into a binary format language model idngram2lm -idngrama.idngram -vocab a.vocab -binary a.binlm • Compute the perplexity of the language model, with respect to some test text b.textevallm -binary a.binlmReading in language model from file a.binlmDone.evallm : perplexity -text b.textComputing perplexity of the language model with respect to the text b.textPerplexity = 128.15, Entropy = 7.00 bits Computation based on 8842804 words. Number of 3-grams hit = 6806674 (76.97%) Number of 2-grams hit = 1766798 (19.98%) Number of 1-grams hit = 269332 (3.05%) 1218322 OOVs (12.11%) and 576763 context cues were removed from the calculation. evallm : quit

23. Perplexity Test Results CMU Perplexity test with test1.txt with language model ruralgt.lm ( rural corpora with Good-Turing Discounting)

24. Perplexity Results

25. Results Perplexity tests based on the language model from the Rural News Corpora

26. Results Perplexity tests based on the language model from the Science News Corpora

27. Conclusion It can clearly be seen the SRILM toolkit performed the better that the CMU toolkit in terms of the perplexity tests. We can also notice that for specific test corpora certain discounting strategies were better suited that the others. For instance the Kneser-Ney discounting seemed to perform relatively better than the others on most of the tests. The CMU language models perplexity results seemed to be a little on the higher side a fix could be to adjust the discounting ranges. Results could have been improved by using some sort of word tokenization.

28. References • SRI International, “The SRI Language Modeling Toolkit”, http://www.speech.sri.com/projects/srilm/ • Cygwin Information and Installation, “Installing and Updating Cygwin”, http://www.cygwin.com/ • National Language Toolkit, http://nltk.sourceforge.net/index.php/Corpora • Carnegie Mellon University, CMU Statistical language Model Toolkit, http://www.speech.cs.cmu.edu