ICASSP 05

1. ICASSP 05

2. Reference • Rapid Language Model Development Using External Resources for New Spoken Dialog Domains • Ruhi Sarikaya1, Agustin Gravano2, Yuqing Gao1 • 1IBM, 2Columbia University • Maximum Entropy Based Generic Filter for Language Model Adaptation • Dong Yu, Milind Mahajan, PeterMau, Alex Acero • Microsoft • Language Model Estimation for Optimizing End-to End Performance of A Natural Language Call Routing System • Vaibhava Goel, Hong-Kwang Jeff Kuo, Sabine Deligne, Cheng Wu • IBM

3. Introduction • LM adaptation consists of four steps • 1. Collection of task specific adaptation data • 2. Normalization step • Abbreviations, data and time, punctuations • 3. Analyze adaptation data and build a task specific LM • 4. Interpolate task specific LM with task independent LM

4. Introduction • Language Modeling research concentrated in tow directions • 1. Improving the language model probability estimation • 2. Obtaining additional training material • The largest data set is the World Wide Web (WWW) • More than 4 billion pages

5. Introduction • Using web data for language modeling • Query generation • Filtering the relevant text from the retrieved pages • The web counts are certainly less sparse than the counts in a corpus of a fixed size • The web counts are also likely to be significantly more noisy than counts obtained from a carefully cleaned and normalized corpus • Retrieve unit • Whole document v.s. sentence (utterance)

6. Build LM for new domain • In practice when we start to build an SDS (spoken dialog system) for a new domain, the amount of in-domain data for the target domain is usually small • Definition • Static resource: corpora collected for other tasks • Dynamic resource: web data

7. Flow diagram for the collecting relevant data

8. Generating search Queries • Using Google as search engine • The more specific a query is the more relevant the retrieved pages are.

9. Similarity based sentence selection • Machine translation’s BLEU (BiLingual Evaluation Understudy) • N is the maximum n-gram length, wn and pn are the corresponding weight and precision, respectively, BP is the brevity penalty • Where r and c are the lengths of the reference and candidate sentences, respectively Threshold is 0.08

10. Experimental result • SCLM: using static corpora for language model • WWW-20 / WWW-100: predefined limit to 20 / 100 pares per sentence

11. E-mail corpus • Dictated and non-dictated

12. Filtering the corpus • Filtering out these non-dictated texts is not an easy job in general • Hand-crafted rules (e.g. regular expressions) • Limitations • It does not generalize well to situations which we have not encountered • Rules are usually language dependent • Developing and testing rules are very costly

13. Maximum Entropy based filter • Consider the filtering task as a labeling problem to segment the adaptation data into two categories: • Category D (Dictated text): • Text which should be used for LM adaptation • Category N (Non-dictated text): • Text which should not be used for LM adaptation • Text is divided into a sequence of text units (such as lines) ti is the text unit, and li is the label associated with ti

14. Label dependency • Assume that the labels of text units are independent with each other given the complete sequence of text units • We further assume that the label for a given unit depends only upon units in a surrounding window of units • k = 1:

15. Classification Model • A MaxEnt model has the form: where is the vector of model parameters

16. Classification Model • Pthresh = 0.5

17. Features

18. Space Splitting

19. Evaluation • Uses only features RawCompact, EOS, and OOV • No space splitting

20. Filtering is especially important and effective for the adaptation data with high percentage of non-dictated text (U2)

21. Efficient linear combination for distant n-gram models David Langlois, Kamel Smaili, Jean-Paul Haton EUROSPEECH 2003 p409~412

22. Introduction • Classical n-gram model • Distant language models

23. Modelization of distance in SLM • Cache model (self-relationship) • The former deals with the self-relationship between a word present in the history and itself: if a word is frequent in the history, it has more chance to appear once again

24. Modelization of distance in SLM (cont.) • Trigger model • the relationship between two words • It deals with couple of words v → w such that if v (the triggering word) is in the history, w (the triggered word) has more chance to appear • But, in fact, the majority of triggers are self triggers (v → v): a word triggers itself

25. d-n-gram model • Nd(.) is the discounted count • 0-n-gram model is the classical n-gram model

26. Evaluation • Voc: 20k words • Training set: 38M words • Development set: 2M words • Test set: 2M words • Baseline classical n-gram models:

27. Integration of distant n-gram models • Distant n-gram model cannot be used alone. It takes into account only a part of the history • Perplexity is 717 for n=2 and d=4 • Several models with distance up to d are combined with the baseline model

28. Improvement 7.1% Improvement 3.1% The utility of distant n-gram models decreases with the distance: a distance greater than 2 does not provide more information

29. Distant trigram lead to an improvement, but it is less important than in distant bigram. overlap between the history of d-trigram and (d+1)-trigram

30. Backoff smoothing b_u_z db_u_z (b_u_z)˙(db_u_z)

31. 7.9% 11.6%

32. Combination weight • Unique weight • The model’s weights depend on each history (the class of each sub-history)

33. Combination of distant n-gram • In order to combine K models, M1,…,MK, a set of weight 1,…,K is defined and the combination is expressed by: • Development corpus is not sufficient to estimate a huge number of parameters • Classify histories and set a weight to each class

34. Classification • Break the history into the several parts (sub-histories). Each sub-history is analyzed in order to estimate its importance in terms of prediction and then put into a class • Such a class is directly linked to the value of the sub-history frequency • This class gathers all sub-histories which have approximately the same frequency

35. 8000 classes/115.4 12.8% improvement to baseline (132.4) 5.3% improvement to the single weight combination (121.9) 4000 classes/85.2 12.8% improvement to baseline (97.8) 1.5% improvement to the single weight combination (86.5)