Minimum Classification Error (MCE) Approach in Pattern Recognition

1. Minimum Classification Error (MCE) Approach in Pattern Recognition Wu Chou, Avaya Labs Research, Avaya Inc., USA

2. Outline (1/2) • Introduction • Optimal Classifier from Bayes Desicion Theory • Discriminant Function Approach to Classifier Design • Speech Recogniation and Hidden Markov Modeling • Hidden Markov Modeling of Speech • MCE Classifier Design Using Discriminant Functions • MCE Classifier Design Strategy • Optimization Methods • Other Optimization Methods • HMM as a Discriminant Function • Relation Between MCE and MMI • Discussions and Comments

3. Outline (2/2) • MCE TRAINING BASED ON EMBEDDED STRING MODEL • String-Model-Based MCE Approach • Combined String-Model-Based MCE Approach • Discriminative Language Model Estimation • SUMMARY

4. Introduction • The advent of powerful computing devices and success of statistical approaches • A renewed pursuit for more powerful method to reduce recognition error rate • Although MCE-based discriminative methods is rooted in the classical Bayes’ decision theory, instead of a classification task to distribution estimation problem, it takes a discriminant-function based statistical pattern classification approach • For a given family of discriminant function, optimal classifier/recognizer design involves finding a set of parameters which minimize the empirical pattern recognition error rate

5. Introduction • Why we take this approach to design classifier? • We lack complete knowledge of the form of the distribution • Training data are inadequate • How to do? • Formulating the problem of self-learning into a classification problem which consists of optimal partitioning of the observation space into regions, Xk, for which the expected risk , R, is minimized • Then we apply generalized probabilistic decent algorithm to achieve the goal

6. Optimal Classifier from Bayes Desicion Theory C1 C2 CM random 要分類 : x 不確定是 Ci，但被分到 Ci 的機率 但，我們並不知道標準答案

7. 定義 loss function : 可以想成 Class i 與 Class j 的 distance, 將 Class i 的observation分到 Class j，分錯的 cost 假設 Class i 是正確答案， 則將 x 分錯而得到的cost之expectation (1) Optimal Classifier from Bayes Desicion Theory

8. Optimal Classifier from Bayes Desicion Theory 當我們作決定時 雖然我們並不知道正確的答案，但可算出作此決定需付出的代價 (2) 如何作出較正確的決定？ 雖然不知道正確答案，但付出的代價愈小，則愈正確 【Decision Rule】 (3)

9. Optimal Classifier from Bayes Desicion Theory 在SR及許多application中，我們常用的 loss function Posterior Probability (5) 所以【Decision Rule】可以改寫 Bayes’ risk MAP decision (6)

10. Optimal Classifier from Bayes Desicion Theory OK!! 若 Posterior Probability知道，一切好辦  over 但一般來說，Posterior Probability 需有已知 class 的 labeled training data來估測 (這是不容易取得的) 本來是classifier design的問題  distribution estimation problem 由Bayes’ Theorem estimate the a posterior probabilities for any to implement the maximum a posterior decision for minimum Bayes risk (7) 可省略！

11. Optimal Classifier from Bayes Desicion Theory • 三個 issue： • Classifier Designed 必需正確估算distribution的parameters，但是，real-world中，distribution常為了容易處理而妥協，使用較簡單或較容易作運算的distribution 如：Gaussian • Real-world中，distribution的parameter一定由『有限』的 training data set來估算，但這需要一個大前題：當training data set 的size改變時，訓練出來的parameter要能保持一致 • unachievable • 否則，則需要一定數量的 training data set 來使parameter較為可信賴，但由於data sparse • unachievable

12. Optimal Classifier from Bayes Desicion Theory • Despite the conceptual optimality of the Bayes decision theory and its applications to pattern recognition, it can’t always be accomplished in practice • Most practical “MAP” decisions in speech and language processing are not true MAP decisions

13. Discriminant Function Approach to Classifier Design 先只考慮 2-class 定義 discriminant function 分類用 One well-studied family of discriminant function is the Linear discriminant functionwhich has computational advantages (9)

14. Discriminant Function Approach to Classifier Design More generally (10) (11)

15. Discriminant Function Approach to Classifier Design 再來考慮 M-class (12) 也就是說，我們要一組『最佳discriminant functions』 (13) When the loss function is specified

16. Discriminant Function Approach to Classifier Design This is quite different from the distribution estimation based approach in pattern classification

17. Score from Acoustic Model Word Sequence Acoustic Feature Score from Language Model Best Word Sequence Speech Recogniation and Hidden Markov Modeling • A decoder performs a maximum a posterior decision

18. Speech Recogniation and Hidden Markov Modeling • Basic components: • Acoustic Feature Extraction： • Used to extract the features from waveform. • We use to represent the acoustic observation feature vector sequence. • Acoustic Modeling： • Provides statistical modeling for the acoustic observation X. • Hidden Markov Model is the prevalent choice. • Language Modeling： • Provides linguistic constraints to the text sequence W. • Based on statistical N-gram language models

19. Speech Recogniation and Hidden Markov Modeling • Decoding Engine： • Search for the best word sequence given the feature and model • This is achieved through Viterbi decoding Discrete observation Probability Word String State Sequence Continuous density HMMs

20. Speech Recogniation and Hidden Markov Modeling • Hidden Markov modeling is a powerful statistical framework for time-varying quasi-stationary process and a popular choice for statistical modeling of speech signal

21. SPEECH RECOGNITION AND HIDDEN MARKOV MODELING • Three basic problems have to be resolved： • The evaluation problem • estimate the probability • The decoding problem • find a best state sequence q • The estimation problem • estimate HMM parameters from a given set of training samples(ML based algorithms such as Baum-Welch al.)

22. MCE Classifier Design Using Discriminant Functions (19) MCE classifier design based on 3 steps

23. MCE Classifier Design Using Discriminant Functions • Misclassification measure (20) Generally we use

24. MCE Classifier Design Using Discriminant Functions • <proof>

25. MCE Classifier Design Using Discriminant Functions • Loss function (21) (22)

26. MCE Classifier Design Using Discriminant Functions • Classifier Performance Measure (23) (24)

27. MCE Classifier Design Using Discriminant Functions If posterior probability is used Then the Bayes’ minimum risk is (25) X 在 Class k 的機率不可最大，也就是說分錯的 loss

28. MCE Classifier Design Using Discriminant Functions If posterior probability is used Then the Bayes’ minimum risk is (26) Empirical loss

29. Optimization Methods • Expected Loss (27) We use GPD-based minimization algorithm to minimize it (28)

30. Optimization Methods 若滿足下面三個properties，則 收斂

31. Optimization Methods • Empirical Loss (31) (32)

32. HMM as a Discriminant Function 使用HMM當作discriminant function (34) discriminant function利用 有三種方式來產生 (35) (36) (37)

33. HMM as a Discriminant Function

34. HMM as a Discriminant Function 假設 Maintain HMM 原有的constraints

35. HMM as a Discriminant Function 所以我們使用parameter transformation來保留這些constraints

36. HMM as a Discriminant Function , discriminant adjustment of the mean vector

37. HMM as a Discriminant Function

38. HMM as a Discriminant Function

39. HMM as a Discriminant Function , discriminant adjustment of the variance

40. HMM as a Discriminant Function

41. HMM as a Discriminant Function

42. HMM as a Discriminant Function

43. HMM as a Discriminant Function • How to design the step size? • If the step size is too large, the classifier will be degraded at the start and sequential learning cannot be made successful • If the step size is too small, the convergence speed of the algorithm is too slow and it is practically not useful • It’s difficult to design it, the general solution is still lacking

44. HMM as a Discriminant Function • Why we normalize mean vector? • The magnitude of variances can vary in the range between 100 and 10-5 . • If using a constant step size for all mean vectors, the algorithm will either not converge or will be too slow to become practically useless • This takes away the dependencies on the variance variations

45. Relation between MCE and MMI

46. Relation between MCE and MMI

47. Relation between MCE and MMI

48. Relation between MCE and MMI

49. Relation between MCE and MMI

50. Relation between MCE and MMI