620 likes | 738 Views
Confidence Measures for Automatic Speech Recognition. National Taiwan Normal University Spoken Language Processing Lab. Advisor : Hsin-Min Wang Berlin Chen. Presented by Tzan-Hwei Chen. Outline. Introduction The category of estimation methods of confidence measure (CM)
E N D
Confidence Measures for Automatic Speech Recognition National Taiwan Normal University Spoken Language Processing Lab Advisor : Hsin-Min Wang Berlin Chen Presented by Tzan-Hwei Chen
Outline • Introduction • The category of estimation methods of confidence measure (CM) • Featured based • Posterior probability based • Explicit model based • Incorporation of high-level information for CM* • The application of CM to improve speech recognition • Summary
Introduction (1/9) • It is extremely important to be able to make an appropriate and reliable judgement based on the error-prone ASR result. • Researchers have proposed to compute a score (preferably 0~1), called confidence measure (CM), to indicate reliability of any recognition decision made by an ASR system.
Introduction (2/9) Some application of CM Confidence Measure Lexicon 1 2 Feature extraction Decoding Verification feature vector recognized word sequence speech signal 臺北 到 魚籃 Acoustic model Language model 1.臺北到魚籃 2.臺北到宜蘭
Introduction (3/9) • First of all, we can backtrack some early research on CM to rejection in word-spotting systems. • Other early CM-related works lie in automatic detection of new words in LVCSR. • From the past few years, the CM has been applied to more and more research areas, e.g., • To improve speech recognition • The algorithm about look-head in LVCSR • To guide the system to perform unsupervised learning • …
Introduction (4/9) • The general procedure of CM for verification Predefined threshold Recognized units Confidence estimation Confidence of unit judgment > threshold < threshold rejection acceptance
魚籃 宜蘭 宜蘭 魚籃 宜蘭 宜蘭 宜蘭 宜蘭 ref hyp hyp ref hyp ref ref hyp Introduction (5/9) • Four situations when judging a hypothesis Accept Correct acceptance reject Correct rejection reject false rejection Accept false acceptance
Introduction (6/9) • The evaluation metric : • Confidence error rate : FA CA FR CA FA 三民 候選人 通過 審查 了 hyp 有 三名 候選人 通過 審查 ref
Introduction (7/9) • The evaluation metric : • Confidence error rate : FA CA CA CA FA 三民 候選人 通過 審查 了 hyp 有 三名 候選人 通過 審查 ref
Introduction (8/9) • The evaluation metric (cont): • Receiver operator characteristics (ROC) curve :simply contains a plot of the false acceptance rate over the detection rate.
Introduction (9/9) • All methods proposed for computing CMs can be roughly classified into three major categories [7]: • Feature based • Posterior probability based • Explicit model based (utterance verification, UV) • Incorporation of high-level information for CM*
Feature-based confidence measure (1/8) • The feature can be collected during the decoding procedure and may include acoustic, language and syntactic information • Any feature can be called a predictor if its p.d.f. of correctly recognized words is clearly distinct from that of misrecognized words misrecognized word correctly recognized word
Feature-based confidence measure (2/8) • Some common predictor features • Pure normalized likelihood score related : acoustic score per frame. • N-best related : count in the N-best list, N-best homogeneity score • Duration related : word duration divided by its number of phones
Feature-based confidence measure (3/8) • Some common predictor features (cont) • Hypothesis density : 三名 候選人 三名 有 三名 通過 候選人 由 結果 沒有 審查 靜音 沒有 候選人 沒有 審查 建國 候選人 通過 又 候選人 三名
今天 天氣 今天 天氣 不佳 今天 天氣 很好 Hypothesized word sequence 今天 天氣 很好 今天 天氣 Hypothesized word sequence 今天 天氣 不佳 Feature-based confidence measure (4/8) • Some common predictor features (cont) • Acoustic stability 天氣 很好 今天 Hypothesized word sequence
Feature-based confidence measure (6/8) • We can combine the above features with any one of the following classifiers • Line discriminant function • Generalized linear model • Neural networks • Decision tree • Support vector machine • Boosting • Naïve Bayes classifier
Feature-based confidence measure (7/8) • Naïve Bayes Classifier [3]
Feature-based confidence measure (8/8) • Experiments [3] • Corpus : an Italian speech corpus of phone calls to the front desk of a hotel
Posterior probability based confidence measure (1/11) • Posterior probability of a word sequence : • To adopt some approximation methods Impossible to estimate in a precise manner
Posterior probability based confidence measure (2/11) • Word graph based approximation 三名 候選人 有 三名 三名 候選人 由 靜音 結果 沒有 三名 又 靜音 靜音 沒有 候選人 沒有 通過 建國 有 靜音 三名 候選人 又 通過 候選人 三名
Posterior probability based confidence measure (3/11) • Posterior probability of a word arc : • Some issues are addressed and the word posterior probability is generalized • Reduced search space • Relaxed time registration • Optimal acoustic and language model weights
Posterior probability based confidence measure (4/11) • Posterior probability of a word arc [6] : 三名 候選人 有 三名 三名 由 候選人 靜音 結果 沒有 三名 又 靜音 沒有 候選人 靜音 沒有 通過 建國 靜音 有 三名 候選人 又 通過 候選人 三名
Posterior probability based confidence measure (5/11) • Posterior probability of a word arc [6] : 三名 候選人 有 三名 三名 由 候選人 靜音 結果 沒有 三名 又 靜音 沒有 候選人 靜音 沒有 通過 建國 靜音 有 三名 候選人 又 通過 候選人 三名
Posterior probability based confidence measure (6/11) • Posterior probability of a word arc [6] : 三名 候選人 三名 有 三名 由 候選人 靜音 結果 沒有 三名 又 靜音 沒有 候選人 靜音 沒有 通過 建國 靜音 有 三名 候選人 又 通過 候選人 三名
Posterior probability based confidence measure (7/11) • Posterior probability of a word arc [6] : 三名 候選人 有 三名 三名 由 候選人 靜音 結果 沒有 三名 又 靜音 沒有 候選人 靜音 沒有 通過 建國 靜音 有 三名 候選人 又 通過 候選人 三名
Posterior probability based confidence measure (8/11) • The drawbacks of the above methods – all need an additional pass. • In [8], the “local word confidence measure” is proposed 今天 今天 今天 今天
bigram applied forward/backward bigram applied Posterior probability based confidence measure (8/11) • local word confidence measure (cont)
Posterior probability based confidence measure (9/11) • Impact of word graph density on the quality of posterior probability [9] Baseline 27.3 15.4
Posterior probability based confidence measure (10/11) • Experiments [6]
Explicit model based confidence measure (1/10) • The CM problem is formulated as a statistical hypothesis testing problem. • Under the framework of binary hypothesis testing, there are two complementary hypotheses • We test against
Explicit model based confidence measure (3/10) • The above LRT score can be transformed to a CM based on a monotonic 1-1 mapping function. • The major difficulty with LRT is how to model the alternative hypothesis. • In practice, the same HMM structure is adopted to model the alternative hypothesis. • A discriminative training procedure plays a crucial role in improving modeling performance.
Explicit model based confidence measure (3/10) • Two-pass procedure : 天氣 很好 今天
Explicit model based confidence measure (4/10) • One-pass procedure 天氣 很好 今天
Explicit model based confidence measure (5/10) • How to calculate the confidence of a recognized word?
Explicit model based confidence measure (6/10) • How to calculate the confidence of a recognized word (cont)?
Explicit model based confidence measure (7/10) • Discriminative training [10] • The goal of the training procedure is to increase the average value of for correct hypotheses and decrease the average value of for false acceptance.
Explicit model based confidence measure (8/10) • Discriminative training (cont)
Explicit model based confidence measure (9/10) Why discriminative training works?
Explicit model based confidence measure (10/10) • Experiments [10] • This task, referred to as the “movie locator”,
U A Incorporation of high-level information for CM (1/4) • LSA • The key property of LSA is that words whose vectors are close to each other are semantically similar words. • These similarities can be used to provide an estimate of the likelihood of the words co-occurring within the same utterance.
Incorporation of high-level information for CM (2/4) • LSA (cont) • The entry of matrix : • The confidence of a recognized word :
Incorporation of high-level information for CM (3/4) • Inter-word mutual information :
Incorporation of high-level information for CM (4/4) • Experiments [14]
三名 候選人 有 三名 三名 候選人 由 靜音 結果 沒有 三名 又 靜音 靜音 沒有 候選人 沒有 通過 建國 有 靜音 三名 候選人 又 通過 候選人 三名 The application of CM to improve speech recognition (1/10) • Statistical decision theory aims at minimizing the expected of making error
The application of CM to improve speech recognition (2/10) • Method 1 [16]:
The application of CM to improve speech recognition (3/10) • Method 2 [18] :