Phone-Level Pronunciation Scoring and Assessment for Interactive Language Learning

1. Phone-Level Pronunciation Scoring and Assessment for Interactive Language Learning Speech Communication, 2000 Authors: S. M. Witt, S. J. Young Presenter: Davidson Date: 2009/07/08, 2009/07/15

2. Contents • Introduction • Goodness of Pronunciation (GoP) algorithm • Basic GoP algorithm • Phone dependent thresholds • Explicit error modeling • Collection of a non-native database • Performance measures • The labeling consistency of the human judges • Experimental results • Conclusions and future work

3. Introduction (1/3) • CAPT systems (Computer-Assisted Pronunciation Training) • Word and phrase level scoring (’93, ’94, ’97) • Intonation, stress, and rhythm • Requires several recordings of native utterances for each word • Difficult to add new teaching material • Selected phonemic error teaching (1997) • Uses duration information or models trained on non-native speech

4. Introduction (2/3) • HMM has been used to produce sentence-level scores (1990, 1996) • Eskenazi’s system (1996) produces phone-level scores but no attempt to relate this to human judgement • Author’s proposed system: • Measures pronunciation quality for non-native speech at the phone level

5. Introduction (3/3) • Other issues • GoP algorithms with refinements • Performance measures for both GoP scores and scores by human judges • Non-native database • Experiments on these performance measures

6. Goodness of Pronunciation (GoP) algorithm: Basic GoP algorithm • A score for each phone • = likelihood of the acoustic segment corresponding to each phone • GoP = duration normalized log of the posterior probability for a phone given the corresponding acoustic segment

7. Basic GoP algorithm (2/5) • = the set of all phone models • = number of frames in • By assuming equal phone priors and approximating by its maximum:

8. Basic GoP algorithm (3/5) • Numerator term is computed using forced alignment with known transcription • Denominator term is determined using an unconstrained phone loop

9. Basic GoP algorithm (4/5) • If a mispronunciation has occurred, it is not reasonable to constrain the acoustic segment used to compute the maximum likelihood phone to be identical to the assumed phone • Hence, the denominator score is computed by summing the log likelihood per frame over the duration of • In practice, this will often mean that more than one phone in the unconstrained phone sequence has contributed to the computation of

10. Basic GoP algorithm (5/5) • Intuitive to use speech data from native speakers to train the acoustic models • However, non-native speech is characterized by different formant structures compared to those from a native speaker for the same phone • Adapt Gaussian means by MLLR • Use only one single global transform of the HMM Gaussian component mean to avoid adapting to specific phone error patterns

11. Phone dependent thresholds • The acoustic fit of phone-based HMMs differs from phone to phone • E.g. fricatives tend to have lower log likelihood than vowels • 2 ways to determine phone-specific thresholds • By using mean and variance for phone • By approximating human labeling behavior

12. Explicit error modeling (1/3) • 2 types of pronunciation errors • Individual mispronunciations • Systematic mispronunciations • Consists of substitutions of native sounds for sounds of the target language, which do not exist in the native language • Knowledge of the learner’s native language is included in order to detect systematic mispronunciation

13. Explicit error modeling (2/3) • Solution: a recognition network incorporating both correct pronunciation and common pronunciation errors in the form of error sublattices for each phone. • E.g. “but”

14. Explicit error modeling (3/3) • Target phone posterior probability • Scores for systematic mispronunciations • GoP that includes additional penalty for systematic mispronunciation

15. Collection of a non-native database (1/2) • Based on the procedures used for the WSJCAM0 corpus • Texts are composed of a limited vocabulary of 1500 words • 6 females and 4 males whose mother-tongues are Korean (3), Japanese (3), Latin-American Spanish (3), and Italian (1). • Each speaker reads 120 sentences • 40 common set of phonetically-balanced sentences • 80 sentences varied from session to session

16. Collection of a non-native database (2/2) • 6 human judges who speaks native British English • Each speaker was labeled by 1 judge • 20 sentences from a female Spanish speakers are used as calibration sentences • Annotated by all 6 judges • Transcriptions reflect the actual sound uttered by the speakers • Including phonemes from other languages

17. Performance measures (1/3) • Compares 2 transcriptions of the same sentence • Transcriptions are either transcribed by human judges or generated automatically • 4 types of performance measures • Strictness • Agreement • Cross-correlation • Overall phone correlation

18. Performance measures (2/3) • Compared on a frame by frame basis • Each error is marked as 1 or 0 otherwise. • Yields a vector of length with • Apply a Hamming window • Transition between 0 and 1 is too abrupt where as in practice the boundary is often uncertain • Forced alignment might be erroneous due to poor acoustic modeling of non-native speech • Window length

19. Performance measures (3/3)

20. Strictness (S) • Measures how strict the judge was in marking pronunciation errors • Relative strictness

21. Overall Agreement (A) • Measures the agreement of all frames between 2 transcriptions • Defined in terms of cityblock distance between 2 transcription vectors

22. Cross-correlation (CC) • Measures the agreement between the error frames in either or both transcriptions • is the Euclidean distance

23. Phoneme Correlation (PC) • Measures the overall agreement of overall rejection statistics for each phone between 2 judges/systems • PC is defined as • is a vector of rejection count for each phone • denotes the mean rejection counts

24. Labeling consistency of the human judges (1/4)

25. Labeling consistency of the human judges (2/4) • All results are within an acceptable range • 0.85<A<0.95, mean = 0.91 • 0.40<CC<0.65, mean = 0.47 • 0.70<PC<0.85, mean = 0.78 • 0.03< <0.14, mean = 0.06 • These mean values can be used as a benchmark values

26. Labeling consistency of the human judges (3/4)

27. Labeling consistency of the human judges (4/4)

28. Experimental results (1/7) • Multiple mixture monophone models • Corpus: WSJCAM0 • Range of rejection threshold was restricted to lie within one standard deviation of the judges strictness • where

29. Experimental results (2/7)

30. Experimental results (3/7)

31. Experimental results (4/7)

32. Experimental results (5/7)

33. Experimental results (6/7) • Add error handling with Latin-American Spanish models to detect systematic mispronunciations

34. Experimental results (7/7) • Transcriptions comparison between human judges and the system with error network

35. Conclusions and future work • 2 GoP scoring mechanism • Basic GoP • GoP with systematic mispronunciation penalty • Refinement methods • MLLR adaptation • Independent thresholds trained from human judgement • Error network • Future work • Information about the type of mistake