1 / 49

Speech Assessment: Methods and Applications for Spoken Language Learning

Speech Assessment: Methods and Applications for Spoken Language Learning. J.-S. Roger Jang ( 張智星 ) jang@cs.nthu.edu.tw http://www.cs.nthu.edu.tw/~jang Multimedia Information Retrieval Lab CS Dept, Tsing Hua Univ, Taiwan. Outline. Introduction to speech assessment Methods

flores
Download Presentation

Speech Assessment: Methods and Applications for Spoken Language Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Speech Assessment: Methods andApplications for Spoken Language Learning J.-S. Roger Jang (張智星) jang@cs.nthu.edu.tw http://www.cs.nthu.edu.tw/~jang Multimedia Information Retrieval Lab CS Dept, Tsing Hua Univ, Taiwan

  2. Outline • Introduction to speech assessment • Methods • Using learning to rank for speech assessment • Demos • Conclusions

  3. Intro. to Speech Assessment • Goal • Evaluate a person’s utterance based on some acoustic features, for language learning • Also known as • Pronunciation scoring • CAPT (computer-assisted pronunciation training)

  4. Computer-Assisted Language Learning (CALL) • 4 aspects of CALL • ListeningEasier • SpeakingHarder • ReadingEasier • WritingHarder • Receptive skills are easier to be assisted by computers, while productive skills are harder to evaluate automatically. • SA plays an essential role in CALL for speech/pronunciation scoring.

  5. Speech Assessment • Characteristics of ideal SA • Assessment levels: as detailed as possible • Syllables, words, sentences, paragraphs • Assessment criteria: as many as possible • timbre, tone, energy, rhythm, co-articulation, … • Feedbacks: as specific as possible • High-level correction and suggestions

  6. Basic Assessment Criteria • Timber (咬字/音色) • Based on acoustic models • Tone (音調/音高) • Based on tone recognition (for tonal language) • Based on pitch similarity with the target utterance • Rhythm (韻律/音長) • Based on duration comparison with the target utterance • Energy (強度/音量) • Based on energy comparison with the target utterance

  7. Additional Assessment Criteria • English • Stress (重音) • Levels (word or sentence) • Intonation (整句音調) • Declarative sentence • Interrogative sentence • Co-articulation(連音) • A red apple. • Did you call me? • Won’t you go? • Raise your hand. • Mandarin • Tone (聲調) • Retroflex (捲舌音) • Co-articulation (連音) • 兒化音 • Others • Pause

  8. Types of SA • Types of SA (ordered by difficulty) • Type 1:有目標文字、有目標語句 • Type 2:有目標文字、無目標語句 • Type 3:無目標文字、有目標語句 • Type 4:無目標文字、無目標語句 • We are focusing on type 1 and 2.

  9. Our Approach • Basic approach to timbre assessment • Lexicon net construction (Usually a sausage net) • Forced alignment to identify phone boundaries • Phone scoring based on several criteria, such as ranking, histograms, posterior prob., etc. • Weighted average to get syllable/sentence scores

  10. Lexicon Net Construction • Lexicon net for “what are you allergic to?” • Sausage net with all possible (and correct) multiple pronunciations • Optional sil between words

  11. Lexicon Net with Confusing Phones • Common errors for Japanese learners of Chinese • ㄖㄌ • 例:天氣熱天氣樂 • ㄑㄐ • 例:打哈欠 打哈見 • ㄘㄗ • 例:一次旅行一字旅行 • ㄢㄤ • 例:晚安晚ㄤ • Rule-based approach to creating confusing syllables • Rules: • Rule 1: re  le • Rule 2: qi  ji • Rule 3: ci  zi • Rule 4: an  ang • Example • 欠 (qian)見 (jian)、嗆 (qiang)、降 (jiang)

  12. Lexicon Net with Confusing Phones • Lexicon net for “天氣熱、打哈欠” • Canonical form: tian qi re da ha qian • 16 variant paths in the net:

  13. Automatic Confusing Syllable Id. Corpus of Japanese learners Of Chinese 強制對位以得到初步切音結果 對華語411音節進行比對 以找出每個音的混淆音 將混淆音節加入辨識網路 再進行強制對位及切音 切音結果不再變動? 輸出混淆音節 及辨識網路 No Yes

  14. Error Pattern Identification (EPI) • Common insertions/deletions from users • 以「朝辭白帝彩雲間」為標準語句 • 任意處結束,例如「朝辭白帝」 • 任意處開始,例如「彩雲間」 • 任意處開始與結束,例如「白帝彩雲」 • 任意處開始與結束,並出現跳字,例如「白彩雲」 • 疊字,例如「朝…朝辭白帝彩雲間」 • 疊詞例如「朝辭…朝辭白帝彩雲間」 • 疊字加換音,例如「朝(cao)…朝(zhao)辭白帝彩雲間」 • 兩字對調,例如「朝辭彩帝白雲間」 • 錯字,例如「朝辭白帝黑山間」

  15. Lexicon Net for EPI (I) • 偵測「從頭開始、在任意處結束」的發音

  16. Lexicon Net for EPI (II) • 偵測「從任意處開始,在尾端結束」的發音

  17. Lexicon Net for EPI (III) • 偵測「從任意處開始,結束於任意處(但不可跳字)」的發音

  18. Lexicon Net for EPI (IV) • 偵測「從任意處開始,結束於任意處,而且可以跳字)」的發音

  19. Design Philosophy of Lexicon Nets • We need to strike a balance between recognition and lexicon • In the extreme, we can have a net for free syllable decoding to catch all error patterns. • The feasibility of free syllable decoding is offset by its not-so-high recognition rate.

  20. Scoring Methods for Speech Assessment • Five phone-based scoring methods • Duration-distribution scores • Log-likelihood scores • Log-posterior scores • Log-likelihood-distribution scores • Rank ratio scores • All based on forced alignment to segment phones

  21. Method 1: Duration-distribution Scores • PDF of phone duration • Obtained from forced alignment • Normalized by speech rate • Fitted by log-normal PDF • Max PDF  score 100

  22. Method 2: Log-likelihood Scores • Log-likelihood of phone with duration of frames : where is the likelihood of the frame with the observation vector

  23. Method 3: Log-posterior Scores • Log-posterior of phone with duration : where

  24. Method 4: Log-likelihood-distribution Scores • Use CDF of Gaussian for log-likelihood • CDF = 1  score = 100

  25. Method 5: Rank Ratio Scores • Rank ratio • RR to score conversion where parameters a, b are phone specific. • Possible sets of competing phones for x+y • *+y • *+*

  26. Intro. to Learning to Rank • Learning to rank • A supervised learning algorithmwhich generates a ranking model based on a training set of partially order items. • Methods • Pointwise (e.g., Pranking) • Pairwise (e.g., RankSVM, RankBoost, RankNet) • Listwise (e.g., ListNet)

  27. Application of LTR to SA • Why use LTR for SA? • Human scoring is rank-based: A+, A, B, B-… • Tsing Hua’s grading system is moving from scores (0~100) to ranks (A, B, C, D…). • Combination of features (scores) • Features are complementary. • Effective determination of ranking • LTR only generates numerical output with a ranking order as close as possible to the correct order. A optimum DP-approach is proposed.

  28. LTR Score Segmentation Given: LTR scores (sorted) Desired rank We want to find the separating scores with score-to-rank function Rank 1 Rank 2 Rank 3 Rank 4 Rank 5 Such that is minimized.

  29. LTR Score Segmentation by DP (I) • Formulate the problem in DP framework • Optimum-value function D(i,j): The minimum cost of mapping to rank • Recurrent equation • Boundary condition: • Optimum cost:

  30. LTR Score Segmentation by DP (II) Computed rank 5 4 3 2 Desired rank 1 Local constraint: Recurrent formula:

  31. LTR Score Segmentation with DP (III) Data distribution: DP path:

  32. Flow Charts of Our Experiment

  33. Corpora for Experiments • WSJ • For training biphone acoustic models for forced alignment • MIR-SD • Recordings of about 4000 multi-syllable English words by 22 students (12 females and 10 males.) with an intermediate competence level. • Originally designed for stress detection • Available at http://mirlab.org/dataSet/public

  34. Human Scoring of MIR-SD • Human scoring • Only 50 utterances from each speaker of MIR-SD are scored by 2 humans, making a total of 1100 utterances • Human scoring are consistent:

  35. Performance Indices • Performance indices used in the literature • hr = [1 3 5 4 2 2], cr = [2 3 5 2 1 4] • Recognition rate rRate = 33.33% • Recognition rate with tolerance 1 = 66.67% • Average absolute difference = 1 • Correlation coef = 0.54

  36. Performance Evaluation of Different Scoring Methods

  37. Overall Performance Comparison • Legends • Score segmentation • Circles: DP • Triangles: k-means • Inside/outside tests • Solid lines: Inside • Dashed lines: Outside • Black lines: Baselines

  38. Demo: Practice of Mandarin Idioms of Length 4 (一語中的) • Level (difficulty) of an idiom is based on it’s freq. via Google search: • 孤掌難鳴 ===> 260,000 • 鶼鰈情深 ===> 43,300 • 亡鈇意鄰 ===> 22,700 • 舉案齊眉 ===> 235,000 • Can be adapted for English learning • Next step: multi-threading, fast decoding via FSM

  39. Support Mandarin & English Support user-defined recitation script Next step: multithreading for recording & recognition Demo: Recitation Machine(唸唸不忘)

  40. For Mandarin, English, Japanese Licensing for PC Applications

  41. SA for Embedded Systems • Embedded platforms: PMP, iPhone, Androids

  42. Demo: Tangible Companions • Chicken run (落跑雞) • Penguin for Tang Poetry (唐詩企鵝) • Robot Fighter (蘿蔔戰士) • Singing Bass & Dog (大嘴鱸魚和唱歌狗)

  43. Tools and Tutorials • Tools • DCPR toolbox • http://mirlab.org/jang/matlab/toolbox/dcpr • SAP toolbox • http://mirlab.org/jang/matlab/toolbox/sap • ASR Toolbox • http://mirlab.org/jang/matlab/toolbox/asr • Tutorials • Data clustering and pattern recognition: • http://mirlab.org/jang/books/dcpr • Audio signal processing • http://mirlab.org/jang/books/audioSignalProcessing • Lab page (with demos): • http://mirlab.org

  44. Other SA Issues to be Addressed • Core technology • Other acoustic features for scoring • Pitch: tone/intonation • Volume • Duration • Pause • Coarticulation • Error pattern identification • Application side • Mulimodal GUI • Extensions • Slightly adaptation • Paragraph-level SA • Text-free SA • Beyond pronunciation • Translation + recognition + assessment • Microphone types

  45. Examples • Coarticulation • Knock it off! • Mom woke her up • Consonant+consonant • Bus stop • Push Shirley • Ask question • Jeff flew south through Tainan • Exception • Change jobs • Which Chair

  46. Examples • Changes due to coarticulation • Would you like it? • Won’t you go? • Raise your hand. • It makes you look younger. • Softened sounds • Junction • Popcorn • Fruitful • Can and can’t • I can read the letter. • I can’t read the letter. • d and t • Better • Cider

  47. Most Likely to be Mispronounced • Within Taiwan • Pleasure/pressure • World/war/word • Shirt/short • Walk/work • Flesh/fresh • Supply/surprise • Some/son • Confirm/conform • Cancel/cancer • Mouth/mouse • Measure/major • Police/please • Version/virgin

  48. Conclusions • Conclusions • SA calls for more cues than ASR • SA requires techniques from ML/IR • Multi-modal approach to SA is a must • “Popcorn”, “Thursday” • On-going & future work • Tone recognition & assessment • Reliable error pattern identification

  49. References • Witt, S. M. and Young, S. J., “Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning”, Speech Communication 30, 95-108, 2000. • Kim, Y., Franco, H., and Neumeyer, L., “Automatic Pronunciation Scoring of Specific phone Segments for Language Instruction”, in Proceedings of the 4th European Conference on Speech Communication and Technology (Eurospeech ’97), pp. 649-652, Rhodes, 1997. • Neumeyer, L, Franco, H., Digalakis, V., and Weintraub, M., “Automatic Scoring of Pronunciation Quality”, Speech Communication 30, 83-93, 2000. • Franco, H., Neumeyer, L., Digalakis, V., and Ronen, O., “Combination of Machine Scores for Automatic Grading of Pronunciation Quality”, Speech Communication 30, 121-130, 2000. • Cincared, T., Gruhn, R., Hacker, C., Nöth, E., and Nakamura, S., “Automatic Pronunciation Scoring of Words and Sentences Independent from the Non-Native’s First Language”, Computer Speech and Language 23, 65-88, 2009. • Crammer, K. and Singer, Y., “Pranking with Ranking”, in proceedings of the conference on Neural Information Processing Systems (NIPS), 2001. • Joachims, T., “Optimizing Search Engines using Clickthrough Data”, in proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), ACM, 2002. • Freund, Y., Iyer, R., Schapire, R. E., and Singer, Y., “An Efficient Boosting Algorithm for Combining Preferences”, in proceedings of ICML, pp170-178, 1998. • Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hullender, G., “Learning to Rank using Gradient Descent”, in proceedings of ICML, pp. 89-96, 2005. • Cao, Z., Qin, T., Liu, T. Y., Tsai, M. F., and Li, H., “Learning to Rank: From Pairwise Approach to Listwise Approach”, in proceedings of the 24th International Conference on Machine Learning, pp. 129-136, Corvallis, OR, 2007. • Liang-Yu Chen , Jyh-Shing Roger Jang, “Automatic Pronunciation Scoring using Learning to Rank and DP-based Score Segmentation”, submitted to Interspeech 2010.

More Related