Subjective Intelligibility Assessment

1. Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken

2. Signal-to-Noise ratio !!!

3. Research Questions Intelligibility versus Quality assessment Evaluation of a system or application Ranking of the performance of a number of systems Diagnostic assessment Prediction of system performance during design

4. Assessment Methods Subjective assessment with subjects (speakers and listeners): representative, limited reproduction, non diagnostic, laborious Objective assessment based on physical properties (measurements): reproducible, diagnostic, fast Prediction of system performance: design tool

5. SUBJECTIVE INTELLIGIBILITY

6. Subjective Intelligibility methods Phoneme level (nonsense words, rhyme words, consonants, vowels) Word level (meaningful words, nonsense words, phonetically balanced PB, equally balanced Eqb) Sentence level (Mean Opinion Score MOS, Speech Reception Threshold SRT)

7. Methodology I Response categories: Open response (e.g., nonsense words) Closed response (Rhyme tests, e.g., MRT, DRT) Scaling (MOS, five point scale: excellent - bad) Ranking (e.g., pair-wise comparison)

8. Methodology II Test design: Words embedded in carrier phrase Reference conditions (e.g. MNRU, …) Speakers (gender, number, non-native, …) Listeners ( number of speaker-listener pairs) Learning effects

9. Listening test with four subjects

11. Embedded CVC words: versta des over en nu fijs uit het woord zek einde noteer lal punt “Semi random”combination of: 17 initial consonants 15 vowels 11 final consonants

12. Methodology III Scoring, data analysis: Phone-word scores Confusion matrices Effective gain (e.g. effective SNR) Statistics (Anova, scaling, multiple regression, ...)

13. Relation Consonants-Vowels

14. How to calculate average word scores Subject responses may require to use the median

15. Example relation MOS-CVC

16. Relation between methods and Qualification

17. Test-retest variability Cronbach α based on split of speaker- listener pairs

18. Common Intelligibility scale (IEC60849) After Barnett and Knight 1994 CIS not linear with SNR  = STI  = 100 - ALcons x = AI  = PB words (256 words)  = Short Sentences  = PB words (1000 words)  = 1000 syllables Barnett and Knight (1995)

19. CVC scores (%) of realistic conditions male female Wide band 90.3 89.3 Telephone band 89.5 85.3 White noise SNR 0 dB 58.0 44.1 Speech noise SNR +3 dB 71.3 60.7 Speech noise SNR -3 dB 43.0 40.6

20. Example of consonant confusions p b f v m n R w p 1068 62 12 4 4 0 0 2 b 112 1002 0 0 11 7 0 50 f 44 1 915 193 0 0 0 0 v 6 4 337 739 0 0 2 43 m 1 5 0 0 1068 113 1 6 n 0 0 0 0 111 1081 0 2 R 1 2 0 2 0 2 1161 3 w 6 3 1 13 30 7 25 1065

21. Two dimensional display of confusions

22. Introduction of phoneme specific frequency weighting Four groups of phonemes (SAMPA notation: • Fricatives (f, s, v, z) • Plosives (b, d, x, p, t, k) • Vowel-like consonants (m, n, l, R, j, w, …) • Vowels (aa, a, ee, e, o, oo, u, uu, au, …)

23. Phoneme group specific spectra

24. Phoneme group specific spectra

25. Frequency weighting (fricatives)

26. Frequency weighting (plosives)

27. Frequency weighting (vowel-like cons)

28. Frequency weightings (vowels)

29. Frequency weightings (CVC words)

30. Prediction of (CVC) word score by aweighted combination of phoneme group probabilities (DUTCH) Ci = 0.294 fric + 0.294 plo + 0.412 Cvo V = V (no weighting) Cf = 0.273 fric + 0.273 plo + 0.454 Cvo CVC score = Ci * V * Cf * 100 %

31. CVC-word prediction (male) S.d.= 4.11% Male speech

32. CVC-word prediction (female) S.d. = 3.63% Female speech

33. ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)

34. Qualification table