Orange Labs Vincent BARRIAC , Recherche & Développement , TECH/OPERA 18th September 2009

1. A state of the art of voice quality measurement techniques and of associated standards at ITU-T SG 12 Orange Labs Vincent BARRIAC, Recherche & Développement, TECH/OPERA18th September 2009

2. section 1 Introduction on Voice quality and associated metrics Recherche & Développement - A state of the art of measurement of voice quality

3. QoS dimensions of VoIP : a network operator point of view • Pre-requesites to service: what is not visible by users but mandatory for the service to work: FCPE (registration, association client/line) DNS (translation logical name -> IP address) FTP (firlware update download) NTP (synchronisation) DHCP (IP address resolution) Registration on serviceplatform • Service VoIP : what is directly visible by users: Call set up Voice quality Access to voice messages Extra services (message indication, number presentation, call signalling, secret, muti-com, etc.) Call retainability Recherche & Développement - A state of the art of measurement of voice quality

4. Voice quality • Voice quality is a subjective notion, related with perception, and impacting the acceptability, and the success of services • It is an end-to-end concept, directly depending on technical characteristics of all involved elements : • Handsets (acoustics and signal processing) • Transmission equipments • Service platforms • Main parameters to consider : • Intrinsic quality of the signal (Mean Opinion Score, or MOS), depending on • Distortion (low bit-rate coding) • Packet loss • Signal level or attenuation • Transmission delay and its variation (jitter) • Echo • Noise Recherche & Développement - A state of the art of measurement of voice quality

5. Principal components of voice quality • Distortion of the signal • Impact on the intelligibility of speech • End-to-end transmission time (also sometimes called delay) • Impact on the interactivity of the conversation • Echo: a two-dimensional phenomenon • Characterized by its delay and its level (or its attenuation) • Impact on the interactivity of the conversation and on transmission times • Noise • Impact on the listening comfort • Vocal level • Impact on the listening comfort Recherche & Développement - A state of the art of measurement of voice quality

6. Overall Packet Loss Perceived Quality Jitter Buffers Overall Delay QoS Service Level Inter-relationship of QoS Factors (one-way quality, VoIP) Network Packet Loss Decoding + signal processing Performance Network Jitter Network Delay • Application Factors • Network Factors Recherche & Développement - A state of the art of measurement of voice quality

7. Evolution of speech quality and customers' satisfaction key factors • 80's and before: • user satisfaction correlated to speech quality key factor: speech quality • 90's: • apparition of competition (users => customers) • apparition of mobile services key factor: accessibility • 00's => today: • apparition of VOIP (new BM) key factor: price • Today, it's more complicated than ever • ungaranteed, time-varying speech quality • heterogeneity of contexts of usage, terminals, quality • relationship between customer satisfaction and quality?key factors: price & simplicity • Tomorrow? [source: ETSI, 2000] Recherche & Développement - A state of the art of measurement of voice quality

8. Things have considerably changed • However, there is a strong demand from network and marketing units for producing speech quality data in order to: • monitor speech quality • market speech quality (essentially wideband) • Today, this demand is mainly answered by the realization of • listening tests • Evaluation techniques based on the judgement of a panel of users in listening or conversational situations • Techniques follow very strict standardized protocols, generally consigned in ITU-T Recommendations, of which most used is P.800 • the use of "objective" (or instrumental) models • Based on tools (hardware or software) able to analyze the contents of the signal (of speech or any other type used for measurement) • Determination of the quality with a maximum correlation with the judgement of the users (depends of the accuracy of the tool) • Less costly in terms of time and resources Recherche & Développement - A state of the art of measurement of voice quality

9. section 2 Voice quality objective methods and models Recherche & Développement - A state of the art of measurement of voice quality

10. Two major ways to measure voice quality • 1) Non-intrusive measurements • Rather dedicated to supervision (network performance) on real communications • Ignores the contribution of terminal • Without additional cost, can supervise a big amount of communications • and thus obtain a good macroscopic view of transmission quality • On IP : protocol analysers dedicated for VoIP • IP parameters (packet loss, jitter, codec type ) • Overall transmission quality rating (ITU-T P.564 or E-model) • Requires a signal decoding and reconstruction for signal-based measurements • 2) Intrusive measurements • For end-to-end assessment close to the client • Good correlation with subjective perception • Access to the signal and thus possible use of PESQ • Performed on test communications • Extra traffic, with potential cost and load • Point to point (microscopic) vision, difficult to extrapolate without a wide spread of probes • Analogue interface (PSTN, GSM or behind a HGW) • MOS with reference (PESQ) and many other possible measurements (levels, echo, delay, etc.) • IP interface • IP parameters (packet loss, jitter, codec type ) • Overall transmission quality rating (ITU-T P.564 or E-model) Recherche & Développement - A state of the art of measurement of voice quality

11. PESQ (Perceptual Evaluation of Speech Quality) • Measurement on speech signal (bandwidth: 300-3400 Hz for narrow-band or 50-7000 Hz for wide-band) • Principle: Comparison of the voice signal to assess with a reference signal • One-way measurement • does not take delay into account • applies only for listening quality (no echo, no double talk). • Independent of speaker gender and language • Estimates delay variations • applies for packet transmission (including VoIP), • measures these variations • Standard Model (ITU-T P.862 Recommendation) • validated against a large variety of subjective databases (ρ=0.93 on 13 databases) • reliable and cheap alternative to subjective listening quality tests • P.862.1 and .2 give a mapping function from raw PESQ scores into MOS-LQON and MOS-LQOM respectively • Requires some precautions to get reliable MOS-LQO scores • original speech signal (level, voice activity rate, duration, etc.) • implementation of the model (+ IPR) Recherche & Développement - A state of the art of measurement of voice quality

12. The E-model • ITU-T G.107 Recommendation • Parametric approach (i.e. not based on signal analysis) • Officially, only a transmission planning tool • But often used as a passive monitoring tool (caution!) • Takes into account the subjective effects of all impairments in an additive way • Concept of "impairment factors" • Default values can be used when no measurement results are available • The E-Model output is a score (R) between 0 and 100 • Can be translated under certain conditions into MOS-LQEN or MOS-CQEN scores • A new version for wide-band audio has been standardised in 2006 • Rwb can now reach a maximal value of 129 • Takes into account : • Transmission delay and echo (speaker, listener and sidetone) • Levels (SNR, noise, loudness ratings) • Packet loss • Low bit rate coding (intrinsic quality and robustness against packet loss) Recherche & Développement - A state of the art of measurement of voice quality

13. Limitations of existing methods • Not yet fully adapted to wide-band (50-7000 Hz) and further audio transmission • Even PESQ-WB (P.862.2) has not been enough validated • Developments ongoing to develop new subjective methods and replace PESQ (P.OLQA, 2010) • The most known tools are for one-way measurements • Do not take delay into account • Apply only for listening quality (no echo, no double talk) • Developments ongoing to develop new methods (P.CQO) • Requires some precautions to get reliable and comparable MOS-LQO (N or M) scores • Original speech signal (level, voice activity rate, duration, gender of speakers etc.) for PESQ • Implementation of the models (+ IPR) • Problem of different quality scales • They don't address quality from a real end-through-end perspective • Need for new tools for correlation and diagnostic Recherche & Développement - A state of the art of measurement of voice quality

14. section 3 Main results and ongoing works in ITU-T SG12 Recherche & Développement - A state of the art of measurement of voice quality

15. ITU-T SG12 Responsible for Recommendations • on the end-to-end transmission performance of terminals and networks, in relation to the perceived quality and acceptance by users of text, data, speech (mostly), and multi-media applications. • Although this work includes the related transmission implications of all networks and all telecommunication terminals, a special focus is given to IP QoS, interoperability and implications for NGN, and also includes work on performance and resource management. Lead Study Group on QoS and Performance Note: free Access to ITU-T Recommendations http://www.itu.int/ITU-T/publications/recs.html Recherche & Développement - A state of the art of measurement of voice quality

16. Non-intrusive Speech Quality Modelling: P.564 An example of "framework" standard Conformance testing for voice over IP transmission quality assessment models (NB and WB)  minimum criteria for objective speech quality assessment models that predict the impact of observed IP network impairments on the one-way listening quality experienced by the end-user in IP/UDP/RTP-based telephony applications Main applications : monitoring of transmission quality for operations and maintenance purposes, measurements in support of service level agreements (SLAs) between service providers and their customers. Recherche & Développement - A state of the art of measurement of voice quality

17. P.564 – 3 modes of operation Mode A : Dynamic Operation In the network, but uses information from end-point (RTCP-XR packets) Mode B : Static Operation In the network, with a-priori knowledge of end-point behaviour Mode 3 : Embedded Operation Model co-located with the jitter buffer in the end‑point with access to the jitter buffer Recherche & Développement - A state of the art of measurement of voice quality

18. On-going work in ITU-T SG12 • Wider bandwidth • From speech (Narrow and wide band) to super-wideband and full-band. Consequences on coders, acoustic interfaces, test signal, analysis methodologies, quality models (including binaural) • Quality Modelling (speech and multimedia) The future "universal Model" for Speech (NB, WB, SWB), including acoustical interfaces, and diagnostic features: P. OLQA A Model to qualify noise cancellation performance: P.ONRA A Model for conversational quality: P.CQO • IPTV - Quality of experience requirements for IPTV (G.1080) - Performance monitoring for IPTV (P.NAMS and P.NBAMS) • IP QoS and network performance metrics Recherche & Développement - A state of the art of measurement of voice quality

19. P.OLQA • P.OLQA is supposed to be the replacement of PESQ • Extension to a bigger bandwidth (up to 14 kHz instead of 7) • Extension to acoustical interfaces • Further conditions taken into account (incl. time warping) • Addition of a degradation decomposition (4 dimensions : frequency content, continuity, noisiness, loudness) to enhance the core model and give diagnostic information • A competition has been launched to select the best model • FT is candidate in partnership with DT • Other competitors : TNO, Psytechnics, Ericsson, Opticom, SwissQual • The results of validation of models are awaited for next February… • The real application of P.OLQA in real measurement tools is not before mid-2010 Recherche & Développement - A state of the art of measurement of voice quality

20. Thank you