QoS Measurement and Management for Multimedia Services

1. QoS Measurement and Management for Multimedia Services Thesis Proposal Wenyu Jiang April 29, 2002

2. Topics Covered • Objective QoS metrics for real-time multimedia • Subjective/Perceived quality • Objective perceptual quality estimation algorithms • Quality enhancement for real-time multimedia • IP telephony deployment • VoIP quality in the current Internet

3. Backgrounds and Motivations • The Internet is still best-effort • Needs QoS monitoring • What to measure/monitor? • Loss, delay, jitter • Must map to perceived quality • What to do if quality is not good? • End-to-End: FEC, LBR • Network provisioning: voice traffic aggregation • IP telephony service deployment • Current ITSPs are not doing well • Lack of study on localized deployment • What is the status of the current Internet?

4. How Real-time Multimedia Works • A/D conversion; Encoding; Packet transmission; Decoding; Playout; D/A conversion • Dominant QoS factors: • Loss  clipping/distortion in audio • Delay  lower interactivity • Jitter  late loss

5. Delay and Loss Measurement • Solutions for clock synchronization • Telephone-based synchronization • RTT-based, assume symmetric delays • GPS-based • Dealing with Clock drift • De-skewing by linear regression • One-way vs. round-trip measurement • Internet load often asymmetric • One-way loss and delay are more relevant to real-time multimedia

6. Loss and Delay Models • Loss Models • Gilbert model • Extended Gilbert model • Others • Delay Models • More difficult to construct • No universal distribution function • Temporal correlation between delays

7. Perceived Quality Estimation • Mean Opinion Score (MOS) • Requires human listeners • Labor and time intensive • Reflective of real quality • Objective perceptual quality estimation algorithms • PESQ, PSQM/PSQM+, MNB, EMBSD • Speech recognition based (new)

8. Network Provisioning for VoIP • Silence suppression • Saves bandwidthstatistical multiplexing • The on/off patterns in human voice depend on the voice codec or the silence detector • Voice traffic aggregation • Multiplexing by token bucket filtering • The on/off patterns in human voice directly affects aggregation performance • Past study assumes exponential distribution

9. IP Telephony Deployment • Localized deployment • More practical than a grand-scale Internet deployment • Can still interoperate with an IP telephony carrier • Issues • PSTN interoperability • Security • Scalability • Billing

10. Research Objectives • Objective QoS metrics • Modeling • Their relationship to perceived quality • Objective perceptual quality estimation algorithms vs. perceived quality (MOS) • Quality improvement measures • End-to-End: FEC vs. LBR • Network-based: voice traffic aggregation • IP telephony deployment issues • VoIP quality measurement over the Internet

11. Completed Work: QoS Measurement Tools • UDP packet trace generator • Clock synchronization and de-skewing tool • Loss and delay modeling tools • By examining a packet trace • Outputs Gilbert and extended Gilbert model parameters • Outputs conditional delay CCDF • Playout simulator • Simulates several common playout algorithms • FEC is also supported

12. Completed Work: Comparison of Loss Models • Loss burst distribution • Roughly, but not exactly exponential • Inter-loss distance • Clustering between adjacent loss bursts

13. Loss Model Comparison, contd. • Loss burstiness on FEC performance • FEC less efficient under bursty loss • Final loss pattern (after playout, FEC) • Generally also bursty

14. Mapping from Loss Model to Perceived Quality • Random vs. bursty loss • Bursty  lower MOS • Effect of loss burstiness • Sometimes very bursty loss does not lead to lower quality

15. A New Delay Model • Conditional CCDF (C3DF) • Allows estimation of burstiness in the late losses introduced by (fixed) playout algorithm

16. Objective vs. Subjective MOS • Algorithms: PESQ, PSQM, PSQM+, MNB, EMBSD Using Original Linear 16 samples as reference signal Using G.729 no loss clip as reference signal

17. Objective MOS Correlation, contd. • Second test set • Stronger “saturation” effect observed for MNB1 and MNB2, but not for PESQ Linear-16 reference signal G.729 reference signal

18. Auditory Distance vs. MOS • EMBSD and PSQM+ appear to have the largest spread, i.e., least correlation w. MOS • PSQM seems to be similar to MNB in terms of correlation

19. Auditory Distance vs. MOS, contd. • Second test set • Similar behaviors observed Linear-16 reference signal G.729 reference signal

20. Analysis of Objective MOS Correlation • Quantitative metric • Correlation coefficient  • But it does not tell everything!

21. Speech Recognition Performance as a MOS predictor • Evaluation of automatic speech recognition (ASR) based MOS prediction • IBM ViaVoice Linux version • Codec used: G.729 • Performance metric • absolute word recognition ratio • relative word recognition ratio

22. Recognition Ratio vs. MOS • Both MOS and Rabs decrease w.r.t loss • Then, eliminate middle variable p

23. Speaker Dependency Check • Absolute performance is speaker-dependent • But relative word recognition ratio is not

24. Speech Intelligibility Results • Human listeners are asked to do transcription • Human recognition result curves are less “smooth” than MOS curves.

25. Analysis of Voice On-Off Patterns • Past study finds spurt & gap distributions to be exponential • Modern voice codecs and silence detectors have different behaviors

26. Voice Traffic Aggregation • Simulation environment • DiffServ token bucket filter • Exponential, CDF and trace-based model simulations • N voice sources • Token buffer size B (packets) • R: ratio of reserved vs. peak bandwidth • Key performance figure • Probability of out-of-profile packet

27. Aggregation Simulation Results • Results based on G.729 VAD • CDF model resembles trace model in most cases • Exponential (traditional) model • Under-predicts out-of-profile packet probability; • The under-prediction ratio increases as token buffer size B increases

28. Simulation Results, contd. • Results based on NeVoT SD (default parameters: high threshold, long hangover) • Similar behavior, although the gap between exponential and CDF model is smaller for NeVoT case

29. Comparisons of FEC and LBR • Forward error correction • Bit-exact recovery • No decoder state drift upon recovery • Low bit-rate redundancy (LBR) • Just the opposite to FEC • Design of an optimal LBR algorithm • State repair via redundant codec • Optimal packet alignment • MOS quality verified to be better than the rat LBR • Allows a more “fair” comparison with FEC

30. MOS Quality of FEC vs. LBR • FEC shows a substantial and consistent advantage over LBR • This is true for all LBR configurations we tested • Main codec is G.729 except for AMR LBR DoD-CELP LBR DoD-LPC LBR

31. MOS of FEC vs. LBR, contd. • AMR LBR: narrowest gap with FEC • (Not shown here) FEC out-performs LBR under random loss as well G.723.1 LBR AMR LBR

32. Optimizing FEC Quality • Packet interval  loss burstiness FEC efficiency  • Result: FEC MOS performance also improves

33. Optimizing Conversational MOS for FEC • A larger packet interval  more delay • Trade-off between quality and delay • The E-model • Considers both delay and loss (and many other transmission quality factors) • Optimizing FEC MOS with the E-model

34. Optimizing FEC MOS, contd. • Validating E-model based prediction with real MOS test results

35. Localized IP Telephony Deployment: Architecture • Component based and distributed architecture • Allows easy integration of all SIP-compliant devices and programs

36. Deployment Issues • PSTN interoperability • T1 configuration and PBX integration • T1 line type (Channelized vs. ISDN PRI) • Line coding and framing (layer 2) • Trunk type: Direct-inward-dialing (DID) • Access permission on the PBX side • SIP/PSTN gateway configuration • Dial-peer: locates the proper SIP server or PSTN trunk • Dial-plan (translating calls from/to PSTN)

37. Deployment Issues, contd. • Security • Issue: gateway has no authentication feature • Solution: • Use gateway’s access control lists to block direct calls • SIP proxy server handles authentication using record-route • Allows easier change in authentication module (software-based) • Certain users can only make certain gateway calls • Scalability • SIP server (DNS SRV scaling) • Gateway; voice-mail server; conference server • Billing • Initial implementation via transaction logging

38. On-going Research • Measurement of the current Internet • How well can it support VoIP? • Or, how easy can VoIP applications adapt to (unfavorable) network conditions? • How fast does network condition change? • Can network redundancy help improve VoIP quality? • Physical redundancy (access links) • Virtual redundancy (overlay networking)

39. Conclusions • Completed research relating to many aspects of real-time multimedia, in particular VoIP • On-going work calls for: • A comprehensive measurement of the Internet • Analysis of the to-be measurement data • An answer to the question: how good is it today, and, how much better can we do?