Quality and Performance Evaluation of VoIP End-points

1. Quality and Performance Evaluation of VoIP End-points Wenyu Jiang Henning Schulzrinne Columbia University NYMAN 2002 September 3, 2002

2. Motivations • The quality of VoIP depends on both the network and the end-points • Extensive QoS literature on network performance, e.g., IntServ, DiffServ • Focus is on limiting network loss & delay • Little is known about the behavior of VoIP end-points

3. Performance Metrics for VoIP End-points • Mouth-to-ear (M2E) delay • C.f. network delay • Clock skew • Whether it causes any voice glitches • Amount of clock drift • Silence suppression behavior • Whether the voice is clipped (depends much on hangover time) • Robustness to non-speech input, e.g., music • Robustness to packet loss • Voice quality under packet loss

4. Measurement Approach • Capture both original and output audio • Use “adelay” program to measure M2E delay • Assume a LAN environment by default • Serve as a baseline of reference, or lower bound

5. VoIP End-points Tested • Hardware End-points • Cisco, 3Com and Pingtel IP phones • Mediatrix 1-line SIP/PSTN Gateway • Software clients • Microsoft Messenger, NetMeeting (Win2K, WinXP) • Net2Phone (NT, Win2K, Win98) • Sipc/RAT (Solaris, Ultra-10) • Robust Audio Tool (RAT) from UCL as media client • Operating parameters: • In most cases, codec is G.711 -law, packet interval is 20ms

6. Example M2E Delay Plot • 3Com to Cisco, shown with gaps > 1sec • Delay adjustments correlate with gaps, despite 3Com phone has no silence suppression

7. Visual Illustration of M2E Delay Drop, Snapshot #1 • 3Com to Cisco 1-1 case • Left/upper channel is original audio • Highlighted section shows M2E delay (59ms)

8. Snapshot #2 • M2E delay drops to 49ms, at time of 4:16

9. Snapshot #3 • Presence of a gap during the delay change

10. Effect of RTP M-bits on Delay Adjustments • Cisco phone sends M-bits, whereas Pingtel phone does not • Presence of M-bits results in more adjustments

11. Sender Characteristics • Certain senders may introduce delay spikes, despite operating on a LAN

12. Average M2E Delays for IP phones and sipc • Averaging the M2E delay allows more compact presentation of end-point behaviors • Receiver (especially RAT) plays an important role in M2E delay

13. Average M2E Delays for PC Software Clients • Messenger 2000 wins the day • Its delay as receiver (68ms) is even lower than Messenger XP, on the same hardware • It also results in slightly lower delay as sender • NetMeeting is a lot worse (> 400ms) • Messenger’s delay performance is similar to or better than a GSM mobile phone.

14. Delay Behaviors for PC Clients, contd. • Net2Phone’s delay is also high • ~200-500ms • V1.5 reduces PC->PSTN delay • PC-to-PC calls have fairly high delays

15. Effect of Clock Skew: Cisco to 3Com, Experiment 1-1 • Symptom of playout buffer underflow • Waveforms are dropped • Occurred at point of delay adjustment • Bugs in software?

16. Clock Skew Rates • Mostly symmetric between two devices • RAT (Ultra-10) has unusually high drift rates, > 300 ppm (parts per million) • High clock skews confirmed in many (but not all) PCs and workstations

17. Drift Rates for PC Clients • Drift Rates not always symmetric! • But appears to be consistent between Messenger 2K/XP and Net2Phone on the same PC • Existence of 2 clocking circuits in sound card?

18. Packet Loss Concealment • Common PLC methods • Silence substitution (worst) • Packet repetition, with optional fading • Extrapolation (one-sided) • Interpolation (two-sided), best quality • Use deterministic bursty loss pattern • 3/100 means 3 consecutive losses out of every 100 packets • Easier to locate packet losses • Tested 1/100, 3/100, 1/20, 5/100, etc.

19. PLC Behaviors • Loss tolerance (at 20ms interval) • By measuring loss-induced gaps in output audio • 3Com and Pingtel phones: 2 packet losses • Cisco phone: 3 packet losses • Level of audio distortion by packet loss • Inaudible at 1/100 for all 3 phones • Inaudible at 3/100 and 1/20 for Cisco phone, yet audible to very audible for the other two. • Cisco phone is the most robust • Probably uses interpolation

20. Effect of PLC on Delay • No affirmative effect on M2E delay • E.g., sipc to Pingtel

21. Silence Suppression • Why? • Saves bandwidth • May reduce processing power (e.g., in conferencing mixer) • Facilitates per-talkspurt delay adjustment • Key parameters • Silence detection threshold • Hangover time, to delay silence suppression and avoid end clipping of speech • Usually 200ms is long enough [Brady ’68]

22. Hangover Time • Measured by feeding ON-OFF waveforms and monitor RTP packets • Cisco phone’s is the longest (2.3-2.36 sec), then Messenger (1.06-1.08 sec), then NetMeeting (0.56-0.58 sec) • A long hangover time is not necessarily bad, as it reduces voice clipping • Indeed, no unnatural gaps are found • Does waste a bit more bandwidth

23. Robustness of Silence Detectors to Music • On-hold music is often used in customer support centers • Need to ensure music is played without any interruption due to silence suppression • Tested with a 2.5-min long soundtrack • Messenger starts to generate many unwanted gaps at input level of -24dB • Cisco phone is more robust, but still fails from input level of -41.4dB

24. Acoustic Echo Cancellation • Important for hands-free/conferencing (business) applications • Primary metric: Echo Return Loss (ERL) • Measured by LAN-sniffing RTP packets • Most IP phones support AEC • ERL depends slightly on input level and speaker-phone volume • Usually > 40 dB (good AEC performance)

25. M2E Delay under Jitter • Delay properties under the LAN environment serves as a baseline of reference • When operating over the Internet: • Fixed portion of delay adds to M2E delay as a constant • Variable portion (jitter) has a more complex effect • Initial test • Used typical cable modem delay traces • Tested RAT to Cisco • No audible distortion due to late loss • Added delay is normal

26. M2E Delay under Jitter, contd. • Cisco phone generally within expectation • Can follow network delay change timely • Takes longer (10-20sec) to adapt to decreasing delay • Does not overshoot playout delay • More end-points to be examined Artificial Trace Real Trace with Spikes

27. Conclusions • Average M2E Adelay: • Low (mostly < 80ms) for hardware IP phones • Software clients: lowest for Messenger 2000 (68.5ms) • Application (receiver) most vital in determining delay • Poor implementation easily undoes good network QoS • Clock skew high on SW clients (RAT, Net2Phone) • Packet loss concealment quality • Acceptable in all 3 IP phones tested, w. Cisco more robust • Silence detector behavior • Long hangover time, works well for speech input • But may falsely predict music as silence • Acoustic Echo Cancellation: good on most IP phones • Playout delay behavior: good based on initial tests

28. Future Work • Further tests with more end-points on how jitter influences M2E delay • Measure the sensitivity (threshold) of various silence detectors • Investigate the non-symmetric clock drift phenomena • Additional experiments as more brands of VoIP end-points become available