Clarity in Voice Performance Measurements with Apdex

1. Clarity in Voice Performance Measurements with Apdex Session 45A CMG International Conference San Diego, California December 5, 2007 Ken Harker Consultant Service Level Management Keynote Systems, Inc. ken.harker@keynote.com (512) 335-7172

2. “Software On Demand” Unique OnDemand T&M Solutions Unique Capabilities Keynote Overview • Largest global on demand test & measurement • Founded 1995, public in 1999 (NASDAQ: KEYN) • 2,600 corporate customers • 170,000 member panel representative of end users • VoIP and Streaming - Accelerate services rollout and ensure service quality • Mobile - Improve mobile data services, applications and content delivery • Web - Improve online experience and site performance • 40+ distributed testing and monitoring services The “Keynote Network” • 240 locations • 160 metropolitan areas worldwide • 2,400 measurement computers deployed worldwide • More than 160 million online performance measurements taken daily

3. Some of What We Measure • Audio Responsiveness • Avoid conversational disruption • One-way delay in milliseconds • Audio Clarity • How good does it sound? • ITU-T P.862 PESQ standard • PESQ has a high correlation to human perception • MOS values are well understood in the industry

4. Why the Average is not Enough • Today there is no ‘standard’ way of aggregating measurements for reporting • Arithmetic means and medians • Geometric means and medians • Arithmetic and geometric standard deviation • Percentages over a threshold • Averaging loses outliers, which may be important • Example, 500 calls, 450 at MOS 4.2, 50 at MOS 3.0 • Average MOS for 500 calls = 4.08 • SLA Threshold of 4.0 is satisfied, but 10% of users unhappy • Standard deviation is difficult for many to understand and use

5. The End User is Everything • We need to translate service level performance data into a quantifiable measurement of end-user perception! • If the Average of MOS over the past month is X, and the standard deviation is Y, will the end user be happy? Unhappy? How do I know?

6. Apdex Solves Two Problems at Once • Apdex is better at capturing the distribution of the end user experience • Two data sets with identical average values can have different Apdex values • Apdex is better at examining performance data against industry standards of end user satisfaction • We can set the Satisfied and Tolerating thresholds to be industry-standard values • A unit-normalized index is accessible to those whose lives do not revolve around the technical infrastructure

7. Industry Standards • MOS: Telecommunications Industry Association Technical Services Bulletin 116 “Voice Quality Recommendations for IP Telephony” • Audio Delay:International Telecommunications Union standard ITU-T G.114 “One-way transmission time”

8. Business Problem • VoIP service provider • Nationwide USA customer base • Internal monitoring tools lack visibility into the open Internet and other networks • Needs to understand the service level performance of their VoIP product as experienced by the end user • Does not know where their product performance places relative to the competition

9. Keynote Monitoring Solution • Four agents in two cities • DSL and Cable network connections • Long distance and local calls • Measure five competitors from the same locations • Monthly benchmark reporting • Portal access to current measurement data and visualization tools

10. Comparisons Using Averages and Thresholds • Six voice services providers compared from the same measurement locations • Providers appear to fall into two performance groups in Average Audio Delay and two other performance groups in Average MOS

11. Apdex applied to Audio Delay • Providers C and E still have the best performance, as the previous table illustrated • Providers A and B are now clearly differentiated from providers D and F

12. Apdex applied to MOS • In our previous approach to reporting, Providers C and F were in the top performance tier • Provider F is meeting the Apdex target for end user satisfaction, but has many Frustrated measurements

13. Market Breakdowns • Both local calling routes appear to have similar Audio Delay performance, as do both long distance routes • The Average MOS on all routes are very similar to one another – just shy of a 4.0 average

14. Apdex applied to Audio Delay • Now, call performance from routes that originate from New York look very much alike • SFO-NY is a much worse end user experience than NY-SFO despite the fact that both routes have averages in the 150-400 ms range

15. Apdex applied to MOS • One of these call routes is not like the others • The SFO-SF call route had an average MOS just one-tenth of a point lower than the other routes, but a much lower Apdex

16. Summary Apdex uncovers performance problems Delay Ave Apdex MOS Ave Apdex C E A B F D C E B A D F F C B E A D F C B E A D Best Best Best Best Middle Middle Middle Middle Worst Worst

17. Customer Success • Addition of Apdex enriched the value of monthly reports • Responsiveness Apdex results helped customer identify problems with ATAs using incorrect gateway SBCs • MOS Apdex results helped customer identify problem with SBC settings in San Francisco • Competitive information was valuable in making strategic investment decisions

18. Thank You Ken Harker Consultant, Service Level Management ken.harker@keynote.com