Automated SIP Interoperability Testing

1. Automated SIP Interoperability Testing Archana Rao and Henning Schulzrinne Department of Computer Science Columbia University http://www.cs.columbia.edu/IRT/voip-testbed/ SIP 2009 (Paris, January 2009)

2. Overview • The problem of SIP interoperability • Far from perfect • Damages “brand”, harms customers, encourages single-vendor deployments • Testbed Overview • Architecture • Components • Experiments • Interoperability study • End-client characteristics • Summary

3. Interoperability Is it a real problem ? Hi Henning,You may remember me from IETFs past -- I haven't attended any in some time because I couldn't find any really interesting projects. I'm finallygetting into SIP. I've got Speakeasy VoIP service, two sipphone accounts, a Cisco 7960 and a copy of x-ten on my Mac. And I still can't make it work. Voice flows in one direction only. I'm not even behind a NAT or firewall -- both machines have global addresses, with no port translations or firewalls. I've been working with Internet protocols for over 20 years. I've implemented and contributed to them. And if *I* can't figure out how to make this stuff work, how is the average grandmother expected to do so? SIP is unbelievably complex, with extraordinarily confusing terms. There must be half a dozen different "names" -- Display Name, User Name, Authorization User Name, etc -- and a dozen "proxies". Even the word "domain" is overloaded a half dozen different ways. This is ridiculous!Sorry. I just had to get this off my chest. Regards, Phil Karn. Excerpts from an email posted on IETF RAI mailing list – Reference: http://www1.ietf.org/mail-archive/web/rai/current/msg00082.html

4. Interoperability issues Why do we have interoperability problems? • SIP is complex • More than 150 RFCs and 500 active Internet Drafts • Efforts like – “draft-ieft-sip-hitchhikers-guide” help to a certain extent, but not sufficient • SIP has no proposed architecture/profile • SIP usage in variety of environments and domains make it impossible • Standardization from IETF is different from that of ISO/ITU • SIP is continually evolving • Non trivial for implementers to follow the life-cycle from IDs to RFCs • Walled gardens, non-standard environments • Vendors need to make products that work in their customer’s environment • Poor implementations, intentional non-interoperability etc.

5. Improving interoperability Current efforts to combat SIP interoperability issues • SIP Interoperability (SIPit) events • Week long gatherings of SIP implementers to test interoperability • Coordinated by SIP Forum • SIP Forum • SIP Connect 1.0 (and 1.1)  outsourced enterprise VoIP • IETF Basic Level of Interoperability for SIP Services (BLISS) • Focus on resolving interpretability issues in SIP features • Line sharing, call parking, automated call handling and call queuing • University of New Hampshire – InterOperability Laboratory • Vendor-neutral lab dedicated to testing data networking technologies • 20 different testing programs, each costing about $20,000 per year

6. Interoperability So, isn’t the problem solved? • These forums focus on details of advanced SIP features • Basic-level of interoperability between innumerable SIP devices in the market • Can I take any SIP phone and make a VoIP call, through any SIP service provider? How do we go about it? 1. Identify the basic real-world scenarios for SIP registration and session establishment 2. Use our VoIP testbed to configure a variety of SIP infrastructure 3. Study the behavior of SIP end-clients and servers for interoperability

7. Studying interoperability • Define a minimum set of call flows constituting basic interoperability • RFC 3665, RFC 3666, RFC 4317 • Use a VoIP testbed to realize a variety of real-world SIP infrastructure setups • Columbia VoIP testbed • Study the behavior of SIP end-clients and servers in each of these scenarios • Interoperability matrices • Categorization of common issues  • Lack of clarity in the specification • Implementation of an older specification • Incomplete implementation of the specification • Incorrect implementation of the specification • Failure against robustness tests

8. Columbia VoIP testbed What? • VoIP infrastructure for experimentation, analysis, testing, prototyping and deployment of SIP/VoIP components in a variety of environments. • Part of a multi-university research project supported by NSF • University of North Texas, Purdue University and University of California, Davis Why? • There’re no telecom testbeds available for research & experimentation • VoIP implementations & experiments have lacked scientific rigor • Continuously emerging standards • Need to develop repeatable, accurate test frameworks • Realistic VoIP experiments require a distributed testbed

9. http://www.cs.columbia.edu/IRT/voip-testbed/ http://www.cs.columbia.edu/IRT/voip-testbed/ your equipment here :-)

10. Columbia VoIP testbed: servers • 5 SIP servers • Asterisk, Pbxnsip, Sipd, SER, OpenSER • 3 Platforms • Microsoft Windows XP • Linux Fedora Core 6 • Sun Solaris X • 3-way Connectivity • the Internet • VPN • PSTN

11. Columbia VoIP testbed: UAs • 20+ SIP hard-phones, soft-phones, WI-FI phones with different capabilities • More details – www.columbia.edu/~ahr2114/VoIPtestbed/end-clients

12. PSTN Network UNIVERSITY-A UNIVERSITY-B UNIVERSITY-D The Internet UNIVERSITY-C NSF VoIP testbed Image courtesy: http://secnet.csci.unt.edu/7.15.funding_application.ppt

13. Columbia VoIP testbed experiments Planned or ongoing • Interoperability study • Can any two SIP devices talk to each other? • SIP end-client characteristics • Signaling and codec features in SIP devices • Robustness • VoIP with/despite NAT • Performance issues • Signaling delay, voice mouth-ear delay, packet loss resilience • Servers – Scalability, maximum capacity • Security • Separate testbed for DOS and other attacks work in progress

14. Interoperability study: call flows Registration (RFC 3665) • Successful new registration • Unsuccessful registration • Cancellation of registration • Update of contact list Session Establishment (RFC 3665) • Successful session establishment • Session establishment through two proxies • Session establishment with multiple proxy authentication • Successful session with proxy failure • Unsuccessful no answer • Unsuccessful busy • Unsuccessful no response from user agent • Unsuccessful temporarily unavailable Codec Negotiation (RFC 4317) • Audio and DTMF • Audio, video and DTMF • Audio and video codec reordering

15. Interoperability study: matrix Interoperability Matrix Test case #1 - Successful new registration

16. Interoperability study: auth name Issue 1 – Use of different formats for authentication name • Authorization: Digest username=“user1”,realm=“proxy01.sipphone.com”,nonce=“xxx”, uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5 Authorization: Digest username=“sip:user1@sipphone.com”, realm=“proxy01.sipphone. com”,nonce=“xxx”,uri=“sip:proxy01.sipphone.com”,response=“yyy”, algorithm=MD5 • Implication • Registration and call setup failure, if both formats are not supported by the UAs • Instances • Polycom PVX/VSX, Wengophone don’t register with 3Com / sipd proxies • Workaround • Use Asterisk between Polycom PVX and 3Com proxy

17. SIP server UAC INVITE (Dest port:5060) 180 Ringing (Src port:12345) 200 OK (Src port:12345) ACK (Dest port:12345) 200 OK (Src port:12345) ACK (Dest port:12345) Interoperability study: rport Issue 2 – Behavior in the absence of rport parameter • Implication • Incorrect/incomplete signaling resulting in multiple retransmissions and failed transaction • Instances • Calls to Cisco 7940 via Sipphone proxy are always unsuccessful • Calls from Polycom VSX via 3com or sipd proxies are always unsuccessful • Workaround • Nothing

18. Interoperability Study: codecs Issue 3 – Codec Negotiation • Implication – Audio/Video failure (mostly unidirectional, but bidirectional sometimes) • Problem Instances • Audio failure when Cisco calls voicemail service of sipphone provider • Video failure when Polycom PVX calls Xlite via Asterisk • [Offer] v=0 o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67 s= SIP Call t=0 0 m=audio 31030 RTP/AVP 0 8 18 101 c=IN IP4 128.59.17.67 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:18 G729/0 a=fmtp:18 annexb=no a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=sendrecv • [Answer] v=0 o=Cisco-SIPUA 28422 0 IN IP4 128.59.17.67 s= SIP Call t=0 0 m=audio 31030 RTP/AVP 18 0 8 101 c=IN IP4 128.59.17.67 a=rtpmap:18 G729/0 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=sendrecv

19. UA characteristics: features • Basic • Configuration, number of lines, call hold, call forwarding, DND, NAT support • Advanced • Encryption, symmetric RTP, conferencing, audio/video codecs, PoE • Audio quality – silence suppression, echo cancellation, comfort noise • Transport, signaling and support protocols – TCP, TLS, HTTP, DHCP, DNS, TFTP, NTP • More details at – http://www.columbia.edu/~ahr2114/VoIPTestbed/EndClients.htm

20. UA characteristics: robustness Robustness • SIP Torture Tests (RFC4475) • A set of SIP messages aimed at stressing the SIP parser, based on experiences at SIPit. • PROTOS • Test suite with 4500+ INVITE requests, developed by University of Oulu Example Results

21. UA characteristics: NATs SIP and NAT • Problematic for both signaling (SIP) and media transfer (RTP) • Circumventing NAT effect – solutions exist for both end-clients and proxies • Quest to overcome this, could lead to techniques uglier than NAT itself! NAT test setup • NAT realization - Linux iptables, CLICK router • Studying the behavior in different scenarios • NAT support enabled in both the proxy server and end-client • NAT support enabled in only one of them etc • Aiming to analyze the existing practices – the good, the bad and the ugly!

22. UA with NATs SIP devices in NAT environment

23. What now? Summary • Tests indicate that basic-level interoperability is far from perfect • Instances of interoperability failures illustrated in our IETF workshop paper • Achieving Interoperability shouldn’t be left just to vendors • non-interoperability damages SIP brand, causes grief for third parties and harms customers Going ahead – some ideas • Establish designated interop liaison for each vendor • calling customer support unlikely to be useful • Encourage vendors to publish interoperability reports • in standard format • Provide self-certification tests, supported by a remotely accessible test rig

24. Summary Current status • The VoIP testbed & VPN connectivity is operational • Systematic study of real-world interoperability issues • Basic-level interoperability is nowhere near 100% • IETF 70 – The 1st SIP Forum Workshop on SIP Interoperability • Evaluating SIP end client characteristics Testbed plans • Experiments on NAT, performance, security • Make the testbed accessible and useful for engineers & deployments • Create a knowledge repository about SIP devices & tools for interoperability testing