1 / 64

TMC Developers Conference San Francisco Aug 03 rd , 2005

ST-09 Network Assurance and Testing During the Migration to VoIP. TMC Developers Conference San Francisco Aug 03 rd , 2005. Andy Huckridge Spirent Communications. Chair, Interop WG, MSF. Agenda. Spirent overview Key implementation issues What is Triple Play / Converged networks?

pembroke
Download Presentation

TMC Developers Conference San Francisco Aug 03 rd , 2005

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ST-09 Network Assurance and Testing During the Migration to VoIP TMC Developers ConferenceSan FranciscoAug 03rd, 2005 Andy Huckridge Spirent Communications. Chair, Interop WG, MSF

  2. Agenda • Spirent overview • Key implementation issues • What is Triple Play / Converged networks? • Specifics on testing SIP • Network Impairments and Parameters that Voice and Video Affect Quality • Metrics for Measuring Voice and Video Quality and Performance • Good test methodology

  3. Spirent Communications • Spirent is the test solution leader • 1,800 employees in 14 countries • More than 1,500 customers • Sales and service capabilities in 30 countries

  4. Performance Analysis Industry standards Manufacturing Quality Assurance Functionality & Conformance Testing Performance Testing Implementation steps - Lab Services Deployment • Characterize system before trial • Validate system scalability • Identify capacity limits • Measure call performance • Automate regression testing

  5. Service Assurance Initial Deployment Network Certification Widespread Network Deployment Pilot Networks Implementation steps - Network • Facilitate vendor selection • Identify performance ceilings • Enable accurate capacity planning • End-to-end service assurance testing • Improve operational performance • Improve customer satisfaction

  6. Key Implementation Issues • Circuit to packet migration • Scalability and Performance • Voice quality • Interoperability and conformance • Budget pressures

  7. Before you deploy! • Network Equipment Manufacturers (Chips, IP-PBX, Gateways, MSs & SSs) • Characterize your system before trial • Validate system scalability • Identify capacity limits • Measure call performance • Service Providers(NSPs, SPs, ITSPs) • Define criteria for vendor selection • Identify performance ceilings • Accurately plan for your capacity needs • End-to-end service assurance testing

  8. Data Transmission“Non-Real-Time” Applications Telenet Name Resolving DNS Web HTTP Email and Messaging POP SMPT Exchange Data Base MS SQL Oracle File Transfer FTP Music Downloading Home control Data Examples: Internet access, Email, File Transfer, Portals, Database Applications, Gaming, Government Services, Online Commerce

  9. Voice and Video “Real-Time Applications” Gaming Single / Multiplayer IP Music/Audio/Radio VoD VoIP, IP Telephony, Video Telephony G.711, G.729, G.728, G.726, G.723 H.261, H.263, SIP, SIP-T, H323, Skinny, MGCP, MEGACO/H.248 Real-Time Online Communications Instant Messenger Webex Netmeeting SIP H.323 IPTV Services Broadcast, On-Demand, Bi-directional / Interactive MPEG1, MPEG2, MPEG4, VC1, H264 Multi-Media RTP H.264,Microsoft AVI, QuickTime (.mov) Windows Media (.wmv, .asf), RealMedia (.rm), • Voice Applications: Phone service integrated with video • Video Applications: Broadcast TV, video on demand, distance learning

  10. Converged Triple Play: Data, Voice and Video With Network Impairments Data Video Dialed Digits On Hook Sigtran Configure Configure Configure Ring Back Notify Off Hook Dial # Good Bye TCP Hello SS7 CAS SIP Voice Conversation Good Bye On Hook Connect Off Hook Ring Disconnect Hello ISDN Connect MEGACO RTP/UDP Impairments can be heard in the voice conversation Signaling Path

  11. Comprehensive and scriptable SIP call flows Complete configurable SIP signaling messages SIP protocol analysis Simplified flow diagrams with visual analysis Comprehensive conformance test suites Testing SIP Conformance

  12. Testing SIP Conformance • ETSI TS 102-027-1 v2.12,Tiphon: • RFC 3261 user agent, proxy and redirect server compliance • Graphical SDL and TTCN tools • Create, edit, compile and execute simulation scripts and conformance tests • Additional SIP messages beyond RFC 3261 • Included in torture tests • Additional tests as defined by the SIP Forum

  13. Testing with Configurable SIP Configurable SIP call setup and call teardown • Configurable call flows and messages • Incoming message filter • Adaptive signaling syntax for SIP • Improves interoperability with new drafts and non-conformant proprietary implementations

  14. Testing with Configurable SIP • Configurable messages: • Invite, ACK, bye, register • Responses: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx • Configurable timers,message intervals • Enable and disable optional messages: • Re-invite, cancel, options, message, info, notify, subscribe, unsubscribe, update, refer, Prack • Fix erroneous incoming messages “on the fly” with the “search and replace” method • Allows interoperability with SIP devices (including drafts, non conformant, prototype)

  15. SIP Message Registration Screen Shots

  16. SIP Message Origination

  17. SIP Message Termination

  18. Incoming Message Filters

  19. TOS for SIP Signaling

  20. Diffserv for SIP Signaling

  21. Testing SIP Robustness • Robustness testing • Passed: does not crash, stable, or acceptable results • Failed: crashes, unstable, or unacceptable results • Security testing • It is crucial to identify SIP security holes • SIP testing tool • Tests SIP robustness and security • Comprehensive negative test suites for SIP

  22. Real Signaling with real RTP • Capability to do signaling with audio • Capability to perform real time measurements • Capability of using signaling without audio • Problems of not using real signaling • Problems of not using real RTP streams • Real time objective metrics

  23. PSTN PSTN Testing SIP-T VoIP Network MGC MGC SIP-T SS7 SS7 SIP-T SIP-T H.248/Megaco H.248/Megaco SIP Proxy RTP/RTCP Trunking Gateway Trunking Gateway SS7 GR303 ISDN CAS V5 1000Base-SX/LX 10/100/1000Base-T POTS POTS

  24. SIP-T Performance Testing Suites Performance testing • Validate and stress-test SS7 ISUP and SIP interworking with optional media, over thousands of emulated user agents • SIT-T testing • Configurable SIP-T calls with intelligent protocols • QoS and CoS testing • Optional TOS/Diffserv and VLAN options in SIP-T media calls, used to measure QoS with PESQ and e-model • Feature testing • Automated and configurable SIP call set-up, teardown, flows, messages

  25. Network Architecture Types of Access Links QoS controlled Edge Routing MTU Size Packet Loss (Frame Loss) Out of order packets One Way Delay (Latency) Variable Delays (Jitter) Background Traffic (Congestion, Bandwidth, Utilization, Network Load, Load Sharing) Timing Drift Route Flapping Signaling protocol mismatches Network faults Link Failures Voice Only Impairments Echo Voice coding algorithms A/D and D/A Conversion Noise – Circuit and External Video Only Impairments Video coding algorithms Fixed vs Variable Frame Rate Network Impairments and Parameters that Affect Voice and Video Quality

  26. IP Network Architecture Affects Data, Voice and Video Quality

  27. Network Operating With Constant Delay Affects Voice and Video Quality

  28. End to End Delay Sources Core Network Originating LAN Terminating LAN Originating Gateway Edge Router Edge Router Terminating Gateway Core Network Routers • Fixed • Serialization • WAN • Fixed • Serialization • WAN • Fixed • Switching • Propagation • Serialization • Variable • Voice contention • Data Contention • Video Contention • Fixed • Look ahead • Encoding • Buffer • VAD • Packetizing • Fixed • Switching • Variable • Voice contention • Data Contention • Video Contention • Fixed • Switching • Variable • Voice contention • Data Contention • Video Contention • Fixed • Decoding • Variable • De-jitter buffer • Packet loss Concealment • Algorithmic delay • Serialization delay • Propagation delay • Component delay Affects Voice and Video Quality

  29. Echo Impairment on Converged network Delay in IP Network makes Echo sound worse Tail Circuit TELR IP Phone MG T1 Link 2 Wire IP Network PBX POTS Phone ERLE ERL IP Phone Echo Canceller in MG reduces Echo Level ERLE – Echo Return Loss Enhancement ERL – Echo Return Loss TELR – Talker Echo Loudness Rating Analog 4-Wire Link Analog 2-Wire Link RX RX E&M T1 Link Affects Voice Quality TX TX Hybrid Transformer POTS Phone Impedance Mismatch PBX

  30. Echo Impairment on Converged network • Electrical Coupling • Impedance Mismatch (Hybrid) • Acoustical Coupling • Speakerphone Affects Voice Quality Converged Network Path A to B Path B to A Echo Path Side A (250ms) Echo Path Side B (250ms) Echo is caused by impedance mismatches in hybrid circuits (2w to 4w) and feedback between the telephone mouth piece and ear piece

  31. Effect of Delay on Voice Quality Voice Quality > 25ms Echo Cancellation Required PSTN <150 ms (with echo cancellation): acceptable 150-400 ms: acceptable if delay expected > 400 ms unacceptable for most applications

  32. Effect of Echo Level on Voice Quality Less Echo Less Echo More Echo More Echo Affects Voice Quality TELR – Talker Echo Loudness Rating (Signal to Echo Ratio)

  33. Variable processing delay A busy router or switch will take longer to look up the routing (address) table Queuing delay Network congestion Network with Variable Delays (Jitter) Affects Voice and Video Quality Delay (ms) Time (s)

  34. Time (s) Jitter Characteristics Delay (ms) Good Delay (ms) Bad Severe Delay (ms) Affects Voice and Video Quality

  35. RED (Random Early Discard) Packet LossExample: Queue Management Threshold Affects Voice and Video Quality Bit Bucket

  36. Speech Compression Impairment Voice Quality G.711 Best Quality • Common Compression Types: G.711, G.729, G.728, G.726, G.723, AMR, EVRC

  37. VAD – Voice Activity Detection Timing may be different No VAD Affects Voice Quality VAD Data is intentionally not sent during times of Silence

  38. Impact Of Packet Size Affects Data, Voice and Video Quality 10 Bytes • Typically Packets are kept small for best results • Many equipment manufacturers use dynamic packet size to optimize for network conditions = 10 ms Speech 20 Bytes Normal size for VoIP applications = 20 ms Speech 40 Bytes = 40 ms Speech 80 Bytes = 80 ms Speech

  39. Mechanisms for Assuring QOS Data • Class of Service (COS) ITU-T Y.1541 defines the 5 classes of service and their application • Type of Services (TOS) • TOS and COS are both elements with in an IP Packet • DIFSER and RSVP provide mechanisms to improve QOS Voice Video Triple Play Affects Data, Voice and Video Quality

  40. TIA-921 and ITU-T G.NIMMTest Profiles Based on QoS (Y.1541) Classes Different test profiles for different Service Level Agreements (SLAs) Profile B Best Effort Managed Network Table 3 Profile A Well Managed Network Table 2 Profile C Un-Managed Network Table 4

  41. Early Voice Quality Testing

  42. Voice Quality Testing • Active (Intrusive) Testing • Sends, Receives and compares Wave Files to measure voice quality • MOS (Mean Opinion Score) • PSQM, PSQM+ (Perceptual Speech Quality Measurement) • PESQ (Perceptual Evaluation of Speech Quality) • R-Value and J-MOS derived from PESQ • Passive Testing • R-Value – ITU-T P.VTQ • Measures Voice Quality on RTP Packets • Based on E-model • Japan – J-MOS • Similar Techniques can be used to measure Video Quality • P.563 (ITU-T recommendation) 3SQM, P-Stream • Measures Voice Quality of Voice traffic based on Audio Siginal • Provides an estimate of PSQM

  43. Active (Intrusive) Voice Quality Testing • MOS, PSQM, PSQM+, PESQ, R-Factor (PESQ Derived) DUT Send Wave Files Example: (ITU-T Female Nice File with Pilot Tone) Receive Wave Files • Measures Voice Quality by Comparing Sent and Received Wave files Sent (Green) and Received (Orange) wave files Expanded Sent (Green) and Received (Orange) wave files PESQ Score vs Number of PESQ Measurements Values are different for Male, Female, different Wave Files and different Languages

  44. Passive Voice and Video Quality Testing R-Factor/Emodel Measure: Video Quality MOS-LQ MOS-CQ MOS-PQ J-MOS Network R User R Burst statistics Diagnostic data PSTN IP Network RTP E1/T1/E3/T3/PRI/GR303, V5,SLC96 Trunking Gateway RTP Measure: Video Quality MOS-LQ MOS-CQ MOS-PQ J-MOS Network R User R Burst statistics Diagnostic data ITU-T P.VTQ IP Telephone

  45. Passive Voice Quality TestingP.563 (P-Stream, 3SQM) RTP DUT Receive Audio • Estimates Voice Quality based on 3 Characteristic of Received Audio

  46. Voice Quality Measurements Emodel P.861 PSQM/PSQM+ P.862 PESQ PAMS 5 0 4.5 ~3.88 ~3.65 ~3.40 ~3.13 ~2.84 6.5 1 -0.5

  47. Sample Voice Quality Test Results MOS PSQM PESQ Created by inducing packets lost G.711 G.723.1 (6300 bps) Comparison of Scores for G.711 and G.723.1 (6300 bps)

  48. Video Quality Measurement • Video Compression • Video Compress schemes affect the video quality • H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4, Microsoft AVI, Windows Media (.wmv, .asf), RealMedia (.rm), QuickTime (.mov) • Interactive real-time applications (e.g., video conferencing, voice over IP) are sensitive to latency and Frame Rate • Typical Video Quality Metrics • Objective MOS • Blockiness • Blur • PSNR • Spatial Resolution • Temporal Resolution • SNR • Edge Noise • Jerkiness • Error Blocks • Object Retention • Color Reproduction Accuracy

  49. Video Quality MeasurementBlockiness Blockiness Original

  50. Video Quality MeasurementReference and Blocky Video Blocky Original (Reference)

More Related