10 Gigabit Networking at TelstraClear Jörg Micheel

1. 10 Gigabit Networking at TelstraClearJörg Micheel

2. Outline • Some Metro Ethernet equipment in use at TelstraClear • Test equipment, testing process • Project: University of Auckland 10 Gigabit Campus Ring • Measurement results and performance figures • Outlook: REANNZ Advanced Network implementation

3. Extreme Networks Black Diamond – metro core switch • Black Diamond 10808 (BD10K) • 22 rack mount units • 1280 Gbps capacity • Up to 48 10-Gigabit ports • Up to 480 10/100/1000 ports • Powerful VLAN, Virtual router Layer2 and Layer3 capabilities • Proprietary EAPS link-protection protocol provides continuity in case of fiber cut • L2/L3 Quality-of-Service • L2/L3 hardware filtering and priority • Jumbo frames at 9212 • High availability, hardware redundancy

4. Extreme Networks Summit X450 – edge switch • X450-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports • X450-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports • Optional dual 10-Gigabit Ethernet uplinks • 1 RU form factor • 160 Gigabits-per-second capacity • 65 million packets-per-second forwarding performance • Coming: X450a with stacking capability • Other features similar to Black Diamond series

5. Network test equipment – EXFO PacketBlazer • 8510 – wire speed full duplex 10/100/1000 • 8510G – wire speed full duplex 10G LAN/WAN PHY • BERT testing at Layer 1 • Frame testing at Layer 2 • RFC 2544 Performance test suite for network devices • Throughput, back-to-back packets, frame loss and latency, jitter • Detailed QoS analysis • Can extend concept of device testing to (parts of) a network

6. University of Auckland 10 Gigabit campus ring (1) Requirements: • Resilient infrastructure to interconnect eight University of Auckland campus sites • Provide multiple 1-Gigabit services at Layer 2 between campuses Solution: • Dual-implementation with Summit X450 at six sites within Auckland metropolitan domain • Dark fiber between the sites • Supports multiple unconstraint 1-Gigabit VLAN services with QoS up to the maximum 10 Gigabit backbone capacity • Resiliency via Ethernet Advanced Protection Switching (EAPS) • SLA: 99.99% end-to-end throughput • SLA: 500 milliseconds maximum round-trip-time • SLA: 250 milliseconds maximum packet delay variation

7. University of Auckland 10 Gigabit campus ring (2) Tamaki ITSS Epsom Symonds St Medicine Newton Data

8. University of Auckland – test setup Tamaki ITSS Epsom Symonds St Medicine Newton Data 12 VLANS 12 VLANS 12 VLANS 12 VLANS 12 VLANS 12 VLANS

9. University of Auckland – throughput results • 10 Gigabit backbone performance: full line rate for any packet sizes between 128 and 9212 bytes • Any single 1-Gigabit loop VLAN: loss free at any packet size • 10 loop 1-Gigabit VLAN performance (saturated core, with all services active):

10. University of Auckland – delay and jitter results • Fiber run length 44.7 km – 218.7 microseconds (RI=1.467) • OTD(min) = 258.4 microseconds • Switching, buffering accounts for 39.6 microseconds (15% of total) • Per switch latency 3.3 microseconds, inclusive serialization • RTTD(64) 517.0 microseconds at 100% of 1-Gigabit load • PDV(64) 1.2 microseconds  Meets SLA requirements comfortably, after adjustment for additional 10 km fiber spur

11. University of Auckland – latency as a factor of packet size

12. University of Auckland – EAPS protection switching • Measured the impact of a ring break upon site-to-site traffic • Worst case scenario – re-routing once around the entire ring • Measurements conducted at single site • Metric – number of packets lost • Worst case timing was measured at 217 milliseconds • Typical timing is around 100-150 milliseconds  Exceeds SLA expectations for sub-second failover

13. University of Auckland – test summary • Installation meets / exceeds original design parameters • Network is capable of delivering unconstraint full 1-Gigabit services across the metro domain • Fiber spur by far dominates latency (85%) • Switching latency is negligible, a few microseconds per packet at most • Serialization delay can become the dominating factor with large packet sizes and many hops  Consider the QoS impact when designing applications for either bulk data transfer or sensitive audio/video conferencing applications

14. REANNZ Advanced Network • 10 Gigabit backbone overlay on TelstraClear DWDM • Gigabit access in all major cities • Layer 2 network with QoS • Routers in Auckland, Wellington, Christchurch • International access: 155 Mbits/sec to Sydney, 622 Mbits/sec to California • Rollout in 2006 and 2007

15. Acknowledgements and references • University of Auckland ITSS department – Robert Beattie, Manager • REANNZ – Charles Jarvie, CEO • A cast of dozens of hands at TelstraClear and JazzTech Thank you!