Optical Networks for the Rest of Us “Customer Empowered Networking”

1. Optical Networks for the Rest of Us“Customer Empowered Networking” NANOG 17 – Montreal 1999 http://www.canarie.ca Background Papers on Gigabit toThe Home and Optical InternetArchitecture Design AvailableOptical Internet News list:Send e-mail to Bill@Canarie.ca Bill.St.Arnaud@canarie.ca http://Tweetie.canarie.ca/~bstarn Tel: +1.613.785.0426

2. CANARIE Inc • Mission: To facilitate the development of Canada’s communications infrastructure and stimulate next generation products, applications and services • Canadian equivalent to Internet 2 and NGI • private-sector led, not-for-profit consortium • consortium formed 1993 • federal funding of $220m (1993-99) • total project costs estimated over $500 M • currently over 140 members; 21 Board members • Phase III funding to be announced 1998-2001 • $55 million announced for Optical Internet -March 1998

3. CA*net 3 National Optical Internet Consortium Partners: Bell Nexxia Nortel Cisco JDS Uniphase Newbridge CA*net 3 Primary Route CA*net 3 Diverse Route GigaPOP ORAN Deploying a 4 channel CWDM Gigabit Ethernet network – 700 km Deploying a 4 channel Gigabit Ethernet transparent optical DWDM– 1500 km Condo Dark Fiber Networks connecting universities and schools Multiple Customer Owned Dark Fiber Networks connecting universities and schools Netera MRnet SRnet ACORN BCnet St. John’s Calgary Regina Winnipeg Charlottetown RISQ ONet Fredericton Montreal Vancouver 16 channel DWDM -8 wavelengths @OC-192 reserved for CANARIE -8 wavelengths for carrier and other customers Halifax Ottawa Seattle STAR TAP Toronto Los Angeles Chicago New York

4. What is an Optical Internet? • WDM fibers where individual wavelengths are the link layer interconnect directly connected to routers via Optical ADM (Add Drop Mux) or WDM coupler • High Performance Router acts as the main switching routing device • Bypass or cut-thru connections via dedicated wavelengths • SONET or Gigabit Ethernet framing (also 10xGbE or SDL) • Use intrinsic self healing nature of Internet for redundancy and protection (don’t require SONET/SDH layer) • Traffic engineering and network management done via MPLS • Network design optimized for unique characteristics of Internet traffic – fractal traffic, asymmetric traffic and congestion at the edge

5. Lessons Learned • Carrier transport people now must learn to deal with customers directly • Require network management tools that give customer a view of “their” wavelengths • A whole new set of operating procedures required • OAM&P issues between router vendors and DWDM remain a challenge • SONET management systems expect to see a contiguous network • CA*net 3 required DCC work arounds • Need network tools to measure end to end performance and throughput at OC-48 or greater speeds – • HP is about to release a couple of beta products • MPLS is proving a lot more difficult in practice to implement • Need tools for management of tunnels • Need Inter-domain MPLS-TE • Mythology of 50msec “fast restoral” still not understood • OSPF with very short hold down timers and GRE tunnels or policy routing may be an adequate alternative

6. 10xGbE & CWDM • Several companies have announced long haul GbE and CWDM with transceivers at 50km spacing • 10GbE coming shortly • Costs are as little as $12K US per node (or transceiver) • Future versions will allow rate adaptive clocking for use with “gopher bait” fiber, auto discovery, CPE self manage • Excellent jitter specification • Most network management and signaling done at IP layer • Anybody with LAN experience can build a long haul WAN – all you need is dark fiber • With CWDM, no EDFA power disbursement • Repeater distance independent number of wavelengths

7. Market drivers for GbE in the WAN • Many ISPs, regional networks, municipalities, school districts are purchasing dark fiber or building dark fiber networks up to 1000 km rather than managed bandwidth • With dark fiber increased bandwidth only entails upgraded equipment costs and no additional monthly charges • Significant savings in relocating servers to central site and using VoIP • Also many carriers willing to sell “gopher bait” fiber (fiber that does not meet stringent SONET/DWDM requirements) at a discount • As such, cost of transmission equipment is becoming a significant factor versus cost of fiber • SONET and ATM networks require specialized engineering knowledge and skills • Customers want a technology in the WAN they are familiar with and that is easily extensible from the LAN e.g. Ethernet • Don’t require the same reliability as telco systems

8. Major Long Haul Componentsfor IP/DWDM network SONET Transport Terminal WDM Coupler $20K 50 km Transponder Wideband Optical Repeater $250K SONET Regen $250k per Tx/Rx 250 km Terabit Router $400K For transponder currently using regen box $125K Approximate Distances for OC-192 system Typical Cost $6000 per km (not counting cost of fiber router, and transponder) for one OC-192 channel Advantage – can support multi-services and well known technology

9. G G G G Major Long Haul Componentsfor 10xGbE CWDM network 50 km CWDM Coupler $5K 10xGbE Switch $20K 10x Transceiver $20K Approximate Distances for 10xGbE system Typical Cost $400 per km (not counting cost of fiber Or 10xGbE switches) for 10 Gbps Advantage – very low cost 1/10 cost of SONET & DWDM Disadvantage – requires 2 fibers and can only carry IP traffic

10. Optical Networks for the Rest of Us • With customer owned dark fiber, 10GbE and 4 channel CWDM anybody can build a 40Gbps network up to 1000km or greater at a fraction of cost of traditional telco network • May not be suitable for mission critical traffic (at least not yet) • But ideal for high bandwidth Internet to the school, small business and home • Ring structures are a customer option – not a mandatory requirement • The driver is NOT new applications, but cost savings –1 year payback • Typical cost is one time $20K US per school for a 20 year IRU • In Ottawa we are deploying a 60km- 96 strand network connecting 22 institutions – cost $500K US

11. Where are we going? • Today the Internet is “virtual” network riding on top of a traditional “connection oriented” network of cooper and fiber • With optical technology such as customer owned dark fiber, customer owned wavelengths, 10GbE etc we can extend the model of the Internet as tool to empower the user to build networks in the physical domain as well as the virtual domain • The future telecommunication’s world may be dominated by thousands of customer owned networks that peer at the physical as well as at the virtual level, “Optical Networks for the Rest of Us” • A national or provincial K-12 network with its own wavelengths and dark fiber • A national banking network with its own wavelengths and dark fiber • A national auto network with its own wavelengths and dark fiber • A radical departure from the “carrier centric” view of the universe

12. 2 Different Views Telco Network “Ring of Rings” 99.999% reliability only in the SONET Ring for the telco, no guarantees for the customer SONET ring CO ISP B Customer Empowered Network “Warp and Weave” Customer responsible for building rings 99.999% reliability to the customer but no guarantees for the ISP ISP A 3 separate dark fiber builds ISP C

13. Customer Empowered Networks Dark fiber Network City C Dark fiber or CWDM Network City A ISP B ISP D ISP C ISP A Dim Wavelength Customer achieves 99.999 reliability by multi-home to different ISPs Optical Label Switched Router Long Haul DWDM ISP C ISP A First Dark fiber Network City B ISP B Second Dark Fiber Network

14. New Challenges and Opportunities • “Customer empowered networks” present a whole new set of research challenges: • Peering and topology protocols in the optical domain – what will be the equivalent to BGP and OSPF in the optical domain • Multi Protocol Lambda Switching? • Defining LSP attributes such as power level, wavelength, encoding, etc? • Interdomain optical MPLS? • Customer controlled establishment of wavelengths, routing and service delivery • Auto discovery of wavelengths? • Management and interface systems, etc, etc