CANARIE canarie CA*net 4 Design Document Last Revised April 22 2001 Version 1.20

1. CANARIE http://www.canarie.caCA*net 4 Design DocumentLast Revised April 22 2001Version 1.20 OBGP documentation and latest version of this document can be found at http://www.canet3.net Bill.St.Arnaud@canarie.ca Tel: +1.613.785.0426

2. The Concept for CA*net 4 • Conventional optical networks are built on the paradigm that a central entity has control and management of the wavelengths • It therefore must have control of the edge device for the setup and tear down of the wavelengths • Will central control and management scale to millions of edge device and thousands of optical wavelengths? • Customer empowered optical networks are built on the paradigm that customer owns and controls the wavelengths (Virtual Dark Fiber) • Customer controls the setup, tear down and routing of the wavelength between itself and other customers • Customer may trade and swap wavelengths with other like minded customers ultimately leading to wavelengths as market commodity • How do you design a network architecture if the routing and control of wavelengths is under the control of the customer at the edge? • Network is now an asset, rather than a service • Analogy to time sharing computing in the early 1970s versus customer owned computers or client-server computing

3. Condo Fiber & Wavelengths • Condo fiber means that separate organizations own individual strands of fiber in a fiber cable • Each strand owner responsible for lighting up the strand • Collectively responsible for sharing costs of maintenance on fiber cable, relocation etc • Condo wavelengths • Number of parties share in the cost of a single strand and that light up with an agreed upon number of wavelengths • Wavelengths are portioned based on percentage ownership • With condo fiber and condo wavelengths institutions can treat network as an asset just like purchasing a computer, rather than a service as today

4. Research Network Issues • Research and Education networks must be at forefront of new network architecture and technologies • Should be undertaking network technology development that is well ahead of any commercial interest • But any network architecture can only be validated by connecting real users with real applications and must solve real world problems • Test networks per se are not sufficient • There is a growing trend for many schools, universities and businesses to control and manage their own dark fiber • Can we extend this concept so that they can also own and manage their own wavelengths? • Will “empowering” customers to control and manage their own networks result in new applications and services similar to how the PC empowered users to develop new computing applications?

5. CA*net 4 Research Objective • To deploy a network architecture where the GigaPOPs and institutions at the edge manage and control their own fiber and their own wavelengths • Condominium fiber and condominium wavelengths • To deploy a novel new optical network of distributed optical IXs that gives GigaPOPs and communities at the edge of the network (and ultimately their participating institutions) the ability to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage • To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of eScience and grid applications to support true peer-to-peer networking • To allow connected regional and community networks to setup peering relationships with CA*net 4 for collaborative research and education and eScience applications • To partner with private sector in building “carrier neutral” distributed optical Internet exchange facilities across Canada and developing new services in fungible wavelengths to enable customer empowered optical networks

6. Current View of Optical Internets ISP AS 1 AS 4 Carrier controls and manages edge devices Optical VLAN NNI AS 1 Customer AS 5 AS 3 UNI Big Carrier Optical Cloud using MPS or ASON for management of wavelengths for provisioning, restoral and protection AS 2

7. Customer Empowered Metro Network City C City A OBGP switch Carrier Neutral IX & OBGP switch Carrier Neutral IX & OBGP switch Condo Wavelengths City B OBGP switch Condo Dark Fiber Condo Wavelengths

8. Future Optical Networks Massive peering at the edge Customer D Customer A Condo Wavelength Customr A elects to cross connect with Customer C rather than D Customer C Customer B Condo Fiber

9. CA*net 4 Research Areas • New optical technologies that support customer empower networking • OBGP, CWDM, hybrid optics and HWDM, customer controlled optical switches • BGP scaling issues • Object Oriented Networking • Wavelengths and optical switch treated as an object and method to be incorporate into middleware • Or treated as fungible product • Distributed Computing Applications and Grids • Wavelength Disk Drives (WDD) • eScience • Grids for weather forecasting, forestry management, education, health, etc

10. Object Oriented Networking • Combines concepts of Active Networks and Grids • See DARPA • See Globus • Customer owns sets of wavelengths and cross connects on an optical switch • Network elements can be treated as a set of objects in software applications or grids • Complete with inheritances and classes, etc • Rather than distributed network objects ( e.g. Java or Corba) distributed object networks • In future researchers will purchase networks just like super computers, telescopes or other big science equipment • Networks will be an asset – not a service • Will be able to trade swap and sell wavelengths and optical cross connects on commodity markets

11. Advantages of OON • With massive peerings to the edge, the loss of one peer is not catastrophic • No need for restoral or protection paths or ring architectures • Networks look more like “star bursts” rather than “ring of rings” • See C Labovitz ACM Sigcomm Aug 2000 – massive peering helps faster convergence • May solve problem of scaling large networks • Today M carriers building meshed networks to N customers with resultant M*N2 requirement for wavelengths • With OON the global requirement for wavelengths grows at X*N where X= average number of wavelengths per customer

12. Example OON • Earthquake Visualization Grid • Globus Middleware Begin • Establish connection to other grid participants • Network Object – wavelength to STAR LIGHT – Chicago • Network Object – wavelength to Research center Amsterdam • Network Object – wavelength to SDSC Visualization Computer • Network Object – wavelength to Seismology Center Calgary • Link objects and create grid • Run Visualization • Release Network objects • Globus Middleware End • Earthquake Visulization End

13. University in Canada willing to exchange these wavelengths Montreal-NY blue NY-Amsterdam red Chicago-Montreal green Wants these wavelengths Chicago-SDSC - purple SDSC- Hawaii - red Chicago- Chile - yellow University in Chicago willing to swap these wavelengths Chicago to SDSC – purple SDSC- Hawaii – Red Chicago to Argonne - Blue Wants wavelength to these locations Chicago to Toronto - yellow Napster OON

14. The eScience Vision • “National grand challenge" e-research projects are on the horizon: with the next generation network, interconnecting to school and community networks, Canadian researchers could use the thousands of computers in schools and communities distributed across Canada • Students at schools and ultimately members of the public could be full participants in basic reserach • The next generation research network should be designed to encourage and enable projects such as these

15. Wavelength Disk Drives St. John’s Regina CA*net 3/4 Calgary Winnipeg Charlottetown Montreal Halifax Fredericton Vancouver WDD Node Ottawa Toronto Computer data continuously circulates around the WDD

16. eScience Grid WDD Grid Customer D Customer A Customers autonomously create WDD ring for high performance applications Customer C Customer B

17. Wavelength Disk Drives • CA*net 4 will be “nation wide” virtual disk drive for grid applications • Big challenges with grids or distributed computers is performance of sending data over the Internet • TCP performance problems • Congestion • Rather than networks being used for “communications” they will be a temporary storage device • Ideal for “processor stealing” applications

18. OBGP • Proposed new protocol to support control and management of wavelengths and optical switch ports • Control of optical routing and switches across an optical cloud is by the customer – not the carrier – true peer to peer optical networking • Use establishment of BGP neighbors or peers at network configuration stage for process to establish light path cross connects • Customers control of portions of OXC which becomes part of their AS • Optical cross connects look like BGP speaking peers – serves as a proxy for link connection, loopback address, etc • Traditional BGP gives no indication of route congestion or QoS, but with DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth • Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths • May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks

19. Opportunity for carrier and industry partners • To participate in a novel new Internet architecture that will allow customers to manage and control their own wavelengths anywhere across the network • Very attractive technology for Tier 2 ISP, research networks and ASPs • Yahoo and Cable and Wireless have already started down this path • It will allow them to create their own network topologies • To provide a valuable new service for customers that will allow them to reduce Internet transit costs by as much as 75% • To develop new value added services in IX brokering and management • To develop new fungible trading services in bandwidth trading and brokering • To experiment with new long haul optical technologies that will dramatically reduce cost of long haul transmission

20. CA*net 4 Possible Architecture Layer 3 aggregation service Optional Service Available to any GigaPOP St. John’s Regina Calgary Winnipeg Large channel WDM system Charlottetown Europe Vancouver Montreal Customer controlled optical switches Fredericton Halifax Seattle Ottawa Chicago New York Toronto

21. Wavelength Scenarios Workstation to Workstation Wavelength University to University Wavelength Campus OBGP switch St. John’s CWDM GigaPOP to GigaPOP Wavelength Regina Winnipeg RISQ Halifax Calgary BCnet Vancouver Montreal Seattle Toronto

22. Wavelength Setup AS 2- AS 5 Peer AS 3 12 10 University Regional Network 3 13 AS 1 2 15 4 AS 5 14 AS 1- AS 6 Peer AS 2 5 7 9 1 AS 4 AS 6 Regional Network 6 8 University Dark Fiber ISP router Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party

23. Wavelength Logical Mapping AS 2- AS 5 Peer AS 3 12 10 University Regional Network 3 13 AS 1 2 15 4 AS 5 14 AS 1- AS 6 Peer AS 2 5 7 9 1 AS 4 AS 6 Regional Network 6 8 University Primary Route ISP router Backup Route

24. University Regional Network 9 13 12 AS 1 15 8 AS 5 1 10 7 OBGP 2 2 3 10 14 1 7 9 AS 2 8 AS 6 University 5 5 6 Regional Network ISP router Potential OBGP Peering Resultant Network Topologies BGP Peering on switches at the edge Packet Forwarding in the core

25. Possible CA*net 4 Node Optional Aggregating Router 8 Channel GbE CWDM to next CA*net 4 node 4 Channel GbE CWDM to local GigaPOP CA*net 4 switch Carrier A OC48 DWDM 2xGbE Carrier A OBGP Switch Carrier B 10xGbE OC192 DWDM

26. ORAN A ORAN B ORAN C ORAN D OBGP links Dark fiber +CWDM CA*net 4 1 4 3 2 2xGbE 2xGbE 2xGbE Carrier A 6 OC48 DWDM 5 Carrier B 10xGbE 10xGbE OC192 DWDM 8 7 Physical Wavelength Configs

27. ORAN A ORAN B ORAN C ORAN D OBGP links CA*net 4 4 3 1 2 GbE over CWDM Carrier A 5 GbE over 2GbE over OC-48 DWDM Carrier B GbE over 10GbE over OC-192 DWDM Logical Wavelength Configs

28. Possible Wavelength Assignment • Illustrative purposes only • Assume 150 wavelength system across Canada • 50 wavelengths assigned to provincial networks based on a number of criteria including ability to extend wavelengths into provincial network and requirements for high bandwidth applications • ORANs encouraged to extend wavelengths to individual institutions • Institutions encouraged to deploy optical switches • On all cross sections a minimum of 100 wavelengths dedicated to CA*net 4 and carrier partners • 2 wavelengths dedicated to CA*net 4 layer 3 aggregation service (looks like old CA*net 3) • 10 wavelengths (and OCX ports) reserved for temporary applications like Grids or eScience • A wavelength and OXC port bartering and exchange mechanism so that ORANs can swap wavelengths will be an important requirement

29. Example Physical Architecture • CANARIE builds heterogeneous network made from many sources e.g (illustrative purposes only): • dark fiber from St. John to Halifax using ULH 16 channel POS • dark fiber condo from Halifax to Fredericton using 16 channel 10GbE • Condo wavelengths from RISQ from Edmonston to Ottawa sharing 32 channel 10GbE system • dark fiber from Ottawa to Winnipeg with Onet using 16 channel POS at OC-192 • Condo wavelengths from Bell Canada from Montreal to Chicago as part of a 140 wavelength system • wavelengths from Telus from Chicago to Winnipeg as part of a 140 wavelength system • dark fiber from GT Telecom from Winnipeg to Calgary using 16 channel 10GbE • wavelengths from Shaw from Calgary to Vancouver as part of a 32 channel 10GbE systems • wavelengths from 360 Networks from Halifax to London as part of a 400 wavelength system • Wavelengths from Teleglobe from Seattle to Honolulu – Sydney – Tokyo - Seoul

30. OBGP Variations • OBGP Cut Thru • OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC • OBGP Optical Peering • External router controls one or more switch ports so that it can establish direct light path connections with other devices in support peering etc • OBGP Optical Transit or QoS • To support end to end setup and tear down of optical wavelengths in support of QoS applications or peer to peer network applications • OBGP Large Scale • To prototype the technology and management issues of scaling large Internet networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers

31. OBGP Optical Peering • Primary intent is to automate BGP peering process and patch panel process • Operator initiates process by click and point to potential peer • Original St. Arnaud concept • Uses only option field in OPEN messages • Requires initial BGP OPEN message for discovery of OBGP neighbors • Virtual BGP routers are established for every OXC and new peering relationships are established with new BGP OPEN message • Full routing tables are not required for each virtual router • No changes to UPDATE messages • No optical transit as all wavelengths are owned by peer • Uses ARP proxy for routers on different subnets • Wade Hong Objects concept • Uses an external box (or process) to setup optical cross connects • SSH is used to query source router of AS path to destination router • Each optical cross connect is treated as an object with names given by AS path • Recursive queries are made to objects to discover optical path, reserve and setup • NEXT_HOP at source router is modified through SSH • End result is a direct peer and intermediate ASs disappear • Requires all devices to be on same subnet

32. OBGP Optical Transit • Wavelengths are under control of another entity who has temporarily allowed them to be available for transit • Viagenie – Marc Blanchet and Florent Parent • Designed specifically for optical transit applications • Uses MBGP and establishes new address family for OBGP • Community tags are used to advertise availability of optical paths as part of NLRI and COMMUNITY TAG • Reservation and setup is done by advertising update NLRI message • Exploring using CR-LDP & RSVP-TE with AS loose routing for path reservation and setup • Changcheng Huang • The same NLRI message is sent back and forth and modified to indicate first availability of wavelengths, reservation and setup • Over rides loop back detection in RIBS for advertised NLRI messages

33. Target Market for OBGP • University research and community networks who are deploying condominium fiber networks who want to exchange traffic between members of the community but who want to maintain customer control of the network at the edge and avoid recreating the need for aggregating traffic via traditional mechanisms • E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc • Next generation fiber companies who are building condominium fiber networks for communities and school boards and who want to offer value added fiber services but not traditional telcommunications service • E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc • Next generation collocation facilities to offer no-cost peering and wavelength routing • Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com, Colo.com, etc etc • Over 500 Ixs and carrier hotels worldwide

34. OBGP Peering • Possible technique for allowing automatic peering at IXs between consenting ISPs • External routers are given control of specific ports on the OXC • The router that controls switch can act as an optical route server notifying all peers of any new consenting OBGP peers • External routers signal to each other if they wish to setup direct optical connection • Choice of partner can be based on size of traffic flows • Partners can be changed through a routing flap • Only see each other’s customers routes – not the default core

35. OIX using OBGP AS 200 170.10.10.0 Institution A Switch Ports are part of institution’s AS Institution B AS 300 180.10.10.0 AS 100 160.10.10.0 Institution C Institution D AS 400 190.10.10.0 Figure 10.0

36. Transport Architecture • Heterogeneous transport architectures used on backbone links • Type of transport architecture on each link determined by length of link between O-ADMs, GbE-ADMs or OBGP switches, requirement for optical repeaters or regenerators, etc • Examples: • 8 or 16 channel GbE used on short haul links (up to 2000 km) between OADMs or OBGPS; or • OC-192 Ethernet over SONET with multiplexed 10 single GbE or trunked 10GbE; or • Proprietary 8 channel 2 x GbE multiplexed into OC48 optics with FEC wrapper • Repeaters: • GbE or 10GbE 2R transceivers every 50-80 km combined with GbE or 10Gbe 3R switches every 200 – 400 km; or • Traditional EDFAs at 1550nm every 50- 80 km with OC-192 regenerators at every 200-400km; or • All optical broadband: Counter rotating Raman amplifiers, multi band EDFAs, EFFs, dispersion correction fiber, etc

37. Tributary Architecture • Customer can connect through OADM,Gbe-ADM, direct to OBGP switch or through CA*net 4 router • Customer access link is either GbE or trunked 10GbE (I.e. 10 separate GbE channels • In future customer will have a choice of protocols, but for now GbE will be basic standard across the network

38. Switch Architecture • Low speed MEMs or similar capacity switch • Could also use non blocking GbE switch • Switch can also be distributed across an optical network using GMPLS or ODSI • Each switch component can be controlled by a socket/port by any external network element with appropriate security mechanisms • If OXC used for traffic engineering or QoS then controlling router manipulates both input and output ports • If OXC used for distributed peering then participating AS only owns either INPUT or OUTPUT ports • Eventually switches can also support optical trunking of many optical paths • Switch commands are kept very simple, leaving all complexity to OBGP messages • Switch does not know or care the direction of the wavelength – that is established with OBGP protocol