Universities and the Future of the Internet

1. CANARIE “Critical Role Universities Will Play in the Future of the Internet”The Customer Empowered Networking Revolution http://www.canarie.ca http://www.canet3.net Bill.St.Arnaud@canarie.ca Tel: +1.613.785.0426

2. Outline • The Message • CANARIE • CA*net 3 • Customer Empowered Networking • CA*net 4 and GigaPOP • eScience

3. The Message • In mid 1990s the prevailing wisdom was that commercial sector would drive design of Internet infrastructure • R&E networks would focus on applications or specialized services • As a result in North America R&E networks were commercialized or discontinued • e.g NSFnet & CA*net • However new network architectures and most importantly dark fiber is allowing R&E networks to once again redefine telecommunications and the future of the Internet • LAN architectures, technologies and most importantly LAN economics are invading the WAN • Carrier neutral IXs (in essence GigaPOPs) are the essence of this network • Control and management of the optics and wavelengths will increasingly be under the domain of the LAN customer at the GigaPOP, as opposed to the traditional carrier in the center • These new concepts in customer empowered networking are starting in the same place as the Internet started – the university and research community. • Internet 2, SURFnet5 and CA*net 3 & 4

4. CANARIE Inc • Mission: To facilitate the development of Canada’s communications infrastructure and stimulate next generation products, applications and services • Canadian equivalent to Internet 2 and NGI • private-sector led, not-for-profit consortium • consortium formed 1993 • federal funding of $300m (1993-99) • total project costs estimated over $600 M • currently over 140 members; 21 Board members

5. CA*net 3 National Optical Internet Consortium Partners: Bell Nexxia Nortel Cisco JDS Uniphase Newbridge CA*net 3 Primary Route CA*net 3 Diverse Route GigaPOP ORAN Deploying a 4 channel CWDM Gigabit Ethernet network – 400 km Deploying a 4 channel Gigabit Ethernet transparent optical DWDM– 1500 km Condo Dark Fiber Networks connecting universities and schools Condo Fiber Network linking all universities and hospital Multiple Customer Owned Dark Fiber Networks connecting universities and schools Netera MRnet SRnet ACORN St. John’s BCnet Calgary Regina Winnipeg Charlottetown RISQ ONet Fredericton Montreal Vancouver 16 channel DWDM -8 wavelengths @OC-192 reserved for CANARIE -8 wavelengths for carrier and other customers Halifax Ottawa Seattle STAR TAP Toronto Chicago New York

6. Customer Empowered Networks • Universities in Quebec are building their own 3500km fiber condo network in partnership with number of carriers • Universities in Alberta are deploying their own 700 km 4xGbe dark fiber network • Soon will be extended to all communities in Alberta • School boards and municipalities throughout North America are deploying their condo open access, dark fiber networks • Illinois Iwire, Indiana Gwire, California DCP, Georgia SURAnet, Stockholm condo fiber builds • Chicago CivicNET, Alberta SuperNET, Canadian NBTF • Carrier are selling “dim wavelengths” managed by customer to interconnect dark fiber networks • Williams, Level 3, Hermes • Typical cost is one time $20K US per school for a 20 year IRU for condominium fiber connection

7. What is condominium fiber? • A number of organizations such as schools, hospitals, businesses and universities get together to fund and build a fiber network • Carrier partners are also invited to be part of condominium project • Several next generation carriers and fiber brokers are now arranging condominium fiber builds • IMS, QuebecTel, Videotron, Cogeco, Dixon Cable, GT Telecom, etc etc • Fiber is installed, owned and maintained by 3rd party professional fiber contractors – usually the same contractors used by the carriers for their fiber builds • Each institution gets its own set of fibers, at cost, on a 20 year IRU (Indefeasible Right of Use) • One time up front cost, plus annual maintenance and right of way cost approx 5% of the capital cost • Institution lights up their own strands with whatever technology they want – Gigabit Ethernet, ATM, PBX, etc • New long range laser will reach 120 km • Ideal solution for point to point links for large fixed institutions • Payback is usually less than 18 months

8. Why Condo Fiber? • First - low cost • Up to 1000% reduction over current telecom prices. 6-12 month payback • Second - LAN invades the WAN – no complex SONET or ATM required in network • Network Restoral & Protection can be done by customer using a variety of techniques such as wireless backup, or relocating servers to a multi-homed site, etc • Third - Enables new applications and services not possible with traditional telecom service providers • Relocation of servers and extending LAN to central site • Out sourcing LAN and web servers to a 3rd party because no performance impact • IP telephony in the wide area (Spokane) • HDTV video • Fourth – Allows access to new competitive low cost telecom and IT companies at carrier neutral meet me points • Much easier to out source servers, e-commerce etc to a 3rd party at a carrier neutral collocation facility

9. Quebec University Condo Network Construit Projet démarré À venir Bande passante louée MAN de Montréal MAN d’Ottawa/Hull MAN de Québec MAN de Sherbrooke Val d’Or/Rouyn Observatoire Mont-Mégantic

10. Construit Projet démarré À venir Bande passante louée Lionel-Groulx Lanaudière Sorel-Tracy Montreal Public Sector Condominium Networks Marie-Victorin Rosemont Montmorency Maisonneuve Ahuntsic Édouard-Montpetit Bois-de-Boulogne Vers Québec St-Laurent/Vanier Champlain Vieux-Montréal Gérald-Godin Dawson John-Abbott André-Laurendeau

11. List of Schoolboard Fiber Builds

12. Alberta SUPERnet • Province wide network of condominium fiber to 420 communities in Alberta • Guaranteed cost of bandwidth to all public sector institutions • $500/mo for 10 Mbps, $700/mo for 100 Mbps • Network a mix of fibre builds and existing supplier infrastructure (swap/buy/lease) • Condominium approach: All suppliers can • Buy (or swap) a share of the fibre (during build or after) • Lease bandwidth at competitive rates • GOA has perpetual right to use (IRU) • Ownership will be held at arms length • GOA/stakeholder rates are costs to run divided over users • Because of fibre capacity, bandwidth can be made available to businesses at urban competitive rate • Total cost $193m • Bell Intrigna prime contractor

13. National Broadband Task Force • Mandate:To map out a strategy and advise the Government on best approaches to make high-speed broadband Internet services available to businesses and residents in all Canadian communities by the year 2004. • To ensure Canada’s competitiveness in a global economy • To address the Digital Divide • To create opportunities for all Canadians • 35 members including carriers, educators, librarians, communities, equipment manufacturers, etc • Chair – David Johnston

14. Fredericton Fiber Build • Started as Economic Development tool • MUSH, Govt., Research - ISP, carriers invited to participate • Build partners emerged quickly, $50,000 “donated” by three firms • Contracting now for 8 km phase 1, $110,000, complete Sept 2001 • 48 fiber min. • Unique experiment to extend off campus Internet access through 802.11 wireless

16. Ottawa Fiber Condominium • Consortium consists of 16 members from various sectors including businesses, hospitals, schools, universities, research institutes • 26 sites • Point-to-point topology • 144 fibre pairs • Route diversity requirement for one member • 85 km run • $11k - $50K per site • Total project cost $CDN 1.25 million • Cost per strand less than $.50 per strand per meter • 80% aerial • Due to overwhelming response to first build – planning for second build under way

18. Condo Fiber Costs - Examples • Des affluents: Total cost $1,500,00 ($750,00 for schools) • 70 schools • 12 municipal buildings • 204 km fiber • $1,500,000 total cost • average cost per building - $18,000 per building • Mille-Isles: Total cost $2,100,000 ($1,500,000 for schools) • 80 schools • 18 municipal buildings • 223km • $21,428 per building • Laval: Total cost $1,800,000 ($1,000,000 for schools) • 111 schools • 45 municipal buildings • 165 km • $11,500 per building • Peel county: Total cost $5m – 100 buildings • Cost per building $50,000

19. Typical Payback for school(Real example – des affluents – north of Montreal) • Over 3 years total expenditure of $1,440,000 for DSL service • Total cost of dark fiber network for 75 schools $1,350,000 • Additional condominium participants were brought in to lower cost to school board to $750,000 • School board can now centralize routers and network servers at each school • Estimated savings in travel and software upgrades $800,000 • Payback typically 8 –16 months • Independent Study by Group Secor available upon request

20. Before After fiber fiber • Antennas 78 0 • Novell Servers 82 1 • SQL Servers 13 3 • Lotus Notes Servers 2 1 • Tape Backup Servers 12 4 • Ethernet switches/hubs 10 98 • Routers 108 3 • Cache/proxy (Linux) 12 0 • Fire walls (Linux) 1 1 Reduction in the number of servers

21. Community Fiber Architecture • A community consortia would put together a plan to fiber up all public sector buildings in their community • A community can be a province, a municipality, village, etc • A fiber splice box that terminates the fiber at the street side nearby each public sector building such as school, hospital, library is called a “Node” • Community should must insure that potential facilities exist near the for private sector equipment to connect up future home owners – colo facility • Colo facility allows private sector to extend wireless, VDSL or HFC services to the neighbourhood around the school • Public sector buildings will have dedicated fiber strands that connect to a “Supernode” which is a fiber splice box on the street beside outside of major public sector central facility such as school board office, city hall, university, etc • Community should insure that facilities exist nearby the Supernode for the private sector to install equipment to service home owners and businesses – colo facility • Additional fibers are made available from the Supernode to all Nodes such that competitive service providers can purchase fiber to the node at some future date

22. Possible architecture for large town Carrier Owned Fiber Central Office For Wireless Company Cable head end School board office Telco Central Office Condominium Fiber with separate strands owned by school and by service providers Colo Facility School School 802.11b Average Fiber Penetration to 250-500 homes VDSL, HFC or Fiber Provisioned by service provider

23. Benefits to Industry • For cablecos and telcos it help them accelerate the deployment of high speed internet services into the community • Currently deployment of DSL and cable modem deployment is hampered by high cost of deploying fiber into the neighbourhoods • Cable companies need fiber to every 250 homes for cable modem service, but currently only have fiber on average to every 5000 homes • Telephone companies need to get fiber to every 250 homes to support VDSL or FSAN technologies • Wireless companies need to get fiber to every 250 homes for new high bandwidth wireless services and mobile Internet • It will provide opportunities for small innovative service providers to offer service to public institutions as well as homes • For e-commerce and web hosting companies it will generate new business in out sourcing and web hosting • For Canadian optical manufacturing companies it will provide new opportunities for sales of optical technology and components

24. CA*net 4 Overall Objective • To deploy a novel new optical network that gives GigaPOPs at the edge of the network (and ultimately their participating institutions) to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage • To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of QoS and eScience applications • To allow connected regional and community networks to setup transit wavelength peering relationships with similar like minded networks to reduce the cost of Internet transit • To offer an “optional” layer 3 aggregation service for those networks that require or want such a facility

25. CA*net 4 Possible Architecture Layer 3 aggregation service Optional Service Available to any GigaPOP St. John’s Regina Calgary Winnipeg Large channel WDM system Charlottetown Europe Vancouver Montreal Customer controlled optical switches Fredericton Halifax Seattle Ottawa Chicago New York Toronto

26. eScience • The ultimate goal of e-science is to allow students and eventually members of the general public to be full participants in basic research. • We have seen in other fields like bird census, comet watching, SETI@home, public are interested in participating in basic research • Using advanced high speed networks like CA*net 4 and novel new concepts in distributed peer to peer computing, called “Grids” many research experiments that used to require high end super computers can now use the computer capabilities of thousands of PCs located at our schools and in our homes. • High performance computers that are part of C3.ca can be seamlessly integrated with eScience distributed computers using CANARIE Wavelength Disk Drive over CA*net 4 • Allows researcher access to the significant computational capabilities of all these distributed computers at our schools and homes • Will also allow students and individuals to be a full participant in the analysis and basic research. • With e-science it might be possible that the next big scientific discovery could be by a student at your local school.

27. SETI@homeDemonstrated that PC Internet Computing Could Grow to Megacomputers • Running on 500,000 PCs, ~1000 CPU Years per Day • Over Half a Million CPU Years so far! • 22 Teraflops sustained 24x7 • Sophisticated Data & Signal Processing Analysis • Distributes Datasets from Arecibo Radio Telescope Arecibo Radio Telescope Next Step- Allen Telescope Array

28. Forest Grid– on CA*Net 4 Univ. ORAN Labs/Ministries. Min. of For. CFS Northern York CFS Laurentian Min. of For. U of A Cornerbrook Min. of For. Min. of For. Waterloo Quebec. UNB Ont. Alberta Sask. Man. CCRS UVic BC Quebec. Maritimes RSI EDC CFS GLFC CSA U. Ottawa Min. of For. CFS CFS-HQ CFS Atlantic Min. of For. UBC Chicago STARTAP. PFC NSF VBNS Min. of For. European New York UCAID Abilene Seattle JPL NISN ESnet DREN NREN

29. Neptune eScience Grid • Joint US-Canadian project to build large undersea dark fiber network off west coast of USA and Canada • Undersea network will connect instrumentation devices, robotic submarines, sensors, under sea cameras, etc • All devices available to students and researchers connected to CA*net 4 and Internet 2 networks • Neptune will be used to gather research data in a variety of fields – seismology, sea vents, fish migrations and population, deep sea aquatic life, etc • Distributed computing and data storage devices on CA*net 4 and Internet 2 will be used to analyze and store data

30. Neptune – Undersea Grid

31. Neptune eScience

32. Wavelength Disk Drives St. John’s Regina CA*net 3/4 Calgary Winnipeg Charlottetown Montreal Halifax Fredericton Vancouver WDD Node Ottawa Toronto

33. Wavelength Disk Drives • CA*net 3 and CA*net will be “nation wide” virtual disk drive for grid applications • Big challenges with grids or distributed computers is performance of sending data over the Internet • TCP performance problems • Congestion • Rather than networks being used for “communications” they will be a temporary storage device

34. Conclusion • Many governments have recognized the importance of access to low cost dark fiber as fundamental economic enabler • It will be the 21st century equivalent to the roads and railways that were built in the 20th century