GÉANT fibre and lighting Designing the next generation GÉANT optical layer

1. GÉANT fibre and lightingDesigning the next generation GÉANT optical layer Guy Roberts, DANTE 13 September, 2010

2. Contents • GÉANT network • PoPs, architecture, fibre types • Wavelength services, regeneration and channel planning • Growth projections • Fibre footprint • Infrastructure analysis - route diversity • Diversity case studies • 40/100G • 40G trial • 100G and coherent, PM-QPSK and fibre types • GÉANT3 RFI: re-engineer the photonic layer? • RFI process and outcomes

3. GÉANT Network

4. GÉANT network • Local campus networks link to NRENs which interconnect via GÉANT backbone • Transfer rates of up to 10Gbps across 50,000 km of network infrastructure • 25 Points of Presence (PoPs), 44 routes and 18 dark fibre routes

5. Dark fibre topology • 12,000 route km • Two dark fibre rings – Western Ring and Eastern Ring • Western Ring:UK-BE-NL-DE-CH-FR • Eastern Ring:SK-HU-HR-SI-AT • Rings interconnected: CH-IT-ATDE-CZ-SK • DF spurs/loops to:ES, IE, DK COP DUB BRU AMS PRA Western Ring LON FRA BRA PAR GEN BUD VIE Eastern Ring BAR MIL LJU ZAG MAD

6. A big European mesh... • GÉANT backbone is only part of the bigger picture • When NREN connectivity is overlaid can see full complexity • CBF – increasing role? • This is needs updating for: • Rediris • Garr • Hungarnet Updates please!

7. GÉANT Fibre • 6 fibre providers: • Level3 • Colt • Interoute • TeliaSonera • Invitel (previously Memorex and Pantel) • Global Crossing • G.655 (E-LEAF) • All Western ring • Madrid, vienna... • G.652 (SMF) • Routes to Copenhagen • Eastern ring (?) Some long routes!

8. Alcatel LightManager • Hybrid networking was introduced in GÉANT2 • ~100 x 10 Gb/s multi-hop wavelengths currently deployed • Lambda services delivered using Alcatel’s 1626LM equipment • 1241km unregenerate reach on G.652 Frankfurt-Copenhagen • Attenuation up to 28dB between amplifiers • Point-to-point only – no ROADM functions used • 40 Gb/s field trial successfully completed, these transponders now to be used for IP: • Amsterdam-Frankfurt • Frankfurt-Geneva 1626LM

9. Managed wavelength service POP A Manually patched at intermediate sites POP C POP D POP B GÉANT • 10G and 40G SDH clients • Static routing and OADM • Full rate 10GE LanPhy

10. GÉANT 10G lambda traffic matrix(Feb 2010) • All end-to-end wavelengths: • Wavelength services • IP wavelengths • GÉANT+ wavelengths

11. GÉANT: 100 x 10Gb/s wavelengths All lambdas....

12. Wavelength exhaust? • System is designed for up to 40 wavelengths on all routes. • G.655 fibre requires spacing of 100GHz for first 20 wavelengths with 50GHz infill available for next 20 wavelengths. • Alcatel have modified their wavelength spacing rules over the past 4 years as they refine their modelling tools. • Currently fibre is approx. 50% full on Western Ring, 19 wavelengths on Amsterdam-Frankfurt and Frankfurt-Geneva spans. • Capacity planning projections suggest that design limit of 40 channels will be exhausted in lifetime of GÉANT3 project on some routes (if no 40/100G channels are deployed).

13. Dark fibre footprint

14. Dark fibre routing • DANTE has an ongoing exercise to enter all GEANT dark fibre and leased wavelength routing into a Geographic Information System. • Currently using Google Earth – data is being distributed to NRENs • Fibre providers supply data of various formats and quality • Main goal is to identify shared risk groups (often very tricky to be certain about these; “intelligence” on deals between operators is often useful as well – swaps, purchases, etc) • Note that shared routes does not necessarily mean shared huts • For completeness this should (but does not yet) include NREN routes • Two case studies presented

15. Fibrefootprint, fibre types • fibre, routing • GÉANT dark fibre routes

16. Case study 1: Frankfurt metro • 4 dark fibre routes, 4 wavelengths and 1 CBF route all terminate in Frankfurt • DF routes go to: Amsterdam, Copenhagen, Prague, Geneva • Difficult to keep track of routing of shared routing risks. This could be a shared trench, shared duct or shared cable – these are not easy to distinguish • Frankfurt city authorities limit the number of roads available for fibres – increases risk of shared routes

17. Case study 1 - Frankfurt metro • Shared risk? • GÉANT: Frankfurt dark fibre metro routes

18. Case Study 2: LHC OPN and Geneva • Major fibre cut between Geneva and Basel - has happened at least once • Resulted in loss of both GEN-FRA and GEN-MIL • IP traffic between Geneva and Milan (and S&E of there) will reroute around a very long loop: • Geneva-Paris-London-Brussels-Amsterdam-Frankfurt-Prague-Bratislava-Vienna/Budapest... • If providing restorable wavelengths then substantial regen would be required to make the restoration path feasible

19. To Frankfurt • To Paris • To Milan • Shared risk? • To Madrid

20. Case Study 2: LHC OPN and Geneva • LHCOPN procured 3rd party leased wavelengths • Analysis of LHCOPNperformed by Michael Enrico (back in 2007) showed that leased wavelengths had some shared risks • For example, up to 8 wavelengths may share a routing risk between Geneva and Basel • Ways to improve resilience levels (3 examples, may be others): • add a (GEANT) fibre route between Marseille (+ 3-d ROADM) and Milan • add a third diverse fibre route out of Geneva to Milan - this will allow for restorable wavelength services • use a CBF route (ensuring it is based on diverse fibre) to introduce a 2nd diverse lambda route between Geneva and Milan - this will allow for restorable sub-wavelength services and improve all round IP resilience

21. DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL UK FR CH T0 NY SRLG analysis Case Study 2: LHC OPN and Geneva Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL Hamburg RAL SARA (Between CERN and BASEL) Following lambdas run in same fibre pair: CERN-GRIDKA CERN-NDGF CERN-SARA CERN-SURFnet-TRIUMF/ASGC (x2) USLHCNET NY (AC-2) Following lambdas run in same (sub-)duct/trench: (all above +) CERN-CNAF USLHCNET NY (VSNL N) [supplier is COLT] Following lambda MAY run in same (sub-)duct/trench as all above: USLHCNET Chicago (VSNL S) [awaiting info from Qwest…] MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

22. 40G and 100G

23. Alcatel 40 Gb/s p-DPSK • Technology/cost challenge led to gong delay between 10G and 40G: Alcatel & Lucent spent 10+ years investigating 40G solutions • Designed to have reach that is similar to existing 10G and no guard bands required • Partial DPSK uses filtering to fit into 50 GHz channel spacing – easy to mix with existing 10G wavelengths • Problems experienced by JANET with PMD are not seenso far on GÉANT fibres • Not clear if 40G has a future (even as an interim step) –is market moving more rapidly to 100G than envisaged sayeven 6 months ago? P-DPSK

24. 40G wavelength trial • 40G trial Frankfurt-Geneva-Milan • Juniper T1600 SDH STM-256 PICs • Ran stability tests with Xena 10G Ethernet testers & Monitored PM data • Stable operation observed 40G SDH analyser 40G PIC DE IT 40G PIC CH 40G pass-through

25. 100 Gb/s - PDM-QPSK andcoherent detection • Alcatel has developed 100G using PDM-QPSK and coherent detection • Polarization Division Multiplexing – Quadrature Phase Shift Keying • Other vendors are also using similar approaches for 100G • Coherent detection (mixing with local oscillator at receiver) allows phase information to be retained, this enables digital signal processing (DSP) to compensate for chromatic dispersion. • Eliminates the need for dispersion compensation fibre modules (DCM). • Detection of polarization modes at the receiver also allows DSP to compensate for polarization mode dispersion (PMD). • Complex Forward Error Correction (FEC) mechanisms now able to provide up to 10dB or more receiver margin, but can add latency.

26. GÉANT3 RFI:Re-engineer the photonic layer?

27. GÉANT RFI process • GÉANT has been engaging equipment vendors in an RFI process • The goal is to understand technology options for GÉANT3 • Particular focus on the WDM/OTN layer • Vendors have been provided with fibre characteristics in the Western ring of the GÉANT network • Respondents have been asked to provide designs based on fibre data and capacity assumptions described in ‘Reference Network’ – see next slide

28. Reference network for RFI • Apply to this reference network: • Full mesh of 3x10G (excl. Bru & Lux) • Full mesh of 1x 40G (ditto) • Full mesh of 1x100G (ditto) • Combination of these COP G.652 1122km G.652 1241km G.655 290km BRU AMS G.655 441km G.655 641km LUX FRA LON PAR GEN G.655 737km G.655 658km G.655 818km

29. All coherent transmission? • GÉANT network designed for 10G transponders, 100G coherent technology offers performance improvements • Should we re-engineer parts of the common photonic layer to take advantage of coherent technology? • Addition of some new ILA sites (where huts have previously been “skipped”) • Removal of dispersion compensation fibre • The addition of gain equalisers in some ILA sites • This will require a significant up-front capital investment to replace 10G transponders with 100G muxponders – cost benefit analysis required. • RFI includes questions to help this process – decision also depends on other issues such as capacity growth projections and architecture choices.

30. RegenReqs (vendor A) RegenReqs (vendor A vs. B) (noDCM, mixed fibre) 35 25 30 20 25 20 15 40G coh 40G coh 15 100G 10 100G 10 5 5 0 0 DCM - mixed noDCM - mixed noDCM - SMFonly Vendor A Vendor B Vendor B - ng100G Coherent transmission – RFI results • Easier for green field rollout • No discrete 10G coherent! Only 4/10x10G muxponders so 10G traffic matrix needs to be commensurate • How will this fit in with GÉANT fibre footprint? Also used Raman!

31. ROADMS, to use or not to use? • Are ROADMs suitable to the GÉANT backbone? • Pros: • Rapid service delivery • Restoration using GMPLS • Reduced regeneration cost on multi-hop routes • Cons: • Complex and expensive • Large floor space requirements • Where 10x10G muxponders used all 10 sub-wavelengths have to be demuxed (like old PDH) at add-drop sites • Potential benefits of reduced regeneration is limited for GÉANT due to very long routes • Restoration requires a lot of extra spare regeneration capacity to support very long restoration paths

32. Summary • It is important to share European fibre footprint information to reduce risk • Common GIF system helpful for this process • Role for CBF? • ROADMs much more appealing for GÉANT than in “pre-coherent era” • RFI results show a significant variation in reach between vendors • Not sure why yet (better xponders?, optimistic/pessimistic and/or more/less thorough modelling???) • The move to coherent technology – an opportunity exists to re-engineer the photonic layer - up-front investment needs cost-benefit analysis • All coherent transmission scheme certainly looks appealing but only if expected (lambda) traffic matrix is suitable • long-haul lambdas (not just POP to neighbouring POP) • and enough of them (e.g. 10G channels may come in chunks of 4)