1 / 38

LHCOPN lambda and fibre routing Episode 4 (the path to resilience…)

LHCOPN lambda and fibre routing Episode 4 (the path to resilience…). Michael Enrico Network Engineering & Planning Manager DANTE LHC Meeting, Geneva, 06-07.11.2007. ?. Connect. Communicate. Collaborate. ???. T0-T1s:. DK. ES. IT. CERN-RAL. CERN-PIC. CERN-IN2P3. CERN-CNAF. SURFnet.

joyceberg
Download Presentation

LHCOPN lambda and fibre routing Episode 4 (the path to resilience…)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LHCOPN lambda and fibre routing Episode 4(the path to resilience…) Michael Enrico Network Engineering & Planning Manager DANTE LHC Meeting, Geneva, 06-07.11.2007

  2. ?

  3. Connect. Communicate. Collaborate ??? T0-T1s: DK ES IT CERN-RAL CERN-PIC CERN-IN2P3 CERN-CNAF SURFnet CERN-GRIDKA T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 CERN-NDGF NL CERN-SARA UK CERN-TRIUMF CERN-ASGC USLHCNET NY (AC-2) USLHCNET NY (VSNL N) USLHCNET Chicago (VSNL S) FR CH T0 NY T0-T1 Lambda routing (schematic, previously) Copenhagen ASGC Via SMW-3 or 4 (?) TRIUMF NDGF ??? BNL Hamburg RAL SARA MAN LAN London Amsterdam Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  4. Connect. Communicate. Collaborate ??? T0-T1s: DK ES IT CERN-RAL CERN-PIC CERN-IN2P3 CERN-CNAF SURFnet CERN-GRIDKA T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 CERN-NDGF NL CERN-SARA UK CERN-TRIUMF CERN-ASGC USLHCNET NY (AC-2) USLHCNET NY (VSNL N) USLHCNET Chicago (VSNL S) FR CH T0 NY Add T1-T1 Lambda routing (schematic, previously) T0-T1 Lambda routing (schematic, previously) Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF T1-T1s: ??? BNL GRIDKA-CNAF SARA Hamburg RAL GRIDKA-IN2P3 GRIDKA-SARA MAN LAN London SARA-NDGF Amsterdam Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  5. Connect. Communicate. Collaborate ??? DK ES IT KEY GEANT2 SURFnet NREN T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 USLHCNET NL UK Via SURFnet T1-T1 (CBF) FR CH T0 NY SRLG analysis Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  6. Failure scenarios (2 of them)

  7. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL UK FR CH T0 NY e.g. Big fibre cut Basel-Geneva Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  8. Connect. Communicate. Collaborate e.g. Big fibre cut Basel-Geneva ?

  9. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL UK FR CH T0 NY e.g. GÉANT2 Fra-Gen EDFA failure Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  10. Connect. Communicate. Collaborate e.g. GÉANT2 Fra-Gen EDFA failure ?

  11. (Reminder of) Conclusions(from last time) • RAL & PIC need backups!!! • CNAF needs a 3rd route into CERN • SARA-NDGF should be routed on northern leg of NORDUnet’s “Southern Cross” (for Cop-Ham part) • via Padborg, Kolding, Odense • Reroute 2 Amsterdam-CERN lambdas via Lyon • Reroute USLHCNET (FNAL-CERN) via Lyon • Further investigations required in particular concerning: • Physical routing of GRIDKA-IN2P3 in Paris area • Leased lambdas passing through London

  12. Mitigations (on the way)…

  13. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL UK FR CH T0 NY Route SARA-CERN via Lyon Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  14. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL AC-1 UK FR CH T0 NY Add USLHCNET lambda #4 Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  15. Connect. Communicate. Collaborate Big fibre cut Basel-Geneva ? ? ?

  16. Connect. Communicate. Collaborate Big fibre cut Lyon-Geneva ? ?

  17. Connect. Communicate. Collaborate Big fibre cut Zurich-Basel ?

  18. Still need to…

  19. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL AC-1 UK FR CH T0 NY Add RAL backup via Frankfurt Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Atlantic Ocean FNAL Zurich Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  20. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL AC-1 UK FR CH +1 T0 NY Add CNAF backup via Vienna Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Prague Atlantic Ocean FNAL Zurich Vienna Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  21. And add PIC backup via Paris (or CBF arrangement) • Can use GÉANT+ (for 1 or 2 Gbit/s circuit) temporarily • Lease lambda Madrid-Paris • Extend lambda via London-Amsterdam-FFM to CERN • Or maybe use CBF for Paris-Strasbourg/Kehl-GRIDKA? • OR can extend lambda down to Geneva via Lyon • Via GÉANT2 fibre or RENATER fibre • OR can use LCAS semi-protected 10GE service… • CBF prospect (via Perpignan) has been mooted in the past but prospect not currently looking good…

  22. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL AC-1 UK FR CH +1 T0 NY Add PIC backup (GÉANT+) Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Prague Atlantic Ocean FNAL Zurich Vienna Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  23. Connect. Communicate. Collaborate ??? DK ES IT SURFnet T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 NL AC-1 UK Then where? Various options exist FR CH +1 T0 NY Add PIC backup (full lambda) Copenhagen ASGC TRIUMF Via SMW-3 or 4 (?) NDGF ??? BNL SARA Hamburg RAL MAN LAN London Frankfurt AC-2/Yellow DE VSNL N VSNL S Paris GRIDKA Starlight Strasbourg/Kehl Stuttgart Prague Needs to be leased Atlantic Ocean FNAL Zurich Vienna Basel Lyon Madrid Barcelona Milan GENEVA IN2P3 CNAF PIC

  24. Connect. Communicate. Collaborate LCAS semi-protected 10GE (is technically possible) • Caveats: • OWD is the longest of the two paths (~13ms for Amsterdam-Geneva) • not quite full 10GE LAN PHY rate (~9.6Gbps) • capacity reduced by 50% during single outage • 10-20s of full outage following failure event (further testing under way) • “protection” is only across GÉANT2 • only possible for Ethernet LAN-PHY services • (more details at end) 10GE LAN MCC 5G (32 x VC-4) 5G (32 x VC-4) 10GE LAN MCC

  25. Updated “what-if” scenarios…

  26. Connect. Communicate. Collaborate Big fibre cut Basel-Geneva ? ? ?

  27. Connect. Communicate. Collaborate Big fibre cut Lyon-Geneva ? ?

  28. Connect. Communicate. Collaborate Big fibre cut Zurich-Basel ?

  29. Connect. Communicate. Collaborate Big fibre cut Strasbourg area ?

  30. Latest Conclusions • SARA-CERN lambda has been rerouted (or soon will be) • 4th diverse USLHCNET lambda will be added • Can do “LCAS protected 10GE” for ASGC • RAL & PIC still need backups!!! • LCAS “semi-protected” 10GE is technically possible but with caveats • CNAF needs a 3rd route into CERN • Long route around “eastern ring” OR • New CBF solution(s)… • Further investigations required in particular concerning: • Physical routing of GRIDKA-IN2P3 in Paris area • Leased lambdas passing through UK

  31. Questions?

  32. Additional slides

  33. LCAS testing (10GE)

  34. LCASTestbed Measurements • LCAS evaluation is performed on the DANTE testbed • Iperf servers are connected directly to MCC 10GE ports • Alcatel’s ‘diverse routing’ option alternates VC-4 trails between two diverse paths • VCG supports up to 32ms of path length difference –note that the net path latency is equal to the latency of the LONGEST path!

  35. LCASTestbed Measurements First diverse path: • London – Amsterdam – Frankfurt – London has a one way delay of 15.5ms

  36. LCASTestbed Measurements Second diverse path • London – Amsterdam – Prague – Paris - London has a one way delay 27.7 ms • Path latency difference is 12.2 ms

  37. LCASTestbed Measurements • Throughput measurements are performed using Iperf on two servers with 10GE NICs in London • TCP and UDP measurements • 3 types of outages: • loss of optical signal on one leg • Remove cross-connects • Adjust bandwidth using LCAS

  38. TCP throughput:Fibre break Outage time TCP slow recovery LCAS recovery

More Related