T1-NREN

1. T1-NREN Luca dell’Agnello CCR, 21-Ottobre-2005

2. The problem • Computing for LHC experiments • Multi tier model (MONARC) • LHC computing based on grid • Experiment computing models • LHC Computing Grid (LCG) • Policy Executive Board (PEB) • Grid Deployment Board (GDB) • Transfer LHC data from CERN to external computing facilities • Players: TO, T1s, NREN, DANTE

3. Task forces & activities • Group of interest on network set up in 2004 by NRENs • Some T1s, NRENs, CERN (D Foster), DANTE • Service Challenge activity (December 2004  service phase) • Progressive tests of data transfer chain in cooperation with experiments • ramp up to LHC data taking phase • Second group (LHCOPN wg) started by GDB in January 2005 • T1s, CERN, NRENs, DANTE represented • European NRENs, ESNET, CANARIE, ASNET • Chair: D. Foster (CERN) • Meetings: • 20-21 January 2005 (SARA/NIKHEF) • 8 April 2005 (SARA/NIKHEF) • 19 July 2005 (CERN) • Next: November 14 (SC05, Seattle)

4. LHCOPN wg activity • Preparation and control of plans for implementation of WAN connectivity requested from the LHC experiments’ Computing Models • Ensure that individual agreements among T1s will provide a coherent infrastructure to satisfy the LHC experiments' Computing Models and that there is a credible solution for the management of the end-to-end network services • First priority: to plan the networking for the Tier-0 and Tier-1 centres • should also cover Tier-2 centres, as appropriate information on requirements becomes available • The group reports regularly to the GDB • Subgroups • IP Addressing and Routing • Monitoring • Operations • Security

5. LHC “instrument” • The LHC and its data collection systems • The data processing and storage units at CERN, i.e. T0 • The data processing and storage sites called T1 • The data processing and storage sites called T2 • Associated networking between all T0, T1, and T2 sites. LHCOPN TO-T1s 10 Gbps permanent light paths form the Optical Private Network (OPN) for the LHC instrument

6. Who does what • CERN will • provide interfaces for T1’s link terminations at CERN • host T1's equipment for T0-T1 link termination at CERN (if requested) • T1s will • organise physical connectivity from the T1's premises to the T0 • make available network equipment for termination point of T1-T0 link at the T1 side • light path termination will be at T1 premises (as in INFN-CNAF case) or at NREN POP • be ready at full bandwidth not later than Q1 2006 • SCs need to test the system (from network to applications) up to full capacity production environment

7. Tier2s L3 Backbones Tier1s Tier0 Main connection Backup connection Network architecture (1) • At least one dedicated 10 Gbit/s light path between T0 and each T1 • every T0-T1 link should handle only production LHC data • T1 to T1 traffic via the T0 allowed BUT T1s encouraged to provision direct T1-T1 connectivity • T1-T2 and T0-T2 traffic will normally be handled by the normal L3 connectivity provided by NRENs • Backup through L3 paths across NRENs discouraged (potential heavy interference with general purpose Internet connectivity of T0 or the T1s) main connection backup connection

8. Network architecture (2) • LHC prefixes • Every T1 and the T0 must allocate publicly routable IP address space (the "LHC prefixes”) aggregated into single CIDR block • T1s and T2s to exchange traffic directly with the T0 must provide the T0 with the list of its LHC prefixes ( next slide) • LHC prefixes should be dedicated to the LHC network traffic • Routing among T0 and T1s sites will be achieved using eBGP (no static routes!) • No default route must be used in T1-T0 routing • Every T1 will announce its own LHC prefixes to T0 • T0 will accept from a T1 only it own LHC prefixes plus prefixes of any T1 or T2 for which that T1 is willing to provide transit for • TO will announce its LHC prefixes and all the LHC prefixes received in BGP to every peering T1 • inter T1s traffic through T0 allowed but not encouraged • T1 will accept T0's prefixes, plus, if desired, some selected T1's prefixes • T0 and T1s should announce their LHC prefixes to their upstream continental research networks (GÉANT2, Abilene, ESnet) in order to allow connectivity towards the T2s • Every Tier must make sure that any of its own machines within the LHC prefix ranges can reach any essential service (for instance the DNS system)

9. Network architecture: the T2s • T2s usually upload and download data via a particular T1 • It is possibile to provide transit for a T2 to the T0, if a T1 announces the T2's prefixes to T0 and the T0 open all the security barriers for it • BUT this assumes a “static” allocation of a T2 to a particular T1 • It is assumed that the T1-T2 and T0-T2 traffic will be handled by the normal L3 connectivity provided by NRENs. • The announcement of prefixes associated with Tier 2 sites is deprecated and any site has the right ignore such announcements.

11. Security on LHCOPN: the basics • Security contact person (+ deputy) needed for each site • Note that a T1 has also security contact persons for its own NREN, LCG/EGEE and maybe at HEPix level  • Incident handling and reporting • Local site procedures for security incident handling will take precedence • Security incident at an OPN site will be reported to the LHCOPN representatives • The report will provide a description of the incident with an assessment of the risk posed to other OPN sites • A site receiving such an incident report, the nominated OPN Security representative will share this information and will abide by the requirements of the local Security officer

12. Security on LHCOPN: implementation • It is not possible to rely on firewalls • throughput and performance problems • ACL-based network security acceptable • It is not a general access network • Low number of expected LHC prefixes • Low number of expected (Grid) applications • ACL general schema (discussion still on going) • Applied inbound and outbound at T0 and T1s borders • Only traffic originating & directed to LHC prefixes allowed on LHCOPN • IP based ACLs to control traffic from and to end points • Transit T1-T1 through TO for LHC prefixes allowed • Extended ACLs should also be used where source/destination port numbers can be associated with data flows

13. LHCOPN Operation • Issues • Circuits provided and managed by NRENs, GEANT,… • IP termination points at T0/T1s • Need for some sort of coordination and monitor of LHCOPN (LHCOPN NCC ?) • Model still under discussion • Proactive IP level monitoring • Coordination of multi circuit problems • Escalation to circuit provider/NREN • Tiers contact LHCOPN NCC

14. Italian LHCOPN Architecture T2 T2 PoP Geant2 - IT RT.RM1 STM-64 IP access RT.MI1 AT CNAF RT1.BO1 10G – leased λ’s 10GE-LANlightpath access eBGP AS513 – AS137 STM-64 PoP Geant2 - CH GARR GFP-F T1 10GE-LAN 10GE-LAN T1 CERN T0

15. INFN Tier1 • LAN connectivity based on 10GE technology with capacity for 10 GE link aggregation • Data flows will terminate on disk buffer system (possibly CASTOR, but also other SRM systems under evaluation) • 1 network prefix for LHCOPN • Security model will be based on L3 filters (ACLs) within the L3 equipment • Monitoring via snmp (presently v2)

16. T2 GARR BD CISCO 7600 INFN-CNAF WAN connectivity GEANT 128.142.224.0/20 default 10 Gbps 1 Gbps default Link LHCOPN Juniper GARR 131.154.128.0/17 131.154.0.0/16 T1 CNAF

18. Glossary • LHC Network Traffic: The data and control traffic that flows between T0, the T1s, and the T2s. • LHC prefixes: IP address space allocated by the T0 and T1s and assigned to the machines connected to the LHC-OPN. • Light path: • a point to point circuit based on WDM technology • a circuit-switched channel between two end points with deterministic behaviour based on TDM technology • concatenations of (1) and (2). • NREN: usually National Research and Education Network; here used as the collective name for a network that service either the research community or the education community or both.