LHC-O ptical P rivate N etwork NETWORK AT GRIDKA 10Gbit LAN/WAN evaluations

1. Firewall FS NAS FS NAS FS NAS German / European Tier-0/Tier-1 (T0/T1) LHC-OPN Lightpath IP viaMPLS Tunnel Tier-21) LHC Lightpath International IP RWTH-Aachen CNAF (2007 10Gbit) Uni-Dresden GSI / FZU Prague / PSNC Poland (2006/7/8 1Gbit) SARA (2007 10Gbit) Fnal Uni-Freiburg Desy (2007 1/10Gbit) MPI u. Uni-München CERN 2007 10Gbit 240Mbyte netto } SLAC Uni-Wuppertal HEP T2 T2 T1 T0 T0‘ T2 T2 DFN X-WiN AR DWDM 10GE 10GE 10GE FZK-Internet Router GridKa-Internet Router 10GE … extern dcache2) pools LoginServer ALICE/Atlas/CMS/LHCb LoginServer Babar/CDF/DZero/Compass dCache Headnode FZK-LAN intern Ethernet 1 Gbit FiberChannel 2Gbit Ethernet 10 Gbit dcache2) pools LCG(se/ce/ui) LCG(RGMA/MON/BDII) … LCG(RB/LFC/DC-SRM) T0 DWDM lightpath … router T0 DWDM lightpath Backup (end 2006) private network FS … T1 DWDM lightpath Bandwidth 10GE FS … SAN … T2 DWDM lightpath Bandwidth 1/10GE switch FS … T2 L2-IP-Tunnel (MPLS) Cisco 12416 … … GEANT Juniper M160 Juniper M160 DFN Cisco 12410 Juniper M160 IntelNIC – tuned forthroughput Chelsio Cisco 6509 ComputeNode ComputeNode Cisco 7609 ComputeNode Gridka (Karlsruhe) … CERN ComputeNode oplapro96 10gtk113 Standard mtu 8160 Bytes mtu 8180 Bytes mtu 9216 Bytes mtu Mmrbc set to 512 bytes Mmrbc set to 2048 bytes Mmrbc set to 4096 bytes Standard mtu 8180 Bytes mtu 9216 Bytes mtu Standard mtu 8180 Bytes mtu 9216 Bytes mtu Standard mtu (server side) 8180 Bytes mtu (server side) 9216 Bytes mtu (server side) Standard mtu 8160 Bytes mtu 8180 Bytes mtu 9216 Bytes mtu Standard mtu (server side) 8160 Bytes mtu (server side) 8180 Bytes mtu (server side) 9216 Bytes mtu (server side) Window size 14.0 MByte TxIntDelay=500 RxIntDelay=500 Window size 2.0 MByte TxIntDelay=500 RxIntDelay=500 Window size 8.0 MByte TxIntDelay=250 RxIntDelay=250 Window size 8.0 MByte TxIntDelay=74 RxIntDelay=74 Window size 8.0 MByte TxIntDelay=15 RxIntDelay=15 NIC – tuned for Latency • - Chelsio 0.020ms • The throughput of the Chelsio NIC is still normal after tuning for latency - Intel 0.016ms - The throughput of the Intel NICs is neglectable after tuning for latency LHC-OpticalPrivateNetwork NETWORK AT GRIDKA 10Gbit LAN/WAN evaluations - PCI-X Bus: theoretical limit 8.5 Gbps approx. - tested max. rate 7.8 Gbps (linux kernel packet generator) WAN setup -- GridKa - CERN • Routing: • External Routing via BGP • eBGP with the Forschungszentrums AS-No.: 34878exterior Boarder Gateway Protocol • iBGP synchronising the eBGP routerinterior Boarder Gateway Protocol • Internal redundance secured via OSPF • Layer-2 tunnel for distributed Storage Server via Policy Based Routing • Private network space separated into switched VLans WAN throughput / TCP cloning 1GE versus 10GE Server Nodes • Total Cost of Ownership (TCO): • Dramatically reduced costs (per NIC/high end server) • the troughput increased from 2.8Gbit to 7.0Gbit with TOE (standard MTU) • eight 1GE standard server = one 10GE Server • higher administration needs to be taken into account • PCI-Express will allow for the full rate of 10GE • 250/6000 Euro per 1GE/10GE router port 2) 1) Email: Bruno.Hoeft@iwr.fzk.de; Marc.Garcia@iwr.fzk.de Phone: +49 7247 82 8172 Web: www.gridka.de www.fzk.de/iwr „Connecting WLCG Tier-2 Centres to GridKa“ of Dr. Andreas Heiss presents more details about the at GridKa assigned Tier-2 sites Dr. Doris Ressmann will present in her talk more detailed information at the dCache setup of GridKa. „using TSM to create a high performance Tape connection“ Track: Computing Facilities and Networking, Monday 14:20-14:40