1 / 21

UltraLight: A Managed Network Infrastructure for HEP

UltraLight: A Managed Network Infrastructure for HEP. Shawn McKee University of Michigan. CHEP06, Mumbai, India February 14, 2006. Introduction . The UltraLight Collaboration was formed to develop the network as a managed component of our infrastructure for High-Energy Physics

jane
Download Presentation

UltraLight: A Managed Network Infrastructure for HEP

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UltraLight: A Managed Network Infrastructure for HEP • Shawn McKee • University of Michigan • CHEP06, Mumbai, India • February 14, 2006

  2. Introduction • The UltraLight Collaboration was formed to develop the network as a managed component of our infrastructure for High-Energy Physics • (See Rick Cavanaugh’s overview from yesterday) • Given a funding level at 50% of our request we had to reduce our scope while maintaining the core network infrastucture. • This talk will focus on the UltraLight network and the activities of the UL network group to enable effective use of 10 Gbps networks. S. McKee, UM - CHEP06

  3. Overview • Overview of UltraLight Network Effort • Network Engineering Plans and Status • Monitoring • Kernel/End-host Development • Status and Summary S. McKee, UM - CHEP06

  4. The UltraLight Network Engineering Team S. McKee (UM,Team Leader) S. Ravot (LHCNet) D. Nae (LHCNet) R. Summerhill (Abilene/HOPI) D. Pokorney (FLR) S. Gerstenberger (MiLR) C. Griffin (UF) S. Bradley (BNL) J. Bigrow (BNL) Many individuals have worked deploying and maintaining the UltraLight Network J. Ibarra (WHREN, AW) C. Guok (ESnet) L. Cottrell (SLAC) C. Heerman (I2/HOPI) D. Petravick (FNAL) M. Crawford (FNAL) R. Hockett (UM) E. Rubi (FIU) S. McKee, UM - CHEP06

  5. UltraLight Backbone • UltraLight has a non-standard core network with dynamic links and varying bandwidth inter-connecting our nodes. • Optical Hybrid Global Network • The core of UltraLight is dynamically evolving as function of available resources on other backbones such as NLR, HOPI, Abilene and ESnet. • The main resources for UltraLight: • LHCnet (IP, L2VPN, CCC) • Abilene (IP, L2VPN) • ESnet (IP, L2VPN) • UltraScienceNet (L2) • Cisco NLR wave (Ethernet) • Cisco Research Service (L3) • HOPI NLR waves (Ethernet; provisioned on demand) • UltraLight nodes: Caltech, SLAC, FNAL, UF, UM, StarLight, CENIC PoP at LA, CERN, Seattle S. McKee, UM - CHEP06

  6. UltraLight Network Infrastructure Elements • Trans-US 10G s Riding on NLR, Plus CENIC, FLR, MiLR • LA – CHI (4 Waves): HOPI (2 Waves), USN, and Cisco Research • CHI – JAX (Florida Lambda Rail/NLR) • Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab); 10G Wave L.A. to Sunnyvale for UltraScience Net Connection • Dark Fiber with 10G Waves (2 Waves): StarLight – Fermilab • Dedicated Wave StarLight (1 + 2 Waves) – Michigan Light Rail • SLAC: ESnet MAN to Provide Links (from July): One for Production, and One for Research • Partner with Advanced Research & Production Networks • LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, GLIF, UKLight, CA*net4 • Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Tokyo, AARNet, Taiwan, China S. McKee, UM - CHEP06

  7. UltraLight Network Infrastructure Elements • Trans-US 10G s Riding on NLR, Plus CENIC, FLR, MiLR • LA – CHI (4 Waves): HOPI (2 Waves), USN, and Cisco Research • CHI – JAX (Florida Lambda Rail/NLR) • Dark Fiber Caltech – L.A.: 2 X 10G Waves (One to WAN In Lab); 10G Wave L.A. to Sunnyvale for UltraScience Net Connection • Dark Fiber with 10G Waves (2 Waves): StarLight – Fermilab • Dedicated Wave StarLight (1 + 2 Waves) – Michigan Light Rail • SLAC: ESnet MAN to Provide Links (from July): One for Production, and One for Research • Partner with Advanced Research & Production Networks • LHCNet (Starlight- CERN), Abilene/HOPI, ESnet, NetherLight, GLIF, UKLight, CA*net4 • Intercont’l extensions: Brazil (CHEPREO/WHREN), GLORIAD, Tokyo, AARNet, Taiwan, China S. McKee, UM - CHEP06

  8. UltraLight Points-of-Presence • StarLight (Chicago) • HOPI (2 x 10GE), USNet (2 x 10GE), NLR (4 x 10GE) • UM (3 x 10GE), TeraGrid, ESnet, Abilene • FNAL, US-LHCNet (2 x 10GE), TIFR (Mumbai) • MANLAN (New York) • HOPI (2 x 10GE), US-LHCNet (2 x 10GE), BNL, • Buffalo (2 x 10GE), Cornell, Nevis • Seattle • GLORIAD, JGN2, Pwave, NLR (2 x 10GE) • CENIC (Los-Angeles) • HOPI (2 x 10GE), NLR (4 x 10GE) • Caltech (2 x 10GE), Pwave • Level3 (Sunnyvale) • USNet (2 x 10GE), NLR, SLAC S. McKee, UM - CHEP06

  9. International Partners • One of the UltraLight program’s strengths is the large number of important international partners we have: UltraLight is well positioned to develop and coordinate global advances to networks for LHC Physics S. McKee, UM - CHEP06

  10. UltraLight Global Services • Global Services support management / co-scheduling of multiple resource types with a strategic end-to-end view. • Provide strategic recovery mechanisms from system failures • Schedule decisions based on CPU, I/O, Network capability and End-to-end task performance estimates, incl. loading effects • Constrained by local and global policies • Global Services Consist of: • Network and System Resource Monitoring • Network Path Discovery and Construction Services • Policy Based Job Planning Services • Task Execution Services • These types of services are required to deliver a managed network. See “VINCI” talk on Thursday S. McKee, UM - CHEP06

  11. UltraLight Network Engineering • GOAL: Determine an effective mix of bandwidth-management techniques for this application-space, particularly: • Best-effort and “scavenger” using “effective” protocols • MPLS with QOS-enabled packet switching • Dedicated paths arranged with TL1 commands, GMPLS • PLAN: Develop, Test the most cost-effective integrated combination of network technologies on our unique testbed: • Exercise UltraLight applications on NLR, Abilene and campus networks, as well as LHCNet, and our international partners • Deploy and systematically study ultrascale protocol stacks (such as FAST) addressing issues of performance & fairness • Use MPLS/QoS and other forms of BW management, to optimize end-to-end performance among a set of virtualized disk servers • Address “end-to-end” issues, including monitoring and end-hosts S. McKee, UM - CHEP06

  12. UltraLight: Effective Protocols • The protocols used to reliably move data are a critical component of Physics “end-to-end” use of the network • TCP is the most widely used protocol for reliable data transport, but is becoming ever more ineffective for higher and higher bandwidth-delay networks. • UltraLight is exploring extensions to TCP (HSTCP, Westwood+, HTCP, FAST, MaxNet) designed to maintain fair-sharing of networks and, at the same time, to allow efficient, effective use of these networks. • We identified a need to provide an “UltraLight” kernel to make protocol testing easy among the UltraLight sites. • UltraLight plans to identify the most effective fair protocol and implement it in support of our “Best Effort” network components. S. McKee, UM - CHEP06

  13. FAST others FAST Protocol Comparisons • Gigabit WAN • 5x higher utilization • Small delay • Random packet loss • 10x higher thruput • Resilient to random loss FAST: 95% Reno: 19% S. McKee, UM - CHEP06

  14. MPLS/QoS for UltraLight • UltraLight plans to explore the full range of end-to-end connections across the network, from best-effort, packet-switched through dedicated end-to-end light-paths. • MPLS paths with QoSattributes fill a middle ground in this network space and allow fine-grained allocation of virtual pipes, sized to the needs of the application or user. TeraPaths SC|05 QoS Data Movement BNL→UM UltraLight, in conjunction with the DoE/MICS funded TeraPaths and OSCARS effort, is working toward extensible solutions for implementing such capabilities in next generation networks See “Terapaths” talk in previous session today S. McKee, UM - CHEP06

  15. Optical Path Developments • Emerging “light path” technologies are arriving: • They can extend and augment existing grid computing infrastructures, currently focused on CPU/storage, to include the network as an integral Grid component. • Those technologies seem to be the most effective way to offer network resource provisioning on-demand between end-systems. • We have developed a multi-agent system for secure light path provisioning based on dynamic discovery of the topology in distributed networks. [See VINCI talk on Thursday, Feb. 16] • We are working to further develop this distributed agent system and to provide integrated network services capable to efficiently use and coordinate shared, hybrid networks and to improve the performance and throughput for data intensive grid applications. • This includes services able to dynamically configure routers and to aggregate local traffic on dynamically created optical connections. S. McKee, UM - CHEP06

  16. Monitoring for UltraLight • Realtime end-to-end Network monitoring is essentialfor UltraLight. You can’t manage what you can’t see! • We need to understand our network infrastructure and track its performance both historically and in real-time to enable the network as a managed robust component of our infrastructure. MonALISA http://monalisa.cern.ch IEPM http://www-iepm.slac.stanford.edu/bw/ • We have a new effort to push monitoring to the “ends” of the network: the hosts involved in providing services or user workstations. S. McKee, UM - CHEP06

  17. MonALISA UltraLight Repository • The UL repository: http://monalisa-ul.caltech.edu:8080/ S. McKee, UM - CHEP06

  18. TCP Settings Host/System Information Network Device Information Host Monitoring with UltraLight • Many “network” problems are actually endhost problems: misconfigured or underpowered end-systems • The LISA application (see Iosif’s talk later) was designed to monitor the endhost and its view of the network. • For SC|05 we developed a Perl script to gather the relevant host details related to network performance and integrated the script with ApMon (an API for MonALISA) to allow us to “publish” this data to a MonALISA repository. • Information on the system information, TCP configuration and network device setup was gathered and accessible from one site. • Future plans are to coordinate this with LISA and deploy this as part of OSG. The Tier-2 centers are a primary target. S. McKee, UM - CHEP06

  19. UltraLight Kernel Kernels and the associated device drivers are critical to the achievable performance of hardware and software. The FAST protocol implementation for Linux requires a modified kernel to work. . We learned to deal with many pitfalls in the configuration and varying versions of linux kernels, particularly how they impact the performance of the system on the network. Currently we are working on a new version based upon 2.6.15.3. Will include FAST, NFSv4, newest drivers for 10GE NICs and RAID cards. Because of the need to have FAST easily available, and the desire to achieve the best performance possible, we plan for all UltraLight “hosts” to install and utilize this Kernel. S. McKee, UM - CHEP06

  20. End-Systems performance • Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec) - 8 TCP streams from CERN to Caltech; windows, 1TB file, 24 JBOD disks • Quad Opteron AMD848 2.2GHz processors with 3 AMD-8131 chipsets: 4 64-bit/133MHz PCI-X slots. • 3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disks • Local Disk IO – 9.6 Gbits/sec (1.2 GBytes/sec read/write, with <20% CPU utilization) • 10GE NIC • 10 GE NIC – 9.3 Gbits/sec (memory-to-memory, with 52% CPU utilization, PCI-X 2.0 Caltech-Starlight) • 2*10 GE NIC (802.3ad link aggregation) – 11.1 Gbits/sec (mem-2-mem) • Need PCI-Express?, TCP offload engines • Use 64 bit OS? Which architectures and hardware? • GOAL: Single server to server at 1 GByte/sec • Discussions are underway with 3Ware, Myricom to try to prototype viable servers capable of driving 10 GE networks in the WAN. • Targeting Spring 2006 S. McKee, UM - CHEP06

  21. UltraLight Network Summary • The network technical group has been hard at work on implementing UltraLight. Network and basic services are deployed and operating. • Significant progress has been made in monitoring, kernels, prototype services and disk-to-disk transfers in the WAN. • Our global collaborators are working with us on achieving the UltraLight vision. • 2006 will be busy, critical (for LHC) and productive year working on UltraLight! S. McKee, UM - CHEP06

More Related