R. Cavanaugh University of Florida (Richard.Cavanaugh@cern.ch) T3 Meeting Princeton

1. The Project : Enabling Data Intensive Science at Universitieshttp://ultralight.org Collaboration Caltech (lead inst.) BNL Michigan MIT Florida Florida International FNAL San Diego SLAC Vanderbilt R. Cavanaugh University of Florida (Richard.Cavanaugh@cern.ch) T3 Meeting Princeton

2. UltraLight: A New Class of Integrated Information Systems • Delivering the next generation of network-aware real-time Grids • The network as an integrated, managed resource • Hybrid packet-switched + dynamic optical paths • Leveraging Trans-US, Transatlantic & Transpacific network partnerships; • With ESnet, USNet, KEK, Kreonet, GLORIAD, CHEPREO, WHREN/LILA, Awave, FLR, Pacific Wave, Translight, Netherlight • Extensions to Korea, Brazil, Japan and Taiwan • MONALISA/VINCI: End-to-end monitoring, tracking, dynamic BW provisioning and workflow optimization

3. UltraLight Network Laboratory Testbed & Facility

4. What is Achievable on the Testbed? Rate [Gbs] Example: • Supercomputing 2005 • 151 Gbps peak rate • 100+ Gbps sustained throughput for hours • 475 Terabytes of physics data transported in less than 24 hours • Sustained rate of 100+ Gb/s translates to > 1 Petabyte per day 100 Gbps mark 0 15 30 45 60 t [min] 476 TB Aggregate in 24 hours http://supercomputing.caltech.edu/ Cumulative [TB] 0 6 12 18 24 t [hours] End-2-end monitoring using MonALISA

5. Caltech/CERN & HEP at SC2006: Petascale Transfers for Physics ~200 CPU 56 10GE Switch Ports50 10GE NICs 100 TB Disk Research PartnersFNAL, BNL, UF, UM, ESnet, NLR, FLR, Internet2, ESNet, AWave, SCInet,Qwest, UERJ, UNESP, KNU, KISTI Corporate PartnersCisco, HP NeterionMyricom DataDirectBlueArc NIMBUS New Disk-Speed WAN Transport Apps. for Science (FDT, LStore)

6. FDT – Fast Data Transport A New Application for Efficient Data Transfers http://monalisa.cern.ch/FDT/ • Reading and writing at disk speed over wide area networks (standard TCP) for the first time • Portable and easy to use: runs on all major platforms. • Based on an asynchronous, flexible multithreaded system, using the Java NIO libraries, that: • Streams a dataset (list of files) continuously, from a managed poolof buffers in kernel space, through an open TCP socket • Smooth flow of data from each disk • No protocol start phase between files • Uses independent threads to read and write on each physical device • Transfers data in parallel on multiple TCP streams, when necessary • Uses appropriate-sized buffers for disk I/O and for the network • Restores the files from buffers asynchronously • Resumes a file transfer session without loss, when needed • GSI Authentication built in FDT being deployed within the Alice experiment

7. New Capability Level: 40-70 Gbps per rack of low cost 1U servers FDT: Fast Data Transport Results 11/14 – 11/15/06 Stable disk-to-disk flows Tampa-Caltech: Stepping up to 10-to-10 and 8-to-8 1U Server-pairs 9 + 7 = 16 Gbps; then Solid overnight. Using One 10G link Measured during the bandwidth challenge: 17.77 Gbps 16 Gbps Measured by MonALISA during preparation ramp up

8. FDT – Fast Data Transport Data transfered in paralel from target disks to source disks, using a separate thread for each deviceFDT used with 3 TCP streams/threads in blocking mode Transfer rate Two systems connected in the wide area network (RTT ~ 200ms) with 10Gb/s. Time to reach the 1.7 Gb/s is ~ 20s. Sender :   2 CPUs Dual Core Intel  Woodcrest  @ 3.00  GHz,  4 GB RAM, 4 x 320 GB SATA Disks Receiver : 2 CPUs Dual Core  AMD Opteron       @  2.75 GHz,  4 GB RAM, 4 x 320 GB SATA Disks Disk activity in MBytes/s: CPU usage for the sender : Load of system and bottleneck in test was disk I/O ~200 MB/s (50 MB/s per disk)

9. Major Communities -OSG -CMS -ALICE -D0 -STAR -VRVS -LGC RUSSIA -SE Europe GRID -APAC Grid -UNAM Grid (Mx) -ITU -ABILENE -ULTRALIGHT -GLORIAD -LHC Net -RoEduNET -Enlightened VRVS ALICE MonALISA Monitoring System Alice web based repository: http://pcalimonitor.cern.ch:8889 ABILENE - - Interactive Client MonALISA based alert system: http://pcalimonitor.cern.ch:8889/xml.jsp VRVS/EVO OSG End-2-end monitoring of many Grids

10. MonALISA Monitoring System MonALISA Today Running 24 X 7 at 280 Sites • Collecting > 750,000 parameters in near real-time • Update rate of 25,000 parameter-updates per sec • Monitoring 40,000 computers > 100 WAN Links • Thousands of Grid jobs running concurrently • http://monalisa.caltech.edu/monalisa__Looking_Glass.htm Demonstrated at: • Telecom World • WSIS 2003 • SC 2004 • Internet2 2005 • TERENA 2005 • IGrid 2005 • SC 2005 • SC 2006 • CHEP 2006 • CENIC 2006 • Innovation Award for High-Performance Applications - - OSG

11. L-Store: File System Interface to Global Storage Cloud http://www.lstore.org/ • Provides a file system interface to (globally) distributed storage devices (“depots”) • Parallelism for high performance and reliability • Uses IBP (from UTenn) for data transfer & storage service • Generic, high performance, wide-area-capable storage virtualization service; transport plug-in support • Write: break file into blocks, upload blocks simultaneously to multiple depots (reverse for reads) • Multiple metadata servers increase performance & fault tolerance • L-Store supports beyond-RAID6-equivalent encoding of stored files for reliability and fault tolerance • Supports scalable streaming of Physics Analysis Data in/out of L-Store over the WAN

12. and VINCI Services (Virtual Intelligent Network Computing Infrastructure) Application Application End User Agent End User Agent System Evaluation & Optimization Authentication, Authorization, Accounting Scheduling; Dynamic Path Allocation Prediction Failure Detection Topology Discovery Control Path Provisioning Learning SNMP GMPLS MPLS OS MonALISA Monitoring Substrate http://monalisa.caltech.edu/monalisa__Service_Applications__Vinci.html

13. VINCI: Dynamic Path Allocation for Automated Dataset Transfer Detects errors & automatically recreates the path in less than the TCP session timeout (<1second) >MLcopy A/fileX B/path/ OS path available Configuring interfaces Starting Data Transfer Normal Routed Path Real time monitoring Regular IP path APPLICATION DATA MonALISA Services Monitor Control A B TL1 LISA AGENT Sets up • Network Interfaces • TCP stack • Kernel parameters • Routes LISA Agent Optical Switch Active light paths Flash animation: http://gae.caltech.edu/movies/ml_optical_path/ml_os.htm

14. Physicist at Tier3 using Root on GBytes of ntuples Loads Clarens Root plugin. Connects to Clarens. Sends analysis code (.C/.h files). Clarens creates Rootlet, passes it .C/.h files Rootlet runs analysis code on TBytes of ntuples, creating high statistics output data. Root at Tier3 receives and plots data “Rootlets” Root embedded in a Clarens server Tier3 Tier2 Analysis.C, Analysis.h    Clarens Plugin XML/RPC Root Tuples  Root Tuples GBytes  ~10 TBytes

15. “Higgs” Data Analysis using Rootletsat Supercomputing ’06 Bandwidth Challenge

16. Summary: UltraLight Network R&DGlobal Planning Services • UltraLight is a global Laboratory, uniquely positioned • Spans Tier-0, some Tier-1s, several Tier-2s, and Tier-3s • UltraLight (optical networks in general) moving towards a managed “control plane” • Light-paths will be allocated/scheduled to data-flow requests via policy based priorities, queues, and advanced reservations • Clear need to match “Network Resource Management” with “Storage Resource Management” • Available UltraLight Infrastructure / Services / Applications for Tier3s • High performance network infrastructure (http://ultralight.org/) • Fast Data Transport Tool – FDT (http://monalisa.cern.ch/FDT/) • MonALISA + VINCI (http://monalisa.caltech.edu/) • LStore (http://www.lstore.org/) • Rootlets • UltraLight working to develop (generic) systems solutions by • Researching and developing Global Planning Services • Using an end-to-end approach • devices, parameters, end-host services, WAN, high-level services, applications Services fabric level application level