1 / 19

WP2/WP7 Demonstration

WP2/WP7 Demonstration. WP7 High Throughput Data transfers WP2/WP7 Replica Selection based on Network Cost Functions WP2 Replica Location Service. High Throughput Data Transfers. Richard Hughes-Jones Jules Wolfrat. NIKHEF. CERN. Demo Setup.

lou
Download Presentation

WP2/WP7 Demonstration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WP2/WP7 Demonstration WP7 High Throughput Data transfersWP2/WP7 Replica Selection based on Network Cost FunctionsWP2 Replica Location Service

  2. High Throughput Data Transfers Richard Hughes-Jones Jules Wolfrat

  3. NIKHEF CERN Demo Setup • We will show data transfers from Mass Storage system at CERN to Mass Storage system at NIKHEF/SARA • 2 systems at CERN, Geneva with datasets from experiment LHCb • 4 Linux systems at NIKHEF/SARA, Amsterdam to where data has to be transferred; each with a disk sub-system I/O bandwidth of ~70 MB/s • All systems have Gigabit Ethernet connectivity • Use GridFTP and Measure disk to disk performance GEANT SurfNet

  4. GridFTP GridFTP Demo Consists of: Data over TCP Streams Raid0 Disk Raid0 Disk Dante Monitoring Site Monitoring Node Monitoring

  5. European Topology: NRNs, Geant, Sites Sara & NIKHEF SURFnet SuperJANET4 CERN

  6. Some Measurements of Throughput CERN -SARA • Using the GÉANT Backup Link • 1 GByte file transfers • Standard TCP • Average Throughput 167 Mbit/s • Users see 5 - 50 Mbit/s! • High-Speed TCP • Average Throughput 345 Mbit/s • Scalable TCP • Average Throughput 340 Mbit/s

  7. WP7 High Throughput Achievements • Close Collaboration with Dante • “Low” layer QOS testing over GEANT • LBE • IP premium • iGrid 2002 and ER 2002 : UDP with LBE • Network performances evaluation • EU Review 2003 : application level transfer with real data between EDG sites • proof of concept

  8. Conclusions • More research on the TCP stacks and its implementation is needed • Continued the collaboration with Dante to: • Understand the behavior of GEANT backbone • Learn the benefits of QoS deployment • WP7 is taking the “Computer Science” research and knowledge of the TCP protocol & implementation and applying it to the network for real Grid users • Enabling Knowledge Transfer to sysadmins and end users • EDG release 1.4.x has configuration scripts for TCP parameters for SE and CE • Firewalls rules recommendations • Network tutorials for end users • Work with users – focus on 1 or 2 sites to try to get improvements

  9. WP2/WP7 Replica Selection based on Network Cost Functions Franck Bonnassieux (WP7) Kurt Stockinger (WP2)

  10. CERN RAL NIKHEF IN2P3 CNAF CERN 46,75 77,78 44,87 35,44 RAL 7,46 2,44 7,12 4,35 NIKHEF 11,13 3,25 11,86 2,66 IN2P3 5,03 10,38 6,24 7,08 CNAF 4,5 6,53 4,04 13,08 CERN RAL NIKHEF IN2P3 CNAF CERN RAL NIKHEF IN2P3 CNAF NetworkCost functionality getNetworkCost FileSize = 10 MB Results = time to transfer (sec.)

  11. NetworkCost R-GMA Globus MDS Archive Raw Distributed Data Collector PCP PingEr IPerf UDPmon GridFTP NetworkCost Architecture Processing Collect And Storage Measure

  12. NetworkCost model • The current cost model is designed for data intensive computing and especially large files transfers • The most relevant metric for that cost model is available throughput • Implementation • Iperf Measurements (current) • GridFTP Logs (future) • Other metrics (future) : UDP, RTT, Jitter, ... • Synchronisation (PCP)

  13. Replica Management Services VO Membership Service Replica Management Services Replica Manager Client Optimization Information Service Replica Metadata File Transfer: GridFTP … Replica Location Service RLS

  14. Testbed Sites & Replica Manager Commands edg-rm copyAndRegisterFile -l lfn:higgsCERN  LYON edg-rm listReplicas -l lfn:higgs edg-rm replicateFile -l lfn:higgs NIKHEF edg-rm listBestFile -l lfn:higgs  CERN edg-rm getAccessCost -l lfn:higgs CERN NIKHEF LYON edg-rm getBestFile -l lfn:higgs  CERN edg-rm deleteFile -l lfn:higgs LYON edg-rm listBestFile -l lfn:higgs  CERN

  15. WP2Replica Location Service Peter Kunszt WP2 – Data Management

  16. Replica Location Service RLS • Local Catalogs hold the actual name mappings • Remote Indices redirect inquiries to LRCs actually having the file • LRCs are configured to send index updates to any number of RLIs • Indexes are Bloom Filters

  17. RLS Demo at SC2002

  18. RLS Demo Topology Today CERN lxshare0344.cern.ch Glasgow grid03.ph.gla.ac.uk California dc-n4.isi.edu Melbourne wombat.unimelb.edu.au Replica Location Index CERN lxshare0342.cern.ch Glasgow grid01.ph.gla.ac.uk California dc-n2.isi.edu Melbourne koala.unimelb.edu.au LocalReplicaCatalog

  19. SUMMARY • Replica Optimization • WP7 Network cost functions are integrated into the Replica Management functionality providing an essential functionality that was missing up to now. • This gives us the necessary framework to start work on high-level optimization algorithms. • Replica Location Service • Scalable distributed catalog as a much-needed replacement for the current Replica Catalog. • Addresses all issues brought up by the experiments. Tests have been conducted with very large catalogs • The lookup time for an entry is independent of the number of catalog. Tested for up to 108 entries. • The catalog withstands simultaneous user queries of over 1000 queries or inserts per second.

More Related