1 / 12

A Test Framework for Movement of MC Data using SAM & SRM Interface

A Test Framework for Movement of MC Data using SAM & SRM Interface. Manoj K. Jha (INFN- Bologna) Antonio Cuomo, Donatella Lucchesi, Gabriele Compostella, Simone Pagan (INFN – Padova) Doug Benjamin (Duke University) Robert Illingworth (Fermilab) 8 th April, 2008 CDF Offline Meeting.

sitara
Download Presentation

A Test Framework for Movement of MC Data using SAM & SRM Interface

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Test Framework for Movement of MC Data using SAM & SRM Interface Manoj K. Jha (INFN- Bologna) Antonio Cuomo, Donatella Lucchesi, Gabriele Compostella, Simone Pagan (INFN – Padova) Doug Benjamin (Duke University) Robert Illingworth (Fermilab) 8th April, 2008 CDF Offline Meeting

  2. Overview • Test Framework • Composition of files • Monitoring Parameters • Time Plot • Transfer Rate • Cumulative Data Plot • Conclusion & Next Step • Issues

  3. Data Movement: Test Framework UCSD SRM V2 srm://t2data2.t2.ucsd.edu:8443 srmcp FNAL SRM V2 srm://cmssrm.fnal.gov:8443 srmcp We are using WAN for transfer of files using srmcp. The no. of channels is 50 at UCSD SRM end. cdfsam15 A process is running on “cdfsam15” which create dummy files of random sizes. The random number generator is being biased for creating files of larger sizes (> 1 GB).

  4. Composition of Files The dummy files generated in sizes from 1 MB to 4GB. Number in the pie chart indicates the number of files generated corresponding to that category from 3rd to 8th April, 2008.

  5. Monitoring Parameters: Transfer Time • Initializations time of SRM is roughly of the order of 10 sec.

  6. Monitoring Parameters: Transfer Rate B C D • Transfer rate increases with increase in file size. • Maximum achievable transfer rate is around 60 MB/sec • User analysis jobs will fall in category B while MC jobs in C & D

  7. Monitoring Parameters: Transfer Time and Rate Comparison of transfer time for diff. categories of file. Comparison of transfer rate for diff. categories of file.

  8. Monitoring Parameters: Cumulative Flow of Data/hr File Size Time • Transfer rate of 25 GB/hr is achievable using WAN. • Transfer rate of 2 TB/hr is achievable using “FermiLight Path Circuits”. For more information, please visit • http://indico.fnal.gov/getFile.py/access?contribId=8&sessionId=19&resId=0&materialId=0&confId=1037

  9. Monitoring Parameters: Cumulative flow of data/day • Around 600 GB/day of data can be transferred using WAN. • For data transfer of higher rate, we would have to consider other options like “FermiLight Path Circuits”.

  10. Conclusion & Next Step • Conclusion: • SRM initialization time is of the order of 10 sec. • Date transfer rate increases with file size. • Maximum transfer rate achievable is around 60 MB/sec • Transfer rate of 25 GB/hr and 600GB/day is achievable using WAN. • Next Steps • Study the monitoring parameters for • Trans Atlantic movement of data. • Data movement between worker nodes and SRM • As a function of various number of requests arriving at SRM • Write a CDFNote explaining the implementation of framewrok within the CAF

  11. Issues ! • Authentication: • Presently, the user’s proxy is being generated on robotic head node. Users cannot take the proxy outside from the head node. What should be the mechanism for accessing the files on SRM, since users cannot access his/her proxy credentials ? • Addition of more SRM managed storage element sites will help us in speedy development of this work. • Compatibility of SAM with SRM version 2 • Present version of SRM in SAM is 1 • Latest version of SRM is 2. SRM version 2 is not backward compatible with SRM version 1. • Present version of SAM needs some development such that it would be compatible with SRM version 2

  12. Acknowledgement ! We would like to thank Jon Bakken (from FNAL), Frank Wuerthwein and Abhishek Singh Rana (from UCSD) for providing resources on SRMs !

More Related