1 / 12

A Test Framework for Movement of MC Data using SAM & SRM Interface

A Test Framework for Movement of MC Data using SAM & SRM Interface. 2 nd March, 2008 CDF Offline Meeting. Overview. Test Framework Composition of files Monitoring Parameters Time Plot Transfer Rate Cumulative Data Plot Query Map Conclusion & Next Step Wish List !.

ivi
Download Presentation

A Test Framework for Movement of MC Data using SAM & SRM Interface

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Test Framework for Movement of MC Data using SAM & SRM Interface 2nd March, 2008 CDF Offline Meeting

  2. Overview • Test Framework • Composition of files • Monitoring Parameters • Time Plot • Transfer Rate • Cumulative Data Plot • Query Map • Conclusion & Next Step • Wish List !

  3. Data Movement: Test Framework UCSD SRM V2 srm://t2data2.t2.ucsd.edu:8443 srmcp FNAL SRM V2 srm://cmssrm.fnal.gov:8443 srmcp Data from SAM, dccp scp cdfsam15 fcdflnx1

  4. Composition of Files A SAM data file is assosciated with its meta data file.

  5. Composition of Files: File Size Meta Data File Size in Bytes Data File Size in Mega Bytes File Size No. of Files No. of Files • Meta data files are of the order of few Kilo bytes. • Data files are the order of several Mega Bytes.

  6. Monitoring Parameters: Transfer Time Meta Data Transfer Time in Sec Data Transfer Time in Sec Time Taken No. of Files No. of Files • Meta data files (few Kilo bytes) take around 14 seconds. • Data files takes (several Mega bytes) around 34 seconds.

  7. Monitoring Parameters: Evolution of files with time Meta Data File Size in Bytes Data File Size in Mega Bytes File Size Time Time • A dip in the right hand side plot (18:32 – 19:02) is being balanced by copying meta data files during this period.

  8. Monitoring Parameters: Transfer Rate Meta Data Tr. Rate in Bytes/sec Data File Tr. Rate in MBs/sec Transfer Rate No. of Files No. of Files • Transfer rate for larger size files is much higher than smaller size files.

  9. Monitoring Parameters: Cumulative Flow of Data File Size Time • More than ~30 Giga Bytes/hr of data has been transferred. • More test is needed to confirm a robust transfer rate.

  10. Monitoring Parameters: Query Map • Not easy to draw any conclusions from the above plot. • But, it will be helpful to display quality data for multiple sites versus time. Each box represents a fixed duration worth of transfers.

  11. Conclusion & Next Step • Conclusion: • Larger the file size, higher the efficiency of the framework. • Next Steps • Different sizes of files will be used • For files of other than above different sizes, joining files with “tar” is one option. But, it is too much cpu consuming. • In the next offline meeting, we will show plots on monitoring parameters for trans Atlantic movement of data.

  12. Wish List ! • Addition of more SRM managed storage element sites for studying the performance of the framework. • Compatibility of SAM with SRM version 2 • Present version of SRM in SAM is 1 • Latest version of SRM is 2. SRM version 2 is not backward compatible with SRM version 1. • Present version of SAM needs some development such that it would be compatible with SRM version 2 • A rough idea of the time frame ( under which the above issues will be resolved) will be helpful in speedy development of this work !

More Related