1 / 11

New KanGA Export Scheme

New KanGA Export Scheme. Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Data Distribution Session 13 th December 2000. ftp TNG Selecting events for export Running the export procedure. Optimising ftp throughput.

jamal
Download Presentation

New KanGA Export Scheme

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New KanGA Export Scheme Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Data Distribution Session 13th December 2000 Tim Adye

  2. ftp TNG • Selecting events for export • Running the export procedure Tim Adye

  3. Optimising ftp throughput • ftp / scp / rsync(syncslac) are not optimised for bulk transfers over the WAN • Optimisations possible • Use multiple-streams • Separate processes/threads copy different parts of the data • Adjust TCP/IP parameters • Large improvements from increasing TCP window size • Tools: bbftp, sfcp, grid-ftp (new version of gsi-ftp) Tim Adye

  4. New “ftp” tools • bbftp[Gilles Farrache, IN2P3] • Retry mechanism • On-the-fly compression • Now with ssh authentication option [TJA] • Greatly simplifies automatic operation • Allows access to bbrdist@datamove3 • sfcp[Artem Trunov and Andy Hanushevsky, SLAC] • Better control of transfer parameters • Simple setup, but Solaris only • No compression, retry, etc. Tim Adye

  5. New “ftp” tools • grid-ftp (gsi-ftp) [GLOBUS] • GLOBUS authentication • Not yet publicly available Tim Adye

  6. Transfer performance • SLAC->RAL Kanga exports • syncslac / rsync gave ~300 kbits/s per job • Scales with number of parallel copies • bbftp with 2 streams gives 3-6 Mbits/s • Nearing 10 Mbits/s Janet-ESnet peering • sfcp with 2 streams gives 600 kbits/s • Scales with number of streams • Need to understand sfcp/bbftp differences • RAL->SLAC SP imports • bbftp gives 10-20 Mbits/s • Benefits from x3 compression of Objy files Tim Adye

  7. Caveat • The optimal bbftp/sfcp choice/settings will vary from site to site. • Rome->RAL: bbftp and sfcp are similar • Eg. SLAC<->INFN gain with larger number of streams • Need systematic tests to give better advice Tim Adye

  8. Kanga export procedures • Kanga export controlled by skimData database • David Kirkby, Alessandra Forti, Alvise Dorego,TJA • skimSqlMirror makes local copy of skimData db • skimSqlSelectselects data for import • skimImportcopies the selected files • skimBackupbackup/archive to tape • skimDeleteremove files from local disk • Documentation under construction at • http://www.pd.infn.it/~dorigoa/BFROOT/KDDIBT.html • BaBar -> Computing -> Data Dist -> skimData remote Tim Adye

  9. skimSqlSelect • Selects data for import by setting a flag in skimData database • Run as part of skimData mirroring to automatically select new data • Use skimData options to select data, eg. • skimSqlSelect -–skimData ‘-sAllEventsKanga:-S>=Skim880f’ Tim Adye

  10. skimImport • Copies files that are marked for import in the skimData database • Can use different ftp programs • Currently implements bbftp and sfcp • Plan to add rsync which can check existing files • Perl OO interface can be used by other scripts • Eg. Automatic SP export procedure • Can distribute files across different filesystems. • Eg. • skimImport --ftp-type=bbftp –config=ral.cfg Tim Adye

  11. Deployment Need guinea pigs to try this all out Tim Adye

More Related