1 / 6

Online/Offline Data Transfer Discussions

Online/Offline Data Transfer Discussions. FC API. ADTDB. LFC. PIT->Castor. DIRAC @ PIT. ONLINE SYSTEM. DISET. RequestDB Svc. Request Database. Data Mover. Online Run Database. DISET. Online Storage. Transfer Agent. Replica Manager. CERN-IT. Castor Storage. File movement.

triveni-ishaan
Download Presentation

Online/Offline Data Transfer Discussions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Online/Offline Data Transfer Discussions

  2. FC API ADTDB LFC PIT->Castor DIRAC @ PIT ONLINE SYSTEM DISET RequestDB Svc Request Database Data Mover Online Run Database DISET Online Storage Transfer Agent Replica Manager CERN-IT Castor Storage File movement Request movement rfcp/lcg-utils

  3. Online ‘Removal’ DIRAC @ PIT ONLINE XML-RPC SERVER xml-rpc Replica Manager DISET? Transfer Agent CERN-IT DISET Online Removal Agent RequestDB Svc Request Database DISET AutoDataTransfer Database Request movement File removal

  4. What is currently working.. • Rfcp file to castor, file registered in AdtDB+LFC • File migration status is checked by OnlineRemovalAgent • Once migrated removal request sent to gateway DIRAC • Gateway DIRAC sends message to data mover

  5. Possible transfer methods • RFIO – rf* • Pros: • md5 support and returned on success • RFIO storage class already implemented • Cons: • Doesn’t go through SRM • Grid tools – lcg-cp • Pros: • Standard utilities used for file upload • Upload through SRM interface • Cons: • No md5 (G-U-C underneath) • SLC4/64bit NOR SLC4/32bit (currently) supported

  6. Possible integrity checks • Md5 • Can check file on castor disk pool against online md5 • If inconsistent • remove file and send message to RunDB (retried) • Can store md5 in file catalogue • Possible to use for future checking of data integrity • Castor checksum • Possible (someone tell me how) to get checksum from castor • If we can generate the same checksum online • Once file is migrated compare • If matches then file can be removed online • If not remove file and send message to RunDB (retried) • Using rfio we can do both • Using grid tools only the latter • During March test generate statistics on occurrences of checksum mismatches -> eventual strategy.

More Related