1 / 13

CASTOR project status

CASTOR project status. CERN IT-PDP/DM October 1999. CASTOR. CASTOR stands for “CERN Advanced Storage Manager” Short term goal: handle NA48 and COMPASS data in a fully distributed environment Long term goal: handle LHC data. CASTOR objectives (1). High performance Good scalability

chi
Download Presentation

CASTOR project status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CASTOR project status CERN IT-PDP/DM October 1999

  2. CASTOR • CASTOR stands for “CERN Advanced Storage Manager” • Short term goal: handle NA48 and COMPASS data in a fully distributed environment • Long term goal: handle LHC data CASTOR/HEPIX’99@SLAC

  3. CASTOR objectives (1) • High performance • Good scalability • High modularity to be able to easily replace components and integrate commercial products • Provide HSM functionality as well as traditional tape staging • Focussed on HEP requirements CASTOR/HEPIX’99@SLAC

  4. CASTOR objectives (2) • Available on all Unix and NT platforms • Support for most SCSI tape drives and robotics • Easy to clone and deploy • Easy integration of new technologies: importance of a simple, modular and clean architecture • Backward compatible with the existing SHIFT software CASTOR/HEPIX’99@SLAC

  5. CASTOR layout: tape access TMS VDQM server NAME server STAGER RTCOPY TPDAEMON STAGE Client VOLUME allocator MOVER RFIOD RFIO Client MSGD DISK POOL CASTOR/HEPIX’99@SLAC

  6. Main components • Cthread and Cpool • to hide differences between thread implementations • CASTOR Database (Cdb) • developed to replace the stager catalog • its use could be extended to other components like the name server • Volume and Drive Queue Manager • central but replicated server to optimize load balancing and number of tape mounts • Ctape • rewritten to use VDQM and provide an API CASTOR/HEPIX’99@SLAC

  7. Remote Tape Copy • Calls Ctape API to mount and position tapes • Transfers data between disk server and tape drive • Uses large memory buffers to cache data while mounting tape • Overlays network and tape I/O using threads and circular buffers • Overlays network I/O from several input files if enough memory is available • Memory buffer per mover = 80 % of the tape server physical memory divided by the number of tape drives CASTOR/HEPIX’99@SLAC

  8. Basic Hierarchical Storage Manager (HSM) • Automatic tape volume allocation • Explicit migration/recall by user • Automatic migration by disk pool manager • A Name Server is being implemented • 3 Database products will be tested: Cdb, Raima and Oracle CASTOR/HEPIX’99@SLAC

  9. Name server • Implement an ufs-like hierarchical view of the name space: files and directories with permissions and access times • API client interface which provides standard Posix calls like mkdir, creat, chown, open, unlink, readdir • This metadata information is also stored as “user” labels together with the data on tape • It can also be exported according to the C21 standard for interchange with non CASTOR systems CASTOR/HEPIX’99@SLAC

  10. Volume Allocator • Determine the most appropriate tapes for storing files according to • file size • expected data transfer rate • drive access time • media cost... • Handle pool of tapes • private to an experiment • public pool • Other supported features: • allow file spanning over several volumes • minimize the number of tape volumes for a given file CASTOR/HEPIX’99@SLAC

  11. Current status • Cthread/Cpool: ready • Cdb: being rewritten for optimization • Rtcopy: Interface to client, network and tape API ready • Tape: client API ready, server being completed • VDQM: ready • Name server: being tested using Raima • Volume allocator: design phase • Stager: interfaced to Cdb, better robustness, API to be provided • RFIO upgrade: multi-threaded server, 64 bits support, thread safe client CASTOR/HEPIX’99@SLAC

  12. Deployment • All developments mentioned above will be gradually put in production over the autumn: • new stager catalog for selected experiments • tape servers • basic HSM functionality • modified stager • name server • tape volume allocator CASTOR/HEPIX’99@SLAC

  13. Increase functionality(Phase 2) • The planned developments are: • GUI and WEB interface to monitor and administer CASTOR • Enhanced HSM functionality: • Transparent migration • Intelligent disk space allocation • Classes of service • Automatic migration between media types • Quotas • Undelete and Repack functions • Import/Export • These developments must be prioritized and the design and coding would start in year 2000 • Collaboration with IN2P3 on these developments has started (RFIO 64 bits) • CDF experiment at Fermilab is interested CASTOR/HEPIX’99@SLAC

More Related