1 / 22

Data Services and Solutions Part 1: Data in the Grid

Data Services and Solutions Part 1: Data in the Grid. Vincenzo Spinoso vincenzo.spinoso@egi.eu EGI.eu/INFN. Outline. Categorisation of data services in EGI State-of-the-art in the grid data services area: status and future plans Use cases and technical details Plans and next. Components.

mmann
Download Presentation

Data Services and Solutions Part 1: Data in the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Services and SolutionsPart 1: Data in the Grid Vincenzo Spinoso vincenzo.spinoso@egi.eu EGI.eu/INFN

  2. Outline • Categorisation of data services in EGI • State-of-the-art in the grid data services area: status and future plans • Use cases and technical details • Plans and next ....

  3. Components • Data management is performed by interoperable components • Different components address different needs • Storage management at site level • Transfer between sites • Security • Catalogue, metadata ....

  4. Storage endpoints How data are managedat site level? ....

  5. Storage endpoints A uniquenamespaceisprovided to the client Authentication and encryptionguarantee confidentiality and integrity Severalprotocolsare supportedfor file access and transfer Distribute data acrossseveral disk serversguaranteesscalabilityat site level Iftapes are provided, access to tape istransparent ....

  6. Storage endpoints StoRM DPM Lustre or GPFS ....

  7. Access, transfers Whataboutinteroperability, access, transfers? ....

  8. Access, transfers • Applications and users can interact with the endpoints using different protocols • SRM offers storage management • disk/tape transparent management • interface between different transfer protocols • standard interface • GridFTP offers advanced data transfer • Parallel streams • Fault tolerance • Security (authorization, encryption) • Optimization «Storage element» WebDAV NFS/pNFS SRM GridFTP Abstractionlayer StoRM DPM ....

  9. Access, transfers • Applications and users can interact with the endpoints using different protocols • WebDAV offers a «web-based network file system» • Widely supported by many OSes • Standard (IETF) • NFS4.1 provides «local access» (fast, POSIX) «Storage element» WebDAV NFS/pNFS SRM GridFTP Abstractionlayer StoRM DPM ....

  10. Access, transfers WebDAV NFS/pNFS WebDAV NFS/pNFS SRM GridFTP SRM GridFTP Abstractionlayer Abstractionlayer DPM ....

  11. Data transfer scheduling Can transfers be scheduled? ....

  12. Data transfer scheduling • schedule continuoussustained data transferacross multiple endpoints • prioritize inter-VO and intra-VO file transfers • Manydifferent clients availabletowardsseveralprotocols (SRM, GridFTP, webdav… ) • Useful in the VO management context to control data transfers ....

  13. Catalogue Where are myfiles? lfn:grid/20150407/store/data/run1312 ....

  14. Catalogue LFC • hierarchical view of files to users, with a UNIX-like client interface • Logical File Name (LFN) to Storage URL (SURL) mappings • authorization on namespace EXAMPLE: lfn:grid/20150407/store/data/run1312 srm://storm-se-01.ba.infn.it:8444/srm/managerv2?SFN=//cms/store/group ....

  15. EGI «whole picture» • Reallycomplexinfrastructurebased on elementary «bricks» • each VO choosesits «recipe» of components • mature and stable • integration in a unified release controlsstability of the «off-line» machinery • operationscontrol stability of the «on-line» machinery ....

  16. What is next… Storage Management overview

  17. Dynamic Federations (DynaFeds) • A set of components that can aggregate on-the-fly storage and metadata farms exposing standard protocols, supporting redirections and WAN data access: • Directories are «merged» so thatfiles in the same directory appear inside the same directory evenifthey come from differentsites • Browse and access a huge repository made of many sites without requiring a static index • No “registration”, no maintenance of catalogues • Redirect intelligently clients asking for replica • Automatically detects and avoid sites that go offline • Accommodates client-geography-based redirection choice • stable demo testbed, using HTTP/DAV http://federation.desy.de

  18. Dynamic Federations (DynaFeds) /voname/docs/file1 /voname/docs/file2 /voname/docs/file3 /voname/software /voname/pub … Aggregation/Abstraction /voname/docs/file1 /voname/docs/file2 /voname/docs/file2 /voname/docs/file3

  19. Dynamic Federations (DynaFeds)

  20. Globus Online • provides robust and easy to use file transfer capabilities • Web interface • Transfer management • Performance monitoring • Retries after failures, autorecover when possible • It’s a service, hosted at www.globusonline.eu (US) • But the files that the service moves among EGI sites DO NOT LEAVE Europe • GridFTP «3rd party transfer» is used • Files copied directly between the EGI endpoint ....

  21. iRODS • Provides high level abstraction layer on top of storage resources • Users focus on their data, not on where they are on the data grid • Provides native metadata catalogue • Multiple authentication plugins (password, PAM, GSI… ) • Multiple access protocols (POSIX, S3, RADOS… ) • Rule-oriented approach: «policies» can be easily implemented as data management tasks • Ongoing integration in the EGI infrastructure Storage Management overview

  22. References EGI http://www.egi.euhttps://wiki.egi.eu/wiki/Main_Page dCache http://www.dcache.org/ DPM/LFC https://svnweb.cern.ch/trac/lcgdm FTS http://fts3-service.web.cern.ch/ FTS Dashboardhttp://dashb-fts-transfers.cern.ch/ui/ Dynamic Federations http://indico.cern.ch/event/287233/session/6/contribution/21/material/slides/ iRODS http://irods.org/ Globus Online Cookbook https://wiki.egi.eu/wiki/Globus_Online_cookbook_for_EGI_VOs ....

More Related