1 / 1

Cluster distributed dynamic storage

Cluster distributed dynamic storage. BACKGROUND Manchester University 1000 nodes new cluster (2x1000 processors) 2x250 GB disks equivalent to ~400 TB of available disk space excluding the space reserved for OS and scratch directories. There is no tape storage behind.

elmo-barnett
Download Presentation

Cluster distributed dynamic storage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cluster distributed dynamic storage BACKGROUND • Manchester University 1000 nodes new cluster (2x1000 processors) • 2x250 GB disks equivalent to ~400 TB of available disk space excluding the space reserved for OS and scratch directories. • There is no tape storage behind. • Cluster internal connectivity is 1 Gb/s and is connected to the external world by the 2.5 Gb/s university production network. It will be replaced by a 10Gb/s dedicated link. • High rate data transfers from Tier1 is possible possible. 350 Mb/s achieved on the university production network. • The aim is to use the space as a cache. APPLICATIONS • Applications will access the data through /pnfs or gsidap doors • Gsidcap doors might be turned off in the future CONFIGURATION • dcache has been chosen as software • There will be 2 types of nodes: external and internal • External nodes will be front end nodes with an active gridftp door for transfer from the outside world. They’ll have a dedicated cpu to handle the transfers. • Internal will accessible only from other nodes • On each node: • 2 dcache partitions 1 on each disk • 2 pools, 1 for each partition • /pnfs mounted • Gsidcapdoor active • No raid arrays will be created it spoils the independence of the discs. In this way • Disk 2 can be assigned to a particular VO • Disk 1 can be used for resilience. • Disk 1 can be wiped out during installation if needed • Disk 2 can be left untouched. • Data will have 2 copies on the system using resilience. OTHER SOLUTIONS • Solution based on http technology is being developed in Manchester: • Uses HTTP or HTTPS for data transfer • Access control via XML policies and X.509 or VOMS certificates • File location using Hypertext Cache Protocol and multicast queries • Avoids need for central database, by enforcing atomic file operations and using status of files on disk to respond to queries • Experiments using root have asked to support Xrootd as a possible alternative solution • Plain root APIs to access dcache are also being evaluated. This turns out useful in the final stages of analysis when the experiment framework becomes irrelevant. A. Forti A. McNab

More Related