1 / 14

Proposal for a Dynamic Megatable

Proposal for a Dynamic Megatable. Flavia Donno CERN/IT. Tier-2 Sites & Capacities. A little difficult to determine exact resources installed in the Tier2s;  must be improved. (Ian Bird, March LCG OB).

pelham
Download Presentation

Proposal for a Dynamic Megatable

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proposal for a Dynamic Megatable Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 9 July 2008

  2. Tier-2 Sites & Capacities • A little difficult to determine exact resources installed in the Tier2s;must be improved (Ian Bird, March LCG OB) • But is provided < pledged because of availability or lack of use? need better reporting • M.C. Vetterli GDB presentation

  3. The requirements • The goal • Provide the management with information about installed capacity (per VO). • Provide the management with information about resource usage (per VO). • This info can also be used by VO operations people in order to "monitor" their usage of the resources • Main focus • Storage resources • Computing resources • Dynamic view • Information should be retrieved as dynamically as possible in order to be reliable • The information will be made available through the WLCG accounting system • Use the information system as source WLCG Grid Deployment Board, CERN 9 July 2008

  4. The documentation • The technical specifications are available in the CCRC twiki in the SSWG section: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCommonComputingReadinessChallenges#Storage_Solution_Working_Group_S • Storage only: • The documentation available reports on the conclusions reached during focused meetings with developers and information providers • Specific solutions found to cover dCache internal specialized buffers and avoid double counting • The document has been agreed by storage developers, storage information providers, data management developers, … WLCG Grid Deployment Board, CERN 9 July 2008

  5. The Storage Resources • Current accounting harvests data from BDII • … but many errors in data. • combination of Glue1.2 and 1.3 • Need to improve information providers and configuration. • Existing Reports • Current status (last hour) • Monthly Report (like manual Tier1 • reports). WLCG Grid Deployment Board, CERN 9 July 2008

  6. The Storage Resources • Use GlueSA class • It describes a Storage Area: logical view of a portion of space (can include disks and tapes) • Online resources only for the moment • GlueSAs do not overlap. • Shared spaces can be represented with one GlueSA with multiple GlueSAAccessControlBaseRule • A GlueSA can be published for unreserved space as well • In this case Total=Reserved Size (see later) • GlueSA can have one or more VOInfo associated objects • VOInfoTag is optional • The VOInfo object can be used to publish the path associated to a VO using the associated GlueSA (in case of shared SAs) WLCG Grid Deployment Board, CERN 9 July 2008

  7. The Storage Resources • GlueSA Sizes • GlueSAReservedOnlineSize • Space physically allocated to a VO (or a set of VOs) – Installed capacity • GlueSATotalOnlineSize (in GB = 109 bytes) • Total Online Space available at a given moment (it does not account for broken disk servers, draining pools, etc.) • GlueSAUsedOnlineSize (in GB = 109 bytes) • Size occupied by files that are not candidates for garbage collection – Resources usage • For Castor this is equal to GlueSATotalOnlineSize for T1D0 (they differ for T0D1) • GlueSAFreeOnlineSize (in GB = 109 bytes) • Free = Total - Used • GlueSACapability • Key/value pair to publish agreed characteristics of Storage Area, i.e. access=read, network=WAN-LAN, etc. • Experiments can use this field to select dCache pools WLCG Grid Deployment Board, CERN 9 July 2008

  8. Storage: Status and plans • Dynamic information providers for CASTOR by J. Jensen • Corrections needed to comply with agreement • Packaging and distribution effort through the CASTOR CVS • First installation foreseen at CERN • Availability: end of July 2008? • Dynamic information providers for dCache available by R. Trompert • Need to verify with dCache developers that pinned files usage info is available, and other details • 2-3 weeks to implement the proposal • The changes will be reflected also in the new official dCache information providers by P. Millar WLCG Grid Deployment Board, CERN 9 July 2008

  9. Storage: Status and plans • Dynamic information providers for DPM by Michel Jouvin • The proposal has been already implemented • Installed at Edinburgh – Testing phase started • Information already available for StoRM by developers • Probable minor additions for VOInfoPath • Need to work with OSG and NorduGrid WLCG Grid Deployment Board, CERN 9 July 2008

  10. The Computing Resources Pledges expressed in terms of KSI2000 per federation in current Megatable Use availableinformation published in BDII about installed capacity Red=deprecated 10 WLCG Grid Deployment Board, CERN 9 July 2008

  11. The Computing Resources • Ongoing work to make published information about installed capacity more reliable (see presentation of Steve Traylen at GDB) • Current tools publish always 1 GlueCluster and 1 GlueSubCluster when a GlueCE is published (problematic CPU counting) • Difference between SubCluster PhysicalCPUs and GlueCETotalCPUs • Is Subcluster PhysicalCPUs coherent with SubCluster Host BenchmarkSI00? • When SubCluster PhysicalCPUs=0 consider GlueCETotalCPUs: Number of slots per queue. Compatible with Subcluster Host BenchmarkSI00? Is this number correct (extracted from batch system)? • VOView TotalCPUs deprecated. Difficult to guess VO shares. WLCG Grid Deployment Board, CERN 9 July 2008

  12. Computing Resources Example • Need to interpret results • Refinement and cleaning (discrepancy between SubClusters and GlueCE, CPUs vs. cores, HEP only, etc.) • Improve information providers (Glue 2.0 helps) Double-counted Resources WLCG Grid Deployment Board, CERN 9 July 2008

  13. Computing: Status and plans • Computing resources • First procedures in place. Looking for better technologies to make results available. • We are currently in the process of understanding and refining information • Work with sites • Check that needs are covered by Glue 2.0 • Help automate the publishing process • Work with OSG and NorduGrid • Provide a working prototype for APEL by the end of August 2008 • Give feedback to GDB and MB • Help producing monthly reports WLCG Grid Deployment Board, CERN 9 July 2008

  14. Thank You WLCG Grid Deployment Board, CERN 9 July 2008

More Related