1 / 17

GLUE 2.0 Profile for EGI

GLUE 2.0 Profile for EGI. Stephen Burke egi.eu EGI TCB November 6 th 2012. Overview. Why we needed a profile document Classification of use cases Classification of importance of publication Validation Installed capacity Document structure Review process Implementation. Introduction.

jamescline
Download Presentation

GLUE 2.0 Profile for EGI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GLUE 2.0 Profile for EGI Stephen Burke egi.eu EGI TCB November 6th 2012

  2. Overview • Why we needed a profile document • Classification of use cases • Classification of importance of publication • Validation • Installed capacity • Document structure • Review process • Implementation GLUE 2.0 Profile - EGI TCB 14

  3. Introduction • The GLUE 2.0 schema is intentionally very flexible • Many ways to use it, not necessarily interoperable • Several SRM implementations • Need a profile to specify how it should be used in EGI • Currently “in EGI” means “in the BDII” – may change • Detailed semantics of each attribute, what should and should not be published • Monitoring tools should enforce the usage • First public draft is available • Some comments received • Will need detailed review, especially by middleware developers • Likely to need updates in the light of experience GLUE 2.0 Profile - EGI TCB 14

  4. Use cases • The schema can potentially satisfy many use cases • Various implications: importance, need for accuracy, update rate, latency, caching ... • The BDII currently has all information, but we may want to have separate systems for different kinds of information • EMIR? • The profile document classifies each attribute as potentially useful for one or more of 5 categories of use case • The fact that something can be used in a particular way doesn’t mean that it will be! GLUE 2.0 Profile - EGI TCB 14

  5. Use case categories • Service Discovery (SD): used as selection parameters in queries to find services, or service attributes returned by such queries • Service Selection (SS): dynamic information used to choose a particular service, e.g. number of queued jobs • Monitoring (M): information useful to monitor the overall state of the Grid, e.g. as used in gstat • Oversight (O): high-level management information, e.g. installed capacity and service versions • Diagnostic (D): information which may help diagnose problems with the services or in the info system itself GLUE 2.0 Profile - EGI TCB 14

  6. Importance • The schema document just classifies attributes as mandatory or optional • For EGI we need more detailed guidance as to which attributes need to be published • Again these are classified in 5 categories GLUE 2.0 Profile - EGI TCB 14

  7. Importance categories • Mandatory (M): information which must be published. Includes all attributes which are mandatory in the schema, but adds some which are important for EGI. • Recommended (R): information which should be published unless there is a good reason, e.g. technically impossible or not meaningful in a particular case. • Desirable (D): information which is likely to be useful, and should be published if reasonably practical, but which may be omitted. • Optional (O): information which is truly optional. Typically these attributes have no use in the existing EGI infrastructure, or are only useful in specialised cases. • Undesirable (U): information which may be damaging in the EGI context, e.g. due to large data volumes or exposure of sensitive information. GLUE 2.0 Profile - EGI TCB 14

  8. Validation • Validation of the published information has traditionally been a weak point • LDAP has few internal constraints, and gives cryptic errors for the things it does check • The definition of which values are allowed is fairly poor • Many things just defined by the code or common practice • The gstat tool has some checks, but with limited coverage and inflexible for updating the tests • The current information system is full of mistakes! • Circular problem: hard to persuade people to publish unused attributes correctly, hard to persuade people to use attributes if there are many errors • For GLUE 2 we should try to do better GLUE 2.0 Profile - EGI TCB 14

  9. Validation categories • FATAL: errors which invalidate the structure of the information, e.g. unique IDs which are not unique. • ERROR: values which are definitely incorrect. • WARNING: values which are likely to be incorrect, or which are valid within the schema in general but invalid in the EGI context, e.g. very large numbers of jobs. • INFO: values which are technically correct but may be wrong, e.g. strings in the wrong case, locations at (0,0), typos, values in the wrong units etc. It is therefore desirable to have heuristics to try to identify such mistakes – e.g. collect a list of commonly-published batch system names and flag names which are not in the list. GLUE 2.0 Profile - EGI TCB 14

  10. Validation tools • We need tools which validate the information in different situations: • Interoperability tests in middleware development • Service-specific tests, e.g. for CE and SE • During middleware acceptance tests • Inside the BDII before publication • Checking the entire published information in a top-level BDII • May be able to have a common set of tests which can be plugged in to different tools? • Need to report errors to the right place • Middleware bugs to developers, not system managers! GLUE 2.0 Profile - EGI TCB 14

  11. Validation tests • The profile document has many suggested validation tests • Intended as guidance and not fixed rules • Some things may be too hard to implement • New/different tests always welcome • INFO-level tests will probably have many false positives • Need to be able to switch them off and/or update them GLUE 2.0 Profile - EGI TCB 14

  12. Installed capacity • Useful to know the total installed capacity (CPU power, disk and tape sizes) and the sharing between VOs • WLCG has a document for GLUE 1: https://twiki.cern.ch/twiki/pub/LCG/WLCGCommonComputingReadinessChallenges/WLCG_GlueSchemaUsage-1.8.pdf • Some things are difficult in GLUE 1, should be easier in GLUE 2 • Profile document has guidance • Computing power straightforward • Logical CPUs, HEPSPEC-06, VO shares • Storage more complex • Including/excluding unavailable disk servers • Treatment of cached files, parity/spare disks, T1D0, ... • Shared space vs space tokens • Logical vs physical view, double counting • May need to iterate with developers GLUE 2.0 Profile - EGI TCB 14

  13. Document structure • Introduction explaining concepts • Detailed section for each schema class • Doesn’t repeat information from schema document unless needed for clarity • Table giving information for every attribute, plus text notes if necessary • E.g. for the Contact class: GLUE 2.0 Profile - EGI TCB 14

  14. Review process • Long, technical document, needs careful review • Especially by developers • May take some time to converge • May well need updates after experience • Document is versioned • Compliance with a given version should be published GLUE 2.0 Profile - EGI TCB 14

  15. Timetable • Internal EGI review since September • Some comments received • Will start updating the document, and publish responses to comments • Version on docDB will stay the same for now • External review • Especially EMI • Comments by the December 7th • “Final” document by the end of the year • Version 1.0 • Likely to evolve with experience GLUE 2.0 Profile - EGI TCB 14

  16. Implementation • Difficulty will vary – many things may be compliant already • Developers can check compliance while reviewing the document • Need to start validation – if only by hand • Most EGI sites now publishing in GLUE 2 • Submit bugs if problems are found GLUE 2.0 Profile - EGI TCB 14

  17. References • Draft profile document • https://documents.egi.eu/public/ShowDocument?docid=1324 • GLUE 2.0 specification • http://www.ogf.org/documents/GFD.147.pdf • LDAP rendering specification (draft) • http://forge.ogf.org/sf/go/doc15518?nav=1 • WLCG installed capacity document • https://twiki.cern.ch/twiki/pub/LCG/WLCGCommonComputingReadinessChallenges/WLCG_GlueSchemaUsage-1.8.pdf GLUE 2.0 Profile - EGI TCB 14

More Related