1 / 21

JRA1 - Middleware Development, Evolution and Integration

JRA1 - Middleware Development, Evolution and Integration. Morris Riedel (JUELICH) JRA1 Leader 1 st EMI Periodic Review Brussels, 22 June 2011. Outline. JRA1 in EMI Year 1 Highlights Achieved Technical Objectives Impacts on EMI 1 release New Functionality (key developments)

brady-boyer
Download Presentation

JRA1 - Middleware Development, Evolution and Integration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JRA1 - Middleware Development, Evolution and Integration Morris Riedel (JUELICH) JRA1 Leader 1st EMI Periodic Review Brussels, 22 June 2011

  2. Outline • JRA1 in EMI • Year 1 Highlights • Achieved Technical Objectives • Impacts on EMI 1 release • New Functionality (key developments) • Integration & Standardization Progress • Quality Control • Responding to DCI Evolutions • EMI Registry & Messaging • EMI Cloud Investigations • Lessons Learned 1st EMI Periodic Review

  3. JRA1 in EMI NA1 Collaborations, exploitation, sustainability Project Management NA1, NA2 Requirements NA1, JRA1 Software & Services Defines Process definition SA1 JRA1 Training NA2 SA2 NA2 In-Reach Dissemination & Out-Reach Certifies Implements Process monitoring JRA1 SA1 Release Candidate 1st EMI Periodic Review

  4. JRA1 Y1 Highlights • Achieved all Y1 technical objectives defined by PTB • 12/16 100% achieved, only 4 achieved with 50%-75% • Status in DJRA1.1.2; DJRA1.2.2; DJRA1.3.2; DJRA1.4.2 • Product developments are part of the EMI 1 release • Four different technologies in one integrated release • Established important agreements • EMI - Execution Services & Storage Accounting specification • Replace proprietary Grid Security Infrastructure (GSI) • Common security attributes & several messaging use cases • Increased open standard adoption • EMI 1 release with a broader set of supported standards 1st EMI Periodic Review

  5. Technical Objectives DNA1.3.1 (1) EMI All-Hands Meeting - Lund

  6. Technical Objectives DNA1.3.1 (2) • … and a lot of preparatory work for technical objectives due in year 2 have been started (e.g. agreements, detailed specifications, common APIs, etc.) 1st EMI Periodic Review

  7. KJRA1.1 Adopted Standards EMI EC Review - Brussels

  8. New Functionality: GLUE2 Adoption Enable effective and efficient operations of the DCIs like EGI and PRACE by improving the existing middleware services with new required functionality focusing on usability, manageability and service operations… [DoW], SO1.3, SO2.2 • Consistent information ecosystem in EMI 1 • Avoids adapter-based transformations & semantic loss across EMI components; easier operations in EGI • Information providers with static and dynamic information • Decision to use standard-based information model GLUE2 • LDAP and XML renderings • GLUE2 support in job and data management services • CREAM CE, ARC CE, UNICORE, FTS, LFC, dCache, DPM, and StoRM • Production support for GLUE1.3 kept where available 1st EMI Periodic Review

  9. New Functionality: file:// Access Enable effective and efficient operations of the DCIs like EGI and PRACE by improving the existing middleware services with new required functionality focusing on usability,manageability and service operations… [DoW], SO1.3, SO4.4 • Storage elements providing file:// access in EMI 1 • POSIX IO compliant access pattern • NFS clients are able to use EMI SEs in an easy manner • Support for NFS4.1/pNFS makes SEs industry competitors • dCache supports NFS4.1/pNFS (production) • DPM supports NFS4.1/pNFS (experimental) • StoRM supports this functionality via the corresponding underlying file system if available (e.g. GPFS) • To be released with next EMI update 1st EMI Periodic Review

  10. New Functionality: HTTP & WebDAV SEs Enable effective and efficient operations of the DCIs like EGI and PRACE by improving the existing middleware services with new required functionality focusing on usability,manageability and service operations… [DoW], SO1.3, SO4.4 • EMI 1 Storage elements support HTTP(S) (+ WebDAV) • HTTP(S) as most widely used standards in the Web & DCIs • Adoption of WebDAV makes EMI SEs improves the usability • Increased competitiveness with business data solutions • dCache offers HTTP(S) (and additionally WebDAV) in EMI 1 • DPM offers HTTP(S) in EMI 1 • StoRM offers support very soon (next release) • Complete adoptions with WebDAV across storage elements planned for EMI 2 1st EMI Periodic Review

  11. New Security Developments Enable effective and efficient operations of the DCIs like EGI and PRACE by improving the existing middleware services with new required functionality focusing on usability,manageability and service operations… [DoW], SO1.3, SO4.4 • VOMS 2.0 / VOMS-Admin • SAML-based access (VOMS-admin) • RESTful interface to obtain attributes • GSI dependency removal (kept for backward compatibility) • Good progress with VOMRS convergence • Argus 1.2 • CREAM CE and UNICORE works together Argus • Argus PDP handler prototype for ARC service container • Blacklisting already used by DPM, dCache and LFC 1st EMI Periodic Review

  12. Integration Activities Integrate emerging components into the broader EMI component ecosystem in order to avoid incompatibilities as well as to ensure that the components can be used together … [DoW], SO4.3 • JRA1 delivered a consolidated and streamlined set of services and components from ARC, gLite, UNICORE, and dCache as integrated release for SA1 • Compliant to harmonized EMI 1 production release criteria • Including re-factoring of existing components to comply with OS guidelines (EPEL, etc.) and project policies • One Globus toolkit version avoids incompatibilities among the different EMI components • OS Integration: SL5/64bit as reference platform in collaboration with SA1 release management 1st EMI Periodic Review

  13. Standard Adoption Progress Guide and validate the open standard adoption process in EMI and participate in standardization bodies in order to promote and drive standards that are relevant to the EMI project … [DoW], SO2.1, SO1.3 • Wide adoption of GLUE2 within EMI products • Whole ‘EMI information ecosystem’ based on GLUE2 • Replacement of legacy (proprietary) GSI • https instead of httpg for the SRM protocol as prototype • Implemented in dCache and VOMS 2.0 • Security delegation group formed to evaluate solutions • Agreement on common profiles • Agreement on SAML profile for common attributes • Agreement on XACML profile for attribute-based policies 1st EMI Periodic Review

  14. Standardization Pre-Studies Guide and validate the open standard adoption process in EMI and participate in standardization bodies in order to promote and drive standards that are relevant to the EMI project … [DoW], SO1.1, SO1.3 • Standardization pre-studies • Agreements in EMI project on the standardization path • Ensure that EMI speaks with one voice to the community • Job execution and management • Agreement on EMI Execution Service Specification (ES) • [http://cdsweb.cern.ch/record/1359908/files/EMI-ES-Specification_v1.0.odt ] • Storage accounting • Storage Accounting Record Specification (StAR) agreement • [http://cdsweb.cern.ch/record/1352472/files/StAR-EMI-tech-doc-final.doc ] 1st EMI Periodic Review

  15. Contributions to Standard Bodies Guide and validate the open standard adoption process in EMI and participate in standardization bodies in order to promote and drive standards that are relevant to the EMI project … [DoW] • Active contributions to OGF Working Groups • Grid Interoperation Now (GIN) CG • EMI chair position; EMI and international Grid interoperability • Production Grid Infrastructure (PGI) WG • EMI chair position; EMI input via EMI ES around job management • Grid Laboratory Uniform Environment 2 (GLUE2) WG • EMI chair position; EMI implements specs & pushes XML rendering • Usage Record (UR) WG • EMI chair position; EMI input via storage accounting StAR spec. • Storage Res. Man (SRM); Simple API for Grid App. (SAGA) • EMI has adoptions and provides feedback & production experience 1st EMI Periodic Review

  16. Quality Control Continuously improve the quality of the grid services by implementing standard Quality Control activities with particular focus on standard compliance and conformance tests, unit and functional tests… [DoW] • JRA1 has followed the SA2 Software Quality Assurance Process (SQAP) • Product teams took established policies into account such as packaging, documentation, change management, etc. • Validation collaboratively by JRA1 and SA1 QC teams • Focus in year 1 on harmonization on release management • JRA1 and SA2 is working on analysis of the metrics of the software engineering process in year 2 • E.g. metrics for Object Oriented Programs (Chidamber & Kemerer metrics suite) such as WMC, DIT, NOC, CBO, RFC 1st EMI Periodic Review

  17. EMI Registry and Messaging Follow and anticipate the needs of the growing infrastructure usage by investigating and adopting technologies to improve scalability, reliability and performance of the grid services… [DoW], SO2.1, SO.2.5 • Provide DCIs with a common EMI registry • Provides functionality to obtain information about the existence of the grid service from the grid services itself • Design is available and early implementation plan exists • [http://cdsweb.cern.ch/record/1359909/files/EMIRegistryDesign-v0.2.doc] • EMI take advantage of the messaging technologies • Early prototype for File Catalogues and SE synchronization • Accounting use case: transport URs via a messaging system • Internal guidelines for messaging & EMI products • [https://twiki.cern.ch/twiki/bin/view/EMI/EMIMessagingGuidelines ] 1st EMI Periodic Review

  18. EMI Cloud and Virtualization Increase the relevance and usability of the EMI grid middleware by actively managing user requirements in collaboration with infrastructure and community projects and initiatives… [DoW], SO2.3 • EMI Cloud Task Force established • Goals: Integration scenarios with off-the-shelf computing cloud systems to be able to execute grid jobs on those (scaling out to clouds) evaluated, but also beyond that • EMI cloud contexts have been researched and several documents are available (report, paper, etc.) • [http://cdsweb.cern.ch/record/1359910/files/EMIVirtCloudReport-v0.7.doc] • Standardization approaches observed (OCCI, CDMI, etc.) • Existing work is baseline for setting up an EMI cloud strategy that is in-line with the common DCI roadmap • [http://cdsweb.cern.ch/record/1277542/files/EMI-DNA1.4-1277542-EMI_Roadmap_DCI_Collaborations-v1.0.pdf ] 1st EMI Periodic Review

  19. Lessons Learned • Quality Control Improvements • Alignment of JRA1 QC and SA1 QC makes sense • More time for testing towards EMI 2 release required • Write more unit tests, functional tests, etc. • Increased set of functional and conformance tests as part of development required • Inter-product/-middleware work and strategies • Plan inter-product work and tests across middlewares/PTs • More standardization compliance and interoperability tests • Plan better long-term standardization strategies • Differentiate better between short-term implementation and long-term standardization activities 1st EMI Periodic Review

  20. Planned Major Year Two Activities • New Key Developments (out of DNA1.3.2) • EMI – ES standardization pre-study Implementations • EMI accounting records in compute and data products • Common libraries for data access and authentication • Argus integration in compute and cata products • NAGIOS probes for the majority of EMI services • Establish further important agreements • Common EMI delegation method • EMI parallel execution framework & EMI cloud strategies • Reduction of products and SLOCs • Increased standardization adoption 1st EMI Periodic Review

  21. Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 1st EMI Periodic Review

More Related