1 / 23

EGI Middleware Function

EGI Middleware Function. Lud ě k Matyska and Mirco Mazzucato Based on input from MW TF members

aletha
Download Presentation

EGI Middleware Function

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EGI Middleware Function Luděk Matyska and Mirco Mazzucato Based on input from MW TF members Alistair Dunlop (UK), Achim Streit (DE), Farid Ould-Saada (NO), Mirco Mazzucato (IT), Ignacio M. Llorente (ES), Uwe Schwiegelshohn (DE), Ludek Matyska (CZ), Ian Bird (CERN) , Christoph Witzig (CH), Sergio Andreozzi (IT) Ludek.Matyska@cesnet.cz

  2. Disclaimer • These slides have been drafted based on the current version of the EGI Blueprint proposal (v1.2, 25 June 2008). • The purpose of this blueprint is to assess a possible model for the future sustainable grid infrastructure in Europe. • The aim of these slides is to support the discussions in the EGI Geneva Workshop (30 June 2008). • The EGI_DS project will modify this draft blueprint on the basis of feedback provided. www.eu-egi.org

  3. Requirements Context • A vision of large high quality pan-European distributed compute and data infrastructure that • Has no internal barriers • Minimizes operational load • Provides guaranteed services • Is dependable, users can rely on its long term existence • Therefore must run on a high quality stable middleware www.eu-egi.org

  4. Transition Context • EGI Grid must not disrupt the successful basic grids in operation today • Must provide a transition path for everybody interested • Must show its large benefits to help NGIs to convince existing grid users to move to the EGI Grid and also to attract new users • Must preserve the large investments and technical achievements done so far • Therefore a non disruptive approach is needed www.eu-egi.org

  5. Current Situation in Europe • EGEE Grid as the largest World Grid infrastructure • gLite based, strong hierarchical model • Guarantee stable and high quality services • NorduGrid • ARC based, many commonalities with EGEE, more flexible model • DEISA as the supercomputing Grid • UNICORE based, less overlap with ARC/gLite • Smaller Grid islands • Mostly Globus and Condor based • Default choice for general grid R&D publications and prototypes • Some serving specific application communities • Includes also activities of some European countries • Commercial products • Usually single components, basic functions, limited use in academic world www.eu-egi.org

  6. Middleware: Basic Options • A shared large scale distributed infrastructure could be theoretically achieved • Interconnecting “free” individual Grid islands – however no proofs that scalability and quality problem can be solved – can not remove all barriers • Accepting a common middleware certified solution – no barriers, quality and scalability at the price of less flexibility, some danger of middleware o(g)ssification – competition limited by available funds • The proposal values “no barriers” higher since this is what all successful e-Infrastructures have adopted (EGEE/DEISA/Nordugrid/OSG..) – trying to identify risks and providing a process for their minimization www.eu-egi.org

  7. The Middleware Problem • Smooth transition • keep what we have • Barrier less Grid • move to a common certified solution (doesn’t mean a single implementation of each component) • High quality • allow maximal competition compatible with budget • Stability and dependability of services • Require to be conservative, do not change/replace too often the selected deployed components and interfaces – accept innovation rate allowed by budget www.eu-egi.org

  8. Proposes Approach • Define a common middleware solution • The Universal Middleware Toolkit (UMT) • Define processes to provide stable solution • Middleware development function as part of EGI • Common policies, rules, quality and standard compliance criteria for UMT components • Define processes to go for highest quality • Service/Component orientation, interface definition and standardization • Testing and verification process under EGI control • Define processes to satisfy users/NGI requirements • Integrated development-deployment-feedback cycle www.eu-egi.org

  9. Universal Middleware Toolkit • A common general component of the EGI infrastructure • A set of highly integrated and interoperable components and services • But initially not unique implementation – leave time to evaluate-generalize-specialize • Based on standards • Their further definition and evolution an integral part of the UMT evolution • A glue that connects the EGI Grid together guaranteeing to NGIs quality and scalability in sharing their resources www.eu-egi.org

  10. UMT Initialization • UMT cannot be defined anew, from scratch • This would violate the smooth transition requirement • Can only be based on existing proved solutions for large scale Grids • gLite, ARC and UNICORE (in Europe) • The mentioned middleware provides a solid foundation for UMT • Initial component selection (next slide) www.eu-egi.org

  11. Initial UMT Component Selection • Definition of specification/criteria should start almost immediately • After the UMT idea is accepted and endorsed • A role for specific task force • Goal to have a list of basic/core services this year • Establish working relation with DEISA/PRACE • Pragmatically will be based on gLite/ARC • Includes a first set of interface definition (“UMT standards”) involving UNICORE • Must also help to define the vertical interfaces (“hooks”) www.eu-egi.org

  12. UMT Evolution • Further extension and improvement of interface definitions (up to standardization) • EGI core middleware function • Testing and certification • New component submission process open to everybody • Testing primary a responsibility of developers • Accepted components must receive a funding by EGI for their maintenance and eventual further evolution • The process includes components removal (out-phasing), too www.eu-egi.org

  13. UMT Further Properties • A path toward “Grid IP” – a common set of necessary standardized services • Provides “vertical” interfaces for independent development of thematic higher services • Such development in general out of scope of UMT/EGI (but not EC, on the contrary!) • Source of new ideas and approaches to UMT • Novel use (through higher services) provides feedback to UMT and improves component’s quality • UMT as a core part of a Grid software ecosystem www.eu-egi.org

  14. EGI.org Core Middleware Function • EGI.org Development Unit • Not doing actual development, but responsible for the UMT evolution process • A place where UMT evolution will be planned and coordinated – real work outsourced • 8 FTEs • Chaired by EGI Chief Technical Officer (CTO) • Will help to put in a close loop UMT component developers, Operations and Users (Applications) • The only known way to guarantee a successful evolution of a moving target like the Grid middleware www.eu-egi.org

  15. Development Unit Objectives • Common baseline architecture; • Full interoperability of existing services through standardization; • Validation and final testing of the released services; • Increasing complementarities and specializations of the included services (“Grid IP”); • Adoption of application and operation requirements; • Convergence and interoperability through the implementation of standard interfaces with Globus, Condor and other non-EU stacks; • Definition of additional APIs that will allow independent development of higher level services. www.eu-egi.org

  16. Conditions for Component Inclusion • Interoperability • Guaranteeing the barrier less EGI Grid • Completeness • No essential service is missing • Scalability • Guaranteeing the large scale EGI Grid • Simplicity • Easy to deploy and manage • Extensibility • A base for independent development of higher services (or services with limited user base) www.eu-egi.org

  17. Interoperability • Other middleware solutions will continue to exists and be developed • UMT will provide both the vertical (“classical”) and horizontal (“hooks”) interoperability • UMT will actively support independent development of other/higher services • Providing horizontal interoperability through interfaces (“hooks”) for external developers • Both for VO/application based development and for (EU) Computer Scientists to use UMT as the reference base www.eu-egi.org

  18. UMT Development • Not direct part of the EGI.org • However, EGI must guarantee that the development is not stalled • And EGI must also guarantee that it has proper tools to steer the UMT development process • For start, based on the work of the European Middleware Consortia (gLite, ARC, UNICORE) • Include progressively other contributions satisfying quality criteria and scalability if required by NGIs and compatible with the UMT architecture • A gradual move towards a UMT community of development teams responsible for the selected components • UMT should become a world reference www.eu-egi.org

  19. Funding – General Principles • Budget as a tool to steer the UMT development process • Currently, the Middleware Consortia are usually funded on 50% through the developer’s institutions and remaining 50% a combination of national and EC funding • We propose • To ask EC to provide the 50% of the UMT development cost as an expression of interest in support of the necessary level of innovation • To keep the 50% co-funding of developer’s institutions • To have EGI.org controlling of the budget www.eu-egi.org

  20. Funding – More Details • The certification and testing will remain a mandatory service of EGI.org • However, a large fraction of these activities can be outsourced to entities submitting components for testing and certification • EGI.org will provide the final certification • The actual support and maintenance of selected components is a service that can gradually be moved to service charge model as these get more consolidated and used • The UMT services/components will be mandatory for NGIs • Alternative (higher level) components can be accepted, serving only a subset of NGIs – these can be charged directly www.eu-egi.org

  21. Estimated Middleware Development Efforts • Numbers are based on the input provided by current consortia on their total development efforts • Provides a rough estimate of the real development and software hardening (quality) efforts • It is a man power consuming task • Even as currently sometimes hidden in other activities, it must be accounted for • Includes UMT conformance testing • Does not include development of tools for operation automation • gLite: 61 FTEs • All current components • ARC: 29–45 FTEs • The higher number for all components, the smaller for components suggested to become part of UMT • UNICORE: 12 FTEs • All components • EGI.org standardization, certification …: 8 FTEs • Only this effort goes directly with EGI.org www.eu-egi.org

  22. Summary • Middleware considered an essential part of EGI • Middleware existence and further development is of utmost importance for EGI Grid – funds should be assured • UMT as a set of EGI middleware components and services • Service/Interface standardization • Open component inclusion process • Responsibility for support and maintenance of selected components • Development costs shared with developers, expected EC contribution to cover the second half • EGI.org in appropriate control www.eu-egi.org

  23. Some Items for Discussion • The UMT general concept • The UMT initialization/start up • Component selection process • Relationship to independent middleware R&D both in Europe and outside • Industrial involvement • Validation and Certification of new components – necessary tools • Budgetary requirements and budget control www.eu-egi.org

More Related