Grid Interoperability Through Standards

1. Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager

2. Compliance testing and QA Benchmarking Component Integration Repository Evaluation Infrastructure The OMII-Europe Standardisation process Identifying new components Standards Implementation Components Re-engineering IN Globus OUT OMII-UK Components CROWN

3. Session Outline • Overall architecture and components in OMII-Europe (Morris Riedel) • Standards and implementations from OMII-Europe • SAML (Valerio Venturi) • BES/JSDL (Stephen Crouch) • RUS and UR (Gilbert Netzer) • GLUE II (Sergio Andreozzi) • DAIS, DMI and ByteIO (Neil Chue Hong) • Future services

4. Future Work: 2008-2010 • Common implementations of specification and services for: • Data storage services • Common data service layer building on DAIS, ByteIO, DMI and GSM • Service discovery • Grid monitoring and service mangement • Grid billing and pricing • Grid visualisation and steering • Grid Authorisation

5. Architecture and SecurityOpen Standards used in real interoperability scenarios and use cases Morris Riedel, Forschungszentrum Juelich (FZJ), Jülich Supercomputing Centre (JSC), Germany Leader Infrastructure Integration (Interoperability) ActivityOpen Grid Forum 21, Seattle, 17th October 2007

6. Outline Motivation: e-Infrastructure Islands in Europe OMII-Europe in Context Common Vision: Interoperability Highway Interoperability scenarios / use cases Infrastructure to test interoperability scenarios Interoperability scenario: WISDOM Conclusion

7. DEISA Grid (Supercomputing/HPC community) Non WS-based UNICORE 5: Proprietary jobs (AJO/UPL) No Virtual Organization Membership Service (VOMS), Full X.509 Suitable for massively parallel scientific jobs (MPI, …) EGEE Grid (mainly HEP community + others) Non WS-based gLite: Proprietary jobs (JDL) Proxy-based X.509 security, but proprietary VOMS support Suitable for embarrassingly parallel scientific jobs Both Grids are currently not technical interoperable Scientists cannot use one middleware to access both Both Middleware’s had so far less adoption of open standards e-Infrastructure Islands in Europe DEISA [2] EGEE [3]

8. OMII – Europe in Context EC e-Infrastructures [4]

9. Common Vision: Interoperability Highway End-users via clients& portals GOAL: Transparencyof Grids for end-users Emerging OpenStandards „Interoperability highway“based on open standards others Grid Middlewares others Grid Resources UNICORE [5] gLite [6] GT [7] CROWN [8]

10. Infrastructure to test interoperability scenarios Venturi et al. [9] Next steps in RED/ORANGE SAML- based Attribute Authority (AA) VOMS getscentralrole & middlewareindependent

11. OMII – Europe implements open standards… OGSA-BES, OGSA-RUS, WS-DAIS (OGSA-DAI), SAML (VOMS)… E.g: OGSA - Basic Execution Services (OGSA-BES) In real deployments is not a ‘vanilla OGSA-BES interface’ available Same exact “client” works not directly with gLite & UNICORE E.g. different security models: X.509 Proxies vs. full X.509 certificates Solution in OMII-Europe: Agree on a Common Security Profile Grid interoperability: use components together… e.g. OGSA-BES job submission secured via VOMS AuthZ Interoperability scenarios / use cases: e-Infrastructure use cases requiring resources in more than one Grid! Interoperability Scenarios / Use cases

12. The following interoperability scenarios have been discussed with members of the respective projects These scenarios demonstrate the required technical interoperability between e-Infrastructures in future that can be provided by OMII-Europe However, the negotiation for getting computational time on resources within these e-Infrastructures is still required by the respective projects Disclaimer

13. WISDOM (Wide In Silicio Docking on Malaria) WISDOM aims at developing new drugs for Malaria WISDOM uses EGEE for large scale in silicio docking Comp. method for prediction of whether one molecule will bind to another Using AutoDock and FlexX software provided via gLite in EGEE Output is a list of best chemical compounds (potential drugs) That is not the final solution, only a potential list of drugs Refine best compound list via molecular dynamics(MD) Fast MD computations use highly scalable AMBER in DEISA AMBER (Assisted Model Building with Energy Refinement) , version 9 Goal: Accelerate drug discovery using EGEE + DEISA Interoperability Scenario: WISDOM (1) WISDOM [1]

14. Interoperability Scenario: WISDOM (2)

15. Conclusion Solutions for European e-Infrastructure Islands Adopt components from OMII-Europe to gain interoperability Why OMII – Europe components… Components going through quality assurance steps Components adopt standards whereever possible Components tested in real interoperability scenarios (e.g. WISDOM) Not only interesting for European e-Infrastructures Beneficial for all Grids that adopt Globus, gLite, and UNICORE E.g. National German Grid Initiative (D-Grid) Continuing work in the open standards working groups! „Interoperability highway…“ realize the „true global Grid“

16. References [1] WISDOM Project, http://wisdom.eu-egee.fr/ [2] DEISA Project, http://www.deisa.org [3] EGEE Project, http://public.eu-egee.org/ [4] European e-Infrastructures, http://www.beliefproject.org/cookbook/cookbook-intro/view [5] European UNICORE Grid middleware, http://www.unicore.eu [6] European gLite Grid middleware, http://glite.web.cern.ch/glite/ [7] US Globus Toolkit, http://www.globus.org/ [8] CROWN, http://www.crown.org.cn/en/ [9] Using SAML-based VOMS for Authorization within Web Services-based UNICORE Grids, Venturi et al., UNICORE Summit 2007 @ EuroPar 2007 [10] Improving Quantum Computing Simulations, http://www.deisa.org/applications/projects2005-2006/iqcs.php

17. Acknowledgements Open Middleware Infrastructure Institute for Europe OMII – Europe project under EC grant RIO31844-OMII-EUROPE, duration May 2006 - April 2008 Jülich Supercomputing Centre (JSC)of Forschungszentrum Jülich (FZJ) in the HELMHOLTZ association Forschungszentrum Jülich in der Helmholtz-Gesellschaft

18. IGIIW @ e-Science 2007 International Grid Interoperability & Interoperation Workshop in conjunction with e-Science 2007, Bangalore, India http://www.omii-europe.org/OMII-Europe/igiiw2007.html

19. IQCS (Improving Quantum Computater Simulation) One project of DECI (DEISA Extreme Computing Initiative) Evaluation period/test runs of IQCS code in EGEE IQCS scientists use currently their own desktop machines or some spare resources within institutes to develop/evaluate/test their code That‘s slow and the management doing it personally is cumbersome This can be significantly improved using EGEE clusters! Final IQCS code production runs within DEISA Leverage massive amount of CPUs/memory available on HPC resources Goal: Accelerate code development with EGEE + DEISA Optional: Interoperability Scenario: DECI IQCS DEISA DECI IQCS [10]

20. Optional: Interoperability appraoch in OMII-Europe Venturi et al. [9] e-Infrastructure Interoperability by using more than one technology!

21. Questions for JRA3 – Task 2 Morris Riedel m.riedel@fz-juelich.de JRA 3 Team jra3@omii-europe.com

22. Attribute Exchange Profile • Problem statement • Interoperable, pluggable, replaceable attribute services • Easier aggregation of attributes coming from difference sources • OGSA Architecture envision the use of authorization policies coming from different stakeholders • Solution • Standard interface for requesting attribute assertions

23. Attribute Exchange Profile • Problem statement • Attributes assertions understood across middleware boundaries • Solution • Agreed format for attribute assertions

24. Status of the specification • The specification is being worked on inside the OGF OGSA Authorization Working Group • Basically profiles existing specification (OASIS, WS-I) for use with Grid Services • Bases are solid • OASIS SAML V2.0 set of specifications • SAML V2.0 Deployment Profiles for X.509 Subjects • Profile presented on Monday • Those interested should join the OGS Authorization WG http://www.ogf.org/gf/group_info/view.php?group=ogsa-authz-wg

25. Specification Implementers • VOMS • gLite and OMII-Europe developed, used in EGEE, OSG, Naregi • Implementing the service • UNICORE • Implementing the client part of the profile in OMII-Europe • Part of the 6.2 release planned for early 2008 • Implementing the service in the chemomentum project • GridShib

26. Implementation status • OMII-Europe is re-engineering the Virtual Organization Membership Service (VOMS) to meet the specification • VOMS is developed also in EGEE • VOMS is an Attribute Authority focused on VO management • It allows users to register to a VO and being assigned attributes such as groups and roles • The attributes are used by Grid Services to drive authorization decisions • VOMS is already used in EGEE, OSG and Naregi • Plugin for the globus Authorization framework • OMII-Europe is also adding VOMS support to UNICORE • UNICORE is implementing the client part of the profile

27. Implementation Status • Alpha version of the VOMS SAML Service available since May 2007 for integration testing purposes • Demonstrated at OGF 20 in Manchester in conjunction with UNICORE OGSA-BES • Beta version available early November • Then into the QA process of OMII-Europe, that comprises standard compliance testing • VOMS SAML Fed back to gLite early 2008 • OMII-Europe support for VOMS • Available from version 6.2 early 2008

28. Implementation Plans • Undergoing integration with BES services re-engineered by OMII-Europe • UNICORE OGSA BES already done, demonstrated at OGF 20 • gLite CREAM BES nearly done • May be demonstrated at SC07 • Planned integration with other OMII-Europe re-engineered components • RUS services: SGAS, gLite DGAS, UNICORE OGSA RUS • OGSA DAI • Integration with BES services is a proof of concept of the first addressed issue • Availability of attributes across middleware boundaries

29. Further plans • Shibboleth 2 on its way out • GridShib will implement the same interface • We plan to test integration scenarios • Shibboleth attributes concern Home Institution Affiliation • VOMS attributes concern Virtual Organization • This would be proof of concept of the second issue addressed • Easy aggregation of attributes coming from different sources

30. Job Submission: OGSA-BES & JSDL Moreno Marzolla (moreno.marzolla@pd.infn.it), Stephen Crouch SOTON s.crouch@omii.ac.uk

31. Contents • The problem of incompatible job submission systems • The solution: interoperability through standards • OGSA-BES • JSDL • Standards status and limitations • Implementations of standards within OMII-Europe • Computation endpoints • Meta-scheduler • Future interactions with other OMII-Europe activities

32. Setting the Scene: Grid Job Submission • User problem: • User (e.g. researcher) has a problem they need to solve computationally • They do not have local access to appropriate resources in terms of computation and/or applications • A non-local provider: • Has the required computation/application resources • Wishes to allow them to be shared by appropriate users • Grid job submission provides the technological ‘glue’ to allow users to utilise the provider’s computational/application resources UserJob Job Submission Comp/AppResources

33. Desc A Desc B … Job Submission B Comp/AppResources … … The Problem Job Submission A Comp/AppResources • Different job submission systems often incompatible in terms of • Interface • Job description • Security (more general problem) • Overhead in terms of • Use – learning curve • Infrastructure support – multiple client installations • Application development – knowledge of multiple APIs • Interoperation solutions expensive in terms of development & maintenance ClientApp Client (A) Client (B)

34. BESI/F Job Sub. A Comp/AppResources ClientApp JSDL BESClient JSDL BESI/F JobSub. B Comp/AppResources … … … Solution: Interoperability through Standards • Two key standards for two key elements: • The Basic Execution Service web service interface (OGSA-BES) • 1 instance of job container within OGSA-EMS (Execution Mngt Service) • Handles basic job lifecycle management • Defines simple (but extendable) job state model • Pending, running, cancelled, failed or finished • The Job Submission Description Language (JSDL) • Specify job executable, data staging and resource requirements

35. Summary of OGSA-BES

36. JSDL Example I <JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <JobIdentification> <JobName>cat job</JobName> <Description>cat job description</Description> <JobAnnotation>no annotation</JobAnnotation> <JobProject>an example project</JobProject> </JobIdentification> <Application> <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/11/ jsdl-posix"> <Executable>/bin/cat</Executable> <Argument>dir1/file1.txt</Argument> <Output>stdout.txt</Output> <Error>stderr.txt</Error> </POSIXApplication> </Application>

37. JSDL Example II <DataStaging> <FileName>dir1/file1.txt</FileName> <CreationFlag>overwrite</CreationFlag> <Source> <URI>http://server.domain/helloworld.txt</URI> </Source> </DataStaging> <DataStaging> <FileName>stdout.txt</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</DeleteOnTermination> <Target> <URI>ftp://anonymous:anonymous@ftpserver.domain:45521/public/output/stdout.txt</URI> </Target> </DataStaging> </JobDescription> </JobDefinition>

38. Standards Status & Limitations • Both JSDL & OGSA-BES have reached version 1.0 (recommendation) • Future support being considered • Parametric jobs – being considered as JSDL extension • Collections of jobs with dependencies • Neither addresses security • Fundamental requirement for interoperability in real deployments • Mutually compatible security model (JRA3/Security, JRA1/VOMS)

39. Implementations: Endpoint Supportfor BES & JSDL • BES 1.0/JSDL 1.0 support developed in OMII-Europe for • UNICORE 6.1 (Jan/Feb 2008): • BES/JSDL support in vendor middleware • gLite 3.1: • CREAM-BES component based on CREAM-CE • Available through OMII-Europe repository in Dec 2007 • Globus 4.2: • Some BES/JSDL support exists within vendor middleware • Work progressing – target release date TBD • Support for BES within other job submission implementations: • OMII-UK, Beihang, Microsoft, University of Virginia, Platform Computing

40. BESI/F BESI/F Job Sub. A Job Sub. B Res. Res. MetaScheduler JSDL BES I/F JSDL JSDL Implementations: Meta-schedulingsupport for BES & JSDL • BES/JSDL 1.0 support in CROWN • Progressing towards support for BES 1.0 • CROWN Meta-scheduler hosted within Globus 4.2: • Accept BES submissions, delegate to BES endpoints based on policy • Release targeted March 2007, available from OMII Europe repository ClientApp BESClient

41. Future Interactions with OMII-Europe Activities • Security (JRA3) • Support common security profile • Compliance testing (SA2) w.r.t. OGSA-BES/JSDL • Test endpoints • Meta-scheduler ‘plugs in’ to compliance test • Benchmarking (JRA4) • Training (NA3)

42. Gilbert Netzer (KTH)‏ Accounting in OMII-Europe

43. Outline • Accounting Standards • Overview • Usage Record Format • Resource Usage Service • OMII-Europe Accounting Activity • Goals • Used Standards • Schedule • OMII-Europe Accounting Scenarios • Questions

44. Benefits of Standardization • Many (batch) systems are alike • All batch systems collect similar metrics • However units, data format and semantics differ • Large systems are heterogeneous • Standards help in building these systems • Off-the-shelf components can be plugged together • Standards give guidance for implementations • Define parts of the requirements • Can help defining the scope of an application • Promote sharing of knowledge

45. OGF Standards for Accounting • Usage Record Format (URF, GFD-R-P. 98)‏ • Recommends a document format for usage information • Does not address transport of information • Proposed Recommendation status • Interoperability workshop scheduled for OGF22 • Resource Usage Service (RUS, GWD)‏ • Specifies means to transport usage information • Uses the URF as data format • Working Draft stage • Meetings were held at OGF19, 20 and 21

46. Accounting Standards Usage Scenario

47. OMII-Europe Goals • Make existing accounting infrastructure interoperable • Use OGF standards as vehicle: • Usage Record Format (UR)‏ • Resource Usage Service (RUS)‏ • Address the three middlewares of the project: • Distributed Grid Accounting System (DGAS)‏ • SweGird Accounting System (SGAS)‏ • UNICORE-RUS • First step, focus on the existing basics only • Augment existing systems to allow easy transition

48. OMII-Europe Schedule • Alpha versions demonstrated at OGF20 (Manchester)‏ • DGAS client uploads/downloads from SGAS server • UNICORE demonstrated using RUS for monitoring • Beta versions November-December 2007 • Will have both server and client components for all three middlewares • Will be available for download from repository • Final versions for general public release • Will be ready in March 2008 • Software will be released via OMII-Europe Repository • We also try to feed SW back to the MW distributions

49. Alpha Demonstration - OGF20 UNICORE gLite DGAS Globus SGAS End User Applications LLView DGAS Clients SGAS Clients Services UNICORE RUS LUTS Resources UNICORE UR Generator DGAS Sensor OMII-Europe added value (RUS)‏ UNICORE Alpha at German eScience 2007

50. OMII-Europe Accounting Scenario UNICORE gLite DGAS Globus SGAS End User Applications LLView DGAS Clients SGAS Clients Services UNICORE RUS HLR RUS Service LUTS Resources UNICORE UR Generator DGAS Sensor JARM LegacyInteraction OMII-Europe added value (RUS)‏