NGS in the future: emerging middleware

1. NGS in the future:  emerging middleware Mike Minetermjm@nesc.ac.uk

2. NGS middleware evolution EGEE… ETF NGS Other software sources UK,Campus and other grids Software with proven capability & realistic deployment experience Prototypes & specifications Operations ‘Gold’ services Feedback & future requirements Engineering Task Force Deployment/testing/advice

3. Outline • Middleware currently being prepared for deployment • Resource broker • (NGS portal – yesterday!) • The context for the next generation of middleware:seeking convergence with web services • Under assessment: • gLite middleware from EGEE • OMII • GT4

4. Resource broker • (This is NOT the SRB!!!) • Current NGS middleware : Toolkits inviting development of higher level services • On the current NGS we have • GRAM to submit jobs • Information service to tell us what queues are busy • The RB takes the work out of deciding where to run a job

5. Input “sandbox” DataSets info Output “sandbox” SE & CE info Job Submit Event Job Query Publish Job Status Storage Element Major components Replica Catalogue “User interface” Information Service Resource Broker Author. &Authen. Input “sandbox” + Broker Info Output “sandbox” Logging & Book-keeping Computing Element Job Status

6. Resource broker • Job Description Language file: describes resources needed by a job • Commands analogous to GT2: • edg-job-submit <jdl filename> • edg-job-status <dg-job-id> • edg-job-get-output <dg-job-id>

7. Example • edg-job-submit myjob.jdl • Myjob.jdl • JobType = “Normal”; • Executable = "$(CMS)/exe/sum.exe"; • InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"}; • OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; • Requirements = other. GlueHostOperatingSystemName == “linux" && • other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000; • Rank = other.GlueCEStateFreeCPUs;

8. More about the RB • Developed by the European DataGrid project, EDG then “hardened” by LCG, and now one of the sources for the EGEE middleware • Uses components of Condor • matchmaker and Condor-G • Try the GENIUS portal on GILDA • GILDA is a dissemination grid running the LCG-2 middleware • Demo site: https://grid-demo.ct.infn.it/ • And look athttp://lcg.web.cern.ch/LCG/http://www.hep.ph.ic.ac.uk/e-science/projects/demo/index.html

9. Implications for the NGS • Integration with NGS core nodes in progress • “UI” requirements??: • LCG user interface + OGSA-DAI + SRB client • Lighter-weight alternatives? • To packaging? • For client software

10. http://www.hep.ph.ic.ac.uk/e-science/projects/demo/index.htmlhttp://www.hep.ph.ic.ac.uk/e-science/projects/demo/index.html • Monitoring of the current LCG-2/ EGEE-0 system, that includes resource brokers

11. Resource broker -summary • The resource broker receives a job description in JDL • It chooses a batch queue for job submission, using the information services • Its an example of the higher services that can be deployed for the NGS, built upon the current toolkits

12. Outline • Middleware currently being prepared for deployment • Resource broker • (NGS portal – yesterday!) • The context for the next generation of middleware:seeking convergence with web services • Under assessment: • gLite middleware from EGEE • OMII • GT4

13. Convergence of Web Services and Grids Web servicesResource Framework Commercial web developments “big Science” research OGSIGrid prototypes Web services World-wide web INTERNET High-end computing High throughput-computing Massively parallel computing

14. Convergence with WS • Why? • Opens integration of grids and WS worlds! • Need service orientation for grids! • Need easier installation!! Grid services SOAP XML Grid services http, https Operating system; TCP/IP O/S; TCP/IP

15. Moving to WS: • Leverage WS hosting environments • OGSI first attempt • Building on WS • Many projects • EGEE’s gLite • Now moving towards “WS-RF” • Set of emerging standards • Globus Toolkit 4 Grid services OGSI – GT3 Now: GT4 Grid services Axis SOAP XML http, https TOMCAT O/S; TCP/IP O/S; TCP/IP

16. Outline • Middleware currently being prepared for deployment • Resource broker • (NGS portal – yesterday!) • The context for the next generation of middleware:seeking convergence with web services • Under assessment: • gLite middleware from EGEE • OMII • GT4 • Introduction to EGEE and OMII follows • You know Globus already!

17. EGEE is building a large-scale production grid service to: Underpin research, technology and public service Link with and build on national, regional and international initiatives Foster international cooperation both in the creation and the use of the e-infrastructure Collaboration Pan-European Grid Operations, Support and training Network infrastructure& Resource centres EGEE – towards e-infrastructure

18. Project Goals A four year programme: • Build, deploy and operate a consistent, robust and secure grid that attracts new computing resources • Improve and maintain the middleware in order to deliver a reliable service to users • Attract new users from science and industry and ensure training and support for them

19. Pilot New In the first 2 years EGEE will • Establish production quality sustained Grid services • 3000 users from at least 5 disciplines • integrate 50 sites into a common infrastructure • offer 5 Petabytes (1015) storage • Demonstrate a viable general process to bring other scientific communities on board • Propose a second phase in mid 2005 to take over EGEE in early 2006

20. 70 leading institutions in 27 countries, federated in regional Grids ~32 M Euros EU funding for first 2 years starting April 2004(matching funds from partners) Leveraging national and regional gridactivities Promoting scientific partnershipoutside EU EGEE Organisation

21. Activities Definition • Network Activities • NA1: Project Management • NA2: Dissemination and Outreach • NA3: User Training and Induction • NA4: Application Identification and Support • NA5: Policy and International Cooperation • Service Activities • SA1: Grid Support, Operation and Management • SA2: Network Resource Provision • Joint Research Activities • JRA1: Middleware Reengineering + Integration • JRA2: Quality Assurance • JRA3: Security • JRA4: Network Services Development Emphasis in EGEE is on operating a production grid and supporting the end-users

22. Co-existence with deployed infrastructure Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid services Site autonomy Reduce dependence on ‘global, central’ services Open source license Service oriented approach Allow for multiple interoperable implementations Lightweight (existing) services Easily and quickly deployable Use existing services where possible Condor, EDG, Globus, LCG, … Portable Being built on Scientific Linux and Windows Security Sites and Applications Performance/Scalability & Resilience/Fault Tolerance Comparable to deployed infrastructure VDT EDG . . . AliEn LCG . . . gLite: Guiding Principles

23. gLite Services for Release 1 JRA3 UK Access Services Grid AccessService API CERN IT/CZ Security Services Authorization Information & Monitoring Services Application Monitoring Information &Monitoring Auditing Focus on key services Authentication Data Services Job Management Services MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataManagement WorkloadManagement ComputingElement Site Proxy

24. RC RC RC RC ROC RC RC RC RC RC ROC RC RC RC CIC CIC RC ROC RC CIC OMC CIC CIC CIC RC RC RC ROC RC RC RC Grid Operations • The grid is flat, but • Hierarchy of responsibility • Essential to scale the operation • CICs act as a single Operations Centre • Operational oversight (grid operator) responsibility • rotates weekly between CICs • Report problems to ROC/RC • ROC is responsible for ensuring problem is resolved • ROC oversees regional RCs • ROCs responsible for organising the operations in a region • Coordinate deployment of middleware, etc • CERN coordinates sites not associated with a ROC RC = Resource Centre

25. Open MiddlewareInfrastructure Institute • The slides that follow were selected and (in a few cases) modified by Mike Mineter (NeSC) from those presented in January 2005 at an OMII training daySteven Newhouse, Peter HendersonStephen Crouch & Karen Ng • Goal of this presentation: to rase awareness of the OMII and its OMII_1 release MM

26. Open MiddlewareInfrastructure Institute OMII goal: to be the source of open source grid software • Institute of the University of Southampton • Utilise existing software and standards • Production focused software development • Integrate, test & document ‘a product’ • Focus on the user experience • Easy to install & use • Utilise existing software and standards • Provide a solid web service base for others to build on

27. Where does our software come from? • Open Source Community • Tomcat, Axis, etc., • Software Repository • Accept software contributions • Software deployed, tested & graded to provide feedback • Managed Programme • Fill gaps to build a solid enabling infrastructure • Projects to bring research software to production quality

28. OMII_1 release:A basic File-Compute Grid • Enables a generic computational task • Move input data from the client to the service provider • Process the data using an application on the service provider • Retrieve the output data from the service provider

29. Resource Mgmt Servlet Acct Mgmt Servlet Account Allocation Data Job TestService Happy Axis Static Webpage AXIS TOMCAT OMII Server Infrastructure ExampleService PBAC WS-Security

30. Try out the OMII_1 client ! • Register at www.omii.ac.uk & login • Goto the downloads page • Download the client distribution • SuSE 9.0 • Client may work on other Linuxs but no exhaustive testing • Windows XP (SP 1 & 2) • Distribution requires JDK 1.4.2_04 • Does not work with ‘just’ a JRE • Will not work with JDK 1.4.2_05/06 & JDK 1.5.0 • No testing with earlier JDKs.

31. Summary • Middleware currently being prepared for deployment • Resource broker • (NGS portal – yesterday!) • The context for the next generation of middleware:service orientation, based on WS and the emerging standards • Under assessment by the Engineering Task Force: • gLite middleware from EGEE • OMII • GT4

32. Additional slides - background

33. Managed Programme • GridSAM (Job Submission & Monitoring service) • BPEL (Workflow service) • Grimoires (Registry service based on UDDI) • FIRMS (Reliable messaging) • FINS (Notification) • GeodiseLab (Matlab toolbox) • WSRF::Lite integration • OGSA-DAI (Database service) • WSeSS (Using SSH to tunnel requests to resources)

34. Grid Architecture Today • The best way of designing Grids… • Loosely coupled services • Message based exchange • The best way of running Grids… • Interoperability between versions & grids • Standards for infrastructure & services • The best way of building Grids… • Leverage existing infrastructure & standards • Use Web Services…

35. Some Web Service Definitions • A service is the logical manifestation of some physical or logical resources (databases, programs, devices, humans, etc) and/or some application logic that is exposed to the network • Service interaction is facilitated by message exchanges • A service is an abstract resource that represents a capability of performing tasks that represents a coherent functionality from the point of view of provider entities and requester entities. To be used, a service must be realised by a concrete provider agent

36. Web Services (WS) • XML: Platform neutral mechanism to describe data • SOAP: Mechanism to describe message exchange • Simple Object Access Protocol • Not simple and nothing to do with Objects! • Service Oriented Access Protocol • Re-engineering of acronym to fit current use! • WSDL: Defines the service interface

37. More WS concepts… • Services have to reside in a supporting environment: • Called: hosting environment or container • Marshals requests into and response out of the service • Service can discover local configuration parameters • Provides a standard infrastructure for service developers • Processing incoming requests & outgoing responses • Called: Message handlers • Manipulates elements of the message header • Primarily the SOAP header • Handlers can be applied to message traffic into or out of the whole container or a specific service

38. Putting it all together… • Architecturally web services provide… • Process of independent loosely coupling services • Defining service interfaces (or contract) • Defining the format of the messages interchange • Platform neutral • Flexible granularity • Clearly defined boundaries • Need an implementation…

39. OMII_1 as a Service Provider • Goal: I want others to access my resources & applications • I want to provide secure controlled access to: • My applications: • Specify who can access which applications • My computational resources: • I can limit external usage of my resources • Provides an interface that allows remote users to access my resources • Enable collaboration with other partners

40. OMII_1 as a User (or Client) • Goal: I want to use other resources & applications • Through a network of service providers I can…: • Gain access to applications that I do not have installed locally • Use remote machines with more CPU, memory or storage • Process larger problems sizes • Transparently switch between different service providers • No exposure to underlying OS, queuing policy, disk layout etc.

41. OMII 1:Basic File-Compute Grid • Consists of: • Base (Tomcat 5.0.25 & Axis 1.2b) • Extensions (Axis Handlers) • WS-Security • Process Based Access Control • Basic Services • Sample application • Plus installers, README’s & documentation

42. OMII 1:Basic Services • Based on a group of four services • Functional: Data & Application execution • Running jobs using pre-installed applications • Movement of input and output data files • Management: Account and Resources • Must have an account with a service provider • Or delegated access to someone else’s account

43. Process Based Access Control:A model for implementing AAA • Authentication: CA issued X.509 certificates • Authorisation: Interaction dependent authorisation process • Access control lists tied to process context and state • i.e. impose server side workflow requirements • Supports “delegation” and “subordination” actions • Accounting: Activity matched against allocated quota • Clients control who can access “their” allocated quota • Collaboration with minimal overhead for service providers