EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833

1. Web Services, EGEE and Richard HopkinsTraining Team, National e-Science Centre, Edinburgh, UKrph@nesc.ac.uk National Grid Services Induction March 11 2005 EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833

2. Outline and Acknowledgements • Grids and Web Services • EGEE • gLite Acknowlegements • This talk has drawn on material produced by many people, including - • Tony Hey • Mike Mineter • EGEE colleagues NGS Induction, NeSc, March 11 2005 – Web Service Grids - 2

3. Multi-processor Processor Pool Embedded Systems Flexible Manufacturing GRIDS . . . Specialised DistributedComputing Grids come within the general field of • Parallel / Distributed Computing A computational task involves coordination of components which occur • Simultaneously And/or • At physically separated locations The Distribution dimension - Parallel Distributed NGS Induction, NeSc, March 11 2005 – Web Service Grids - 3

4. Grids & Web Services A very inclusive formulation of “Grid Computing” is • Coordination of computational components • Processing / storage resources • Of up to international level of geographic separation • Crossing organisation boundaries A (Web) service is a • S/W system designed to support interoperable machine-to-machine interaction over a network. Similar concepts • For grids, the focus is on a computational model – • Running a job • Marshalling the data needed • For web services – the focus is on service inter-operability • A service can be any functionality you want to provide via the web Need to bring these technologies together • First we look closer at Web Services NGS Induction, NeSc, March 11 2005 – Web Service Grids - 4

5. Web Services Approach Web Services is the next step in the automation of inter-enterprise interaction Web Browsing • Human travel agent provides “organise holiday” service by surfing the web to look for and invoking services – book a hotel; book a plane; book a car hire; ….; confirm bookings of best options to meet client needs. Web Services • The aspiration of Web services is to provide a framework that allows that same model to be used in writing an application – • which is itself becomes an “organise a holiday” service, finding and using useful services Mode human intervention at – service provider service consumer E-mail Web browsing Web Services Yes Yes No Yes No No NGS Induction, NeSc, March 11 2005 – Web Service Grids - 5

6. Service Interaction I organise holidays I know the weather Get a car rental quote locate service ask for quote Is quote good enough? Yes Reserve it … get other resources reserved Confirm booking I locate services Service Identifier quote I book car Rentals I book hotels I convert currency I book planes NGS Induction, NeSc, March 11 2005 – Web Service Grids - 6

7. Essentials of WS Approach Need to achieve effective cooperation even though • the different services are produced by different organisations, without any design collaboration • the services are autonomously evolving • Loose coupling – minimum prior shared information between the designer of the two components of an interaction This requires • Self-description – Meta data • Tolerance of partial understanding Collaboration is on defining generic standards, rather than on specific design NGS Induction, NeSc, March 11 2005 – Web Service Grids - 7

8. I book hotels I organise holidays I locate services Core WS Policy Quote- held - 10 mins Prov Reserv. Held - 2 hours …. Basic Requirements • Service Description – WSDL • Web services Definition language, defines - • logical service – Operations. Message structures for each • Physical – protocol, location • Allows service user to bind to the service • Statically – use WSDL to generate code • Dynamically – using - • Service Registry and Discovery Services – UDDI • Universal Description, Discovery and integration • Allows one service to register with a registry known by potential users • User can discover it and obtain its description • Usability Negotiation– Policies register NGS Induction, NeSc, March 11 2005 – Web Service Grids - 8

9. Give me a quote for … Quote Type = Quote Validity = 1 hour Price = £200 …. Core WS Basic Requirements • Communication Protocol - SOAP • Request - Response • Other patterns – e.g notification Ask car rental service to notify me of any special deals To notify me if my reservation hold time is about to expire • Common Language – XML Documents • Tag-value pair • Defined by a Schema • Used for messages, policies, service definition • Specific extensibility mechanism • Self Description – tagged values, WSDL • Necessary for loose coupling • Autonomous evolution • Autonomous evolution – discovery service – UDDI or out of band NGS Induction, NeSc, March 11 2005 – Web Service Grids - 9

10. WS-Transaction Framework WS-notification UDDI* WS- security … WSRF Core WS WSDL* SCHEMAS* WS-addressing SOAP* WS-MetaData Exchange DTD XML* *WS-Interoperability Evolving Standards • Collaboration is on defining generic standards, not specific design • Two main standards bodies – • W3C – actually produces “recommendations” – web community • OASIS – industry – IBM, Microsoft, Sun, …. • These standards are factored to allow partial adoption and combination • The core standards • WS-I – clarifications to aid interoperability • Higher level standards built on them NGS Induction, NeSc, March 11 2005 – Web Service Grids - 10

11. WS-Transaction Framework WS-notification UDDI* WS- security … WSRF Core WS WSDL* SCHEMAS* WS-addressing SOAP* WS-MetaData Exchange DTD XML* *WS-Interoperability Core WS • XML – the standard format for all information • SCHEMA –the standard language for defining the structure (syntax/type) of a unit of information DTD is a deprecated predecessor of Schemas • WSDL – the language for defining a service – • Operations; Logical Message Structure; Bindings; locations • SOAP – the standard message format NGS Induction, NeSc, March 11 2005 – Web Service Grids - 11

12. WS-Transaction Framework WS-notification UDDI* WS- security … WSRF Core WS WSDL* SCHEMAS* WS-addressing SOAP* WS-MetaData Exchange DTD XML* *WS-Interoperability Some Further Standards • WS-Security – Framework for authentication and confidentiality • WS-Transaction Framework – for robustness of correlated interactions, e.g two phase – provisionally book everything, then confirm everything • UDDI – standard repository interface (included in WS-I) • WS-MetaDataExchange – how to communicate meta-data • …. NGS Induction, NeSc, March 11 2005 – Web Service Grids - 12

13. WS-Transaction Framework WS-notification UDDI* WS- security … WSRF Core WS WSDL* SCHEMAS* WS-addressing SOAP* WS-MetaData Exchange DTD XML* *WS-Interoperability WSRF-Related Standards • WS-Addressing - For communication of identities between services • WS-Notification - Framework of notification interaction – subscribe, publish • WSRF – Web Services Resource Framework • E.g. a quote is a persistent entity which will need to be identified in subsequent interactions to finalise a provisional booking – a resource • Consistent standard framework for creating, identifying, destroying resources NGS Induction, NeSc, March 11 2005 – Web Service Grids - 13

14. Grid Services Web Service Grids • Grid middleware can be seen as a special application area of web services dealing mainly with – • VOs • data replication • job execution • Many advantages to basing grids on web services • Cooperation between autonomously evolving components • Mix-and-match of components • Combining Aspects of grids with other Web Service application domains • Rapid Development of higher-level services • Leverage industrially-produced support packages • Previously, independent simultaneous developments • Much of existing middleware components are not based on web services NGS Induction, NeSc, March 11 2005 – Web Service Grids - 14

15. Grid Services • Now trying to unify these two major developmental thrusts - OGSA – Open Grid Services Architecture • Quite central to Grids is dynamically created resources, particularly • Files • Jobs • VO membership • Thus WSRF is important to this unification – • WSRF originated from the grid world • A (web) resource is a concept similar to object in O-O programming • Multiple instances of the same type • Each with its own (changing) state NGS Induction, NeSc, March 11 2005 – Web Service Grids - 15

16. Submit Job Job J6 Job Id : J6 Notify Started : J6 Web Service grids : WSRF A resource can • Be created Globally unique id www/grid:J6 • Change state • Notify change of state • Be destroyed • Self-destruct (scheduled) Actual Grid (www/grid) – Services & Resources A Grid Access Service www/grid cancel: J6 NGS Induction, NeSc, March 11 2005 – Web Service Grids - 16

17. Standards that have broad industry support and multiple interoperable implementations Web Service Grids: Interoperability WS-I – covers Schemas, WSDL, SOAP, UDDI (assumes XML) ‘WS-I+’ profile minimal for grids BPEL - workflows WS-Addressing WS-ReliableMessaging Specifications that have/will enter a standardisation process but are not stable and are still experimental ‘WS-I+’ next- WSRF Notification Security Portlets WS-I Specifications that are emerging from standardisation process and are recognised as being ‘useful’ NGS Induction, NeSc, March 11 2005 – Web Service Grids - 17

18. Outline • Grids and Web Services • EGEE • gLite NGS Induction, NeSc, March 11 2005 – Web Service Grids - 19

19. EGEE is building a large-scale production grid service to: Underpin research, technology and public service Link with and build on national, regional and international initiatives Foster international cooperation both in the creation and the use of the e-infrastructure Collaboration Pan-European Grid Operations, Support and training Network infrastructure& Resource centres EGEE – towards e-infrastructure NGS Induction, NeSc, March 11 2005 – Web Service Grids - 20

20. Background • By 2003: • Grid technology shown to be viable • Large amount of functional middleware • …thanks to: • FP5 : DataGrid, DataTAG, CrossGrid, etc… • USA: VDT, Globus, Condor, etc. • … and others • Next step - major production infrastructure • EGEE was proposed to the EU in 2003 • 2 year project began in April 2004, with a 4-year vision. NGS Induction, NeSc, March 11 2005 – Web Service Grids - 21

21. Grids for e-Infrastructure… • In 2003, what was missing? • Production-quality (stable, mature) Grid middleware • Production-quality operational support • Grid Operation Centres, Helpdesks, etc. • Multi-discipline grid-enabled application environment • Now led by HEP, Bio-info • Administrative and policy decision framework in order to share resources at pan-European scale (and beyond) • Areas such as AAA (Authentication, Authorisation, Accounting) • End-to-end issues (Network related) • Funding Policies (Grid economics) • Resource Sharing Policies • Usage Policies • EGEE project is tackling most of the above issues NGS Induction, NeSc, March 11 2005 – Web Service Grids - 22

22. Pilot New In the first 2 years EGEE will • Establish production quality sustained Grid services • 3000 users from at least 5 disciplines • integrate 50 sites into a common infrastructure • offer 5 Petabytes (1015) storage • Demonstrate a viable general process to bring other scientific communities on board • Propose a second phase in mid 2005 to take over EGEE in early 2006 NGS Induction, NeSc, March 11 2005 – Web Service Grids - 23

23. 70 leading institutions in 27 countries, federated in regional Grids ~32 M Euros EU funding for first 2 years starting April 2004(matching funds from partners) Leveraging national and regional gridactivities Promoting scientific partnershipoutside EU EGEE Organisation NGS Induction, NeSc, March 11 2005 – Web Service Grids - 24

24. Activities Definition • Network Activities • NA1: Project Management • NA2: Dissemination and Outreach • NA3: User Training and Induction • NA4: Application Identification and Support • NA5: Policy and International Cooperation • Service Activities • SA1: Grid Support, Operation and Management • SA2: Network Resource Provision • Joint Research Activities • JRA1: Middleware Reengineering + Integration • JRA2: Quality Assurance • JRA3: Security • JRA4: Network Services Development Emphasis in EGEE is on operating a production grid and supporting the end-users NGS Induction, NeSc, March 11 2005 – Web Service Grids - 25

25. Operations - Introduction • Strategy has been to • have a robust certification and testing activity, • simplify as far as possible what is deployed, and to make that robust and useable. • In parallel construct the essential infrastructure needed to operate and maintain a grid infrastructure in a sustainable way. • Current service based on work done in LCG – culminating in the current service (“LCG-2”) • Now at the point where in parallel we need to deploy and understand gLite – whilst maintaining a reliable production service. NGS Induction, NeSc, March 11 2005 – Web Service Grids - 26

26. EGEE Service Activities • Create, operate, support and manage a production quality infrastructure • Offered services: • Middleware deployment and installation • Software and documentation repository • Grid monitoring and problem tracking • Bug reporting and knowledge database • VO services • Grid management services NGS Induction, NeSc, March 11 2005 – Web Service Grids - 27

27. SA1 – Operations Structure • Operations Management Centre (OMC): • At CERN – coordination etc • Core Infrastructure Centres (CIC) • Manage daily grid operations – oversight, troubleshooting • Run essential infrastructure services • Provide 2nd level support to ROCs • UK/I, Fr, It, CERN, + Russia (M12) • Taipei also run a CIC • Regional Operations Centres (ROC) • Act as front-line support for user and operations issues • Provide local knowledge and adaptations • One in each region – many distributed • User Support Centre (GGUS) • In FZK – manage PTS – provide single point of contact (service desk) • Not foreseen as such in TA, but need is clear NGS Induction, NeSc, March 11 2005 – Web Service Grids - 28

28. RC RC RC RC ROC RC RC RC RC RC ROC RC RC RC CIC CIC RC ROC RC CIC OMC CIC CIC CIC RC RC RC ROC RC RC RC Grid Operations • The grid is flat, but • Hierarchy of responsibility • Essential to scale the operation • CICs act as a single Operations Centre • Operational oversight (grid operator) responsibility • rotates weekly between CICs • Report problems to ROC/RC • ROC is responsible for ensuring problem is resolved • ROC oversees regional RCs • ROCs responsible for organising the operations in a region • Coordinate deployment of middleware, etc • CERN coordinates sites not associated with a ROC RC = Resource Centre NGS Induction, NeSc, March 11 2005 – Web Service Grids - 29

29. (Human) Networking Activities • Dissemination and Outreach: 5% of EGEE budget • Dissemination – to actively promote and raise awareness of the EGEE project • Outreach – to identify and contact potential new user communities • Training and Induction: 4% of EGEE budget • Induction – to introduce and orient - users and members • Training – to create, collate, make available and deliver material and courses • Application Identification and Support • Process for selecting new application areas • Supports selected VO’s in porting applications • International cooperation NGS Induction, NeSc, March 11 2005 – Web Service Grids - 30

30. Dissemination • 1st project conference, Cork, April • 2nd conference in The Hague • 22-26 November • http://public.eu-egee.org/conferences/2nd • Over 300 delegates • Websites, Brochures and press releases • For project and general public www.eu-egee.org • Information packs for the general public, press and industry NGS Induction, NeSc, March 11 2005 – Web Service Grids - 31

31. Training and Induction NeSC Edinburgh UK & Ireland IHEP Protvino Russia IMPB RAS Moscow Russia ITEP Moscow Russia JINR Dubna Russia KU-NATFAK Copenhagen Denmark PNPI Petersburgh Russia RRCKI Moscow Russia GUP Linz Austria FZK Karlsruhe Germany Innsbruck Austria II-SAS Bratislava Slovakia GRNET Athens Greece ICM Warsaw Poland PSNC Poznan Poland ICI Bucharest Romania BUTE Budapest Hungary ELUB Budapest Hungary MTA SZTAKI Budapest Hungary TAU Tel Aviv Isreal INFN Rome Italy CESNET Prague Czech Rep. NGS Induction, NeSc, March 11 2005 – Web Service Grids - 32

32. Replica Catalogue Input “sandbox” “User interface” DataSets info Information Service Output “sandbox” Resource Broker SE & CE info Job Submit Event Author. &Authen. Input “sandbox” + Broker Info Job Query Output “sandbox” Publish Job Status Storage Element Logging & Book-keeping Computing Element Job Status Current production m’ware: LCG-2 NGS Induction, NeSc, March 11 2005 – Web Service Grids - 33

33. Co-existence with deployed infrastructure Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid services Site autonomy Reduce dependence on ‘global, central’ services Open source license Service oriented approach Allow for multiple interoperable implementations Lightweight (existing) services Easily and quickly deployable Use existing services where possible Condor, EDG, Globus, LCG, … Portable Being built on Scientific Linux and Windows Security Sites and Applications Performance/Scalability & Resilience/Fault Tolerance Comparable to deployed infrastructure VDT EDG . . . AliEn LCG . . . gLite: Guiding Principles NGS Induction, NeSc, March 11 2005 – Web Service Grids - 34

34. gLite Services for Release 1 JRA3 UK Access Services Grid AccessService API CERN IT/CZ Security Services Authorization Information & Monitoring Services Application Monitoring Information &Monitoring Auditing Focus on key services Authentication Data Services Job Management Services MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataManagement WorkloadManagement ComputingElement Site Proxy NGS Induction, NeSc, March 11 2005 – Web Service Grids - 35

35. VDT EDG . . . AliEn LCG . . . Approach • Role: gLite is intended as the middleware for Pan-European Grid • Approach: Combining and re-engineering existing components (as far as possible) • Middleware (MW) components now work reasonably well – problems usually elsewhere • Substantial body of experience of MW development, packing and use • Based on LCG2 • What is currently used within the EGEE project • Can be viewed as an initial version of Pan-European Grid middleware • gLIte deals with • LCG2 shortcomings • Advanced application needs • State-of-the-art Internet approaches and standards NGS Induction, NeSc, March 11 2005 – Web Service Grids - 36

36. Improvements • Extended Functionality • Incorporates features from other products • Some new functionality • Increased Modularity • Previous products tend to be monolithic – assumed to be used in entirety • Now need to move towards smaller scale of component combinability • Improved Deployability • Modularity • Installation support • Non-dedicated machines • Coherent architecture • extending functionalities and increasing modularity requires – • a revised picture of the organisation of functions • rationalisation of interface • Adopt Web Services Approach and Standards NGS Induction, NeSc, March 11 2005 – Web Service Grids - 37

37. Guiding Principles • Lightweight - Easily and quickly deployable • Mix-and-match components • Multiple services running on the same physical machine (if possible) • Interoperable • Client may talk to different independent implementations of the same service • Resilient and Fault Tolerant • Co-exist with deployed infrastructure • Reduce requirements on site components • Co-existence with LCG-2 essential for the EGEE Grid service • Platform support- Goal is to have portable middleware • Building & Integration on RHEL 3 and windows • Initial testing (at least 3 sites) using different Linux flavours (including free distributions) NGS Induction, NeSc, March 11 2005 – Web Service Grids - 38

38. Guiding Principles • Service oriented approach • Based on web services standards • Service autonomy • User may talk to services directly or through other services (like access service) • Open Source software license • No restriction on usage (academic or commercial) beyond acknowledgement • That's for MW - for application software, Sites must obtain appropriate licenses before installation • Main Documentation Application requirements: http://egee-na4.ct.infn.it/requirements/ Architecture: https://edms.cern.ch/document/476451 Design: https://edms.cern.ch/document/476451 Release plan: https://edms.cern.ch/document/468699 NGS Induction, NeSc, March 11 2005 – Web Service Grids - 39

39. Application communities and EGEE • LCG and Bio-informatics from day 1 • New application communities are selected by the EGEE Generic Applications Advisory Panel: • For new applications • See: EGEE web site (NA4 activity) and also http://agenda.cern.ch/age?a042351 • Selected are: • Computational chemistry • Earth sciences • Earth observation • Astrophysics • Also working with DILIGENT: • Virtual digital data libraries NGS Induction, NeSc, March 11 2005 – Web Service Grids - 40

40. Development Cycle LCG-1 LCG-2 gLite-1 gLite-2 Globus 2 based Web services based • Prototyping short development cycles for fast user feedback – Requirements Planning & Design Implementation Testing • Deployment : • Prototype Infrastructure • → Pre-production service • → Production Service NGS Induction, NeSc, March 11 2005 – Web Service Grids - 41

41. Open Source Software License • The existing EGEE grid middleware (LCG-2) is distributed under an Open Source License developed by EU DataGrid project • Derived from modified BSD - no restriction on usage (academic or commercial) beyond acknowledgement • Approved by Open Source Initiative (OSI) • Same approach for new middleware (gLite) • New license agreed by partners is derived from the EDG license and takes into account feedback from the World Intellectual Property Office (WIPO) NGS Induction, NeSc, March 11 2005 – Web Service Grids - 42

42. Further Information • EGEEwww.eu-egee.org • LCGlcg.web.cern.ch/LCG/ • “Concertation event” and Second EGEE conference:http://public.eu-egee.org/conferences/2nd/programme/outline.html • The GridCafewww.gridcafe.org • More EU sites: • http://www.cordis.lu/ist/grids/fp6_grid_projects.htm • http://www.gridstart.org/concertation_mtg.shtml • “e-Infrastructures Reflection Group http://www.e-irg.org • NeSC www.nesc.ac.uk NGS Induction, NeSc, March 11 2005 – Web Service Grids - 43

43. THE END NGS Induction, NeSc, March 11 2005 – Web Service Grids - 44