1 / 26

NA3 Strategy for Future Grids Malcolm Atkinson Director of the National e-Science Centre

NA3 Strategy for Future Grids Malcolm Atkinson Director of the National e-Science Centre. Induction Course, Edinburgh, 27 April 2004. www.eu-egee.org. EGEE is a project funded by the European Union under contract IST-2003-508833. Contents. What is the Web Service Resource Framework

mfisher
Download Presentation

NA3 Strategy for Future Grids Malcolm Atkinson Director of the National e-Science Centre

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NA3 Strategy forFuture GridsMalcolm AtkinsonDirector of the National e-Science Centre Induction Course, Edinburgh, 27 April 2004 www.eu-egee.org EGEE is a project funded by the European Union under contract IST-2003-508833

  2. Contents • What is the Web Service Resource Framework • Why has it emerged? • What has it to do with Grids? • What are the parts of WSRF? • What is the status of WSRF? • Standards process • Implementations • Globus Alliance Plans • WSRF in Perspective • What is important? NA3 Induction Course Development, Edinburgh, 27 April 2004 - 2

  3. Reminder: what are our goals? • Address the challenge • build inter-enterprise systems • NOT just connecting systems • WORK together to build infrastructure • That persistently and adaptively supports multiple Virtual Organisations • VOs that span organisational structures • Distributed implementation and operation • Pioneering new ways of working • John Taylor’s vision We pioneer & transfer results to industry Easy tasks can use any technology Only someof our goalsalign withindustry NA3 Induction Course Development, Edinburgh, 27 April 2004 - 3

  4. Why OGSI adopted web services • Expectation: WS would meet several Grid needs • E.g. Standard interface definition language • Foundation for better engineering • E.g. Standard invocation mechanism • Foundation for interoperability • But other channels used for performance • Good commercial tooling (eventually) • Reliability and performance • Service-Oriented Architecture • Has valuable scalability and durability properties • E.g. ICENI using Jini NA3 Induction Course Development, Edinburgh, 27 April 2004 - 4

  5. Web Servicescomponents and framework • Not a silver bullet or a complete solution! • Most of the engineering effort • What you do when you get a message • Not how you address, package and deliver it • Most of the standardisation effort • Agreeing how to factor large systems and the semantics of services • Agreeing conventions for information in messages • Confusing & Rival standards proposals • Limited quality public implementations • Don’t give up – engage and help fix it? • Is this the role of EGEE? • Is there just one answer? • Incremental adoption of WS-I, WS-Security, WS-Addressing • Incremental adoption of WSRF as it emerges We all agreethat it is agood strategyto use webservices. Theissue are: Which ones to adopt when? andWhat conventionsorganise oursystems? NA3 Induction Course Development, Edinburgh, 27 April 2004 - 5

  6. OGSI & GT3 investment Three forms of investment • Architectural effort • OGSA: Use cases, Design Patterns, … • Factoring & describing a complex engineering domain • Standardisation effort • OGSI, DAIS, WS-Agreement, etc. • WSDL 2.0, WSDM, WS-Security, etc. • Implementation effort • Combined OGSI & Grid component work This investment carries forward into WSRF NA3 Induction Course Development, Edinburgh, 27 April 2004 - 6

  7. GT1 GT2 OGSI Started far apart in apps & tech Have been converging ? WSDL 2, WSDM WSDL, WS-* HTTP CombiningGrid and Web Services – First try Technology intercept is not easy Grid People accepted OGSI Web • Several (some partial) implementations • Issues: technical, political & commercial • Successes: a number of operational grids NA3 Induction Course Development, Edinburgh, 27 April 2004 - 7

  8. GT1 GT2 OGSI Started far apart in apps & tech Have been converging WSRF WSDL 2, WSDM WSDL, WS-* HTTP CombiningGrid and Web Services: Second try Technology intercept is still not easy Grid Not the only possible technical solution Web • Support from major WS vendors • especially service management suppliers • e.g., CA, HP, IBM, Fujitsu, BEA, SAP, … NA3 Induction Course Development, Edinburgh, 27 April 2004 - 8

  9. Core Ideas in WSRF • Preserves OGSI functionality • Lifetime, properties, notification, error types, … • Separates service from resource • Service is static and stateless • Resource is dynamic and stateful • Builds on WS-Addressing • Is WS-I compliant • But note that WS-I alone doesn’t make the problems go away, still need to worry about how to manage lifetime, naming, state, … NA3 Induction Course Development, Edinburgh, 27 April 2004 - 9

  10. “Components” of WSRF WS-Addressing March 04 www-106.ibm.com/developerworks/library/specification/ws-add/ WSRF White paper on modelling stateful resources www.globus.org/wsrf/ WS-ResourceLifetime March 04 www.globus.org/wsrf/ WS-ResourceProperties March 04 www.globus.org/wsrf/ WS-BaseFaults March 04 www.globus.org/wsrf/ WS-RenewableReferences March 04 www.globus.org/wsrf/ WS-ServiceGroup March 04 www.globus.org/wsrf/ WS-Notification WS-BaseNotification March 04 …/specification/ws-notification/ WS-Topics March 04 …/specification/ws-topics/ WS-BrokeredNotification March 04 …/specification/ws-pubsub/ NA3 Induction Course Development, Edinburgh, 27 April 2004 - 10

  11. From OGSI to WSRF:Refactoring and Evolution Identity &naming is being done by OGSA NA3 Induction Course Development, Edinburgh, 27 April 2004 - 11

  12. techPre interop GGF10 OASIS GW TC1 • Improved robustness, scalability, performance, • usability GT3.2 4.0 b Q2 4.0 Q3 3.2 March GT4.0 • WSRF; some new functionality; further usability, performance enhancements 4.2 Q2 ‘05 GT4.2 Numerous new WSRF-based services GT & WSRF Timeline 2004 2005 Not waiting for finalisation of WSRF specs. Use as submitted NA3 Induction Course Development, Edinburgh, 27 April 2004 - 12

  13. GSI WU GridFTP MDS2 WS-Security RFT (OGSI) WS-Index (OGSI) RLS Security Data Management Information Services Components in GT 3.0 JAVA WS Core (OGSI) Pre-WS GRAM OGSI C Bindings WS GRAM (OGSI) Resource Management WS Core NA3 Induction Course Development, Edinburgh, 27 April 2004 - 13

  14. WU GridFTP JAVA WS Core (OGSI) Pre-WS GRAM MDS2 GSI WS GRAM (OGSI) RFT (OGSI) OGSI C Bindings WS-Index (OGSI) WS-Security RLS OGSI Python Bindings (contributed) CAS (OGSI) OGSA-DAI SimpleCA pyGlobus (contributed) XIO Security Data Management Resource Management Information Services WS Core Components in GT 3.2 NA3 Induction Course Development, Edinburgh, 27 April 2004 - 14

  15. New GridFTP JAVA WS Core (WSRF) Pre-WS GRAM MDS2 GSI RFT (WSRF) C WS Core(WSRF) WS-GRAM (WSRF) WS-Index (WSRF) WS-Security CSF (contribution) RLS CAS (WSRF) OGSA-DAI SimpleCA pyGlobus (contributed) Authz Framework XIO Security Data Management Resource Management Information Services WS Core Planned Components in GT 4.0 NA3 Induction Course Development, Edinburgh, 27 April 2004 - 15

  16. Importance of collaboration: VDT • A highly successful collaborative effort • VDT Working Group • VDS (Chimera/Pegasus) team • Provides the “V” in VDT • Condor Team • Globus Alliance • NMI Build and Test team • EDG/LCG/EGEE • Middleware, testing, patches, feedback … • PPDG • Hardening and testing • Pacman • Provides easy installation capability • Currently Pacman 2, moving to Pacman 3 soon Used by many projects Systematic testing Rich integration of components EGEE is part of this – exploit test bedcontribute components Thanks to Miron Livny Collaboration is a two way street – or should be! NA3 Induction Course Development, Edinburgh, 27 April 2004 - 16

  17. VDT Timeline Highlights • Fall 2001: VDT started by GriPhyN and iVDGL • Supported US-CMS testbed in 2002 • March 2002: VDT support system inaugurated • Early 2003: Adopted by European Data Grid & LHC Computing Grid • April 2003: VDT Testers group started • Fall 2003:Supporting Grid3 • Fall 2003: Adopted by Particle Physics Data Grid • Nov 2003: Nightly test infrastructure deployed Thanks to Miron Livny NA3 Induction Course Development, Edinburgh, 27 April 2004 - 17

  18. VDT Growth VDT 1.1.8 First real use by LCG VDT 1.1.11 Grid2003 VDT 1.0 Globus 2.0b Condor 6.3.1 VDT 1.1.7 Switch to Globus 2.2 VDT 1.1.3, 1.1.4 & 1.1.5 pre-SC 2002 Thanks to Miron Livny NA3 Induction Course Development, Edinburgh, 27 April 2004 - 18

  19. Condor Group Condor/Condor-G DAGMan Fault Tolerant Shell ClassAds Globus Alliance Job submission (GRAM) Information service (MDS) Data transfer (GridFTP) Replica Location (RLS) EDG & LCG Make Gridmap Certificate Revocation List Updater Glue Schema/Info prov. ISI & UC Chimera & Pegasus NCSA MyProxy GSI OpenSSH UberFTP LBL PyGlobus Netlogger Caltech MonaLisa VDT VDT System Profiler Configuration software Others KX509 (U. Mich.) Tools in the VDT 1.1.13 Thanks to Miron Livny NA3 Induction Course Development, Edinburgh, 27 April 2004 - 19

  20. Relative Importance • What envelopes you put your messages in • How they are delivered • Infrastructure to organise a common technical platform – the foundations of communication NA3 Induction Course Development, Edinburgh, 27 April 2004 - 20

  21. Relative Importance • What envelopes you put your messages in • How they are delivered • Infrastructure to organise a common technical platform – the foundations of communication • What information you send in your messages • Their patterns of Use - sequences that mean something • Their Contents • The Grammar and Vocabulary of Communication • Agreed Interpretations NA3 Induction Course Development, Edinburgh, 27 April 2004 - 21

  22. Relative Importance Technical Experts • What envelopes you put your messages in • How they are delivered • Infrastructure to organise a common technical platform – the foundations of communication • What information you send in your messages • Their patterns of Use - sequences that mean something • Their Contents • The Grammar and Vocabulary of Communication • Agreed Interpretations • What you do when you get a message • The Application Code you Execute • The Middleware Services • Security, Privacy, Authorisation, Accounting, Registries, Brokers, … • Integration Services • Multi-site Hierarchical Scheduling, Data Access & Integration, … • Portals, Workflow Systems, Virtual Data, Semantic Grids • Tools to support Application Developers, Users & Operations • Incremental deployment tools, diagnostic aids, performance monitoring, … NA3 Induction Course Development, Edinburgh, 27 April 2004 - 22

  23. Relative Importance Domain Experts • What envelopes you put your messages in • How they are delivered • Infrastructure to organise a common technical platform – the foundations of communication • What information you send in your messages • Their patterns of Use - sequences that mean something • Their Contents • The Grammar and Vocabulary of Communication • Agreed Interpretations • What you do when you get a message • The Application Code you Execute • The Middleware Services • Security, Privacy, Authorisation, Accounting, Registries, Brokers, … • Integration Services • Multi-site Hierarchical Scheduling, Data Access & Integration, … • Portals, Workflow Systems, Virtual Data, Semantic Grids • Tools to support Application Developers, Users & Operations • Creative Actions and Judgements of Researchers, Designers & Clinicians • Data, Models & Analyses • In Silico Experiments, Design, Diagnosis & Planning • Creating the Scientific Record NA3 Induction Course Development, Edinburgh, 27 April 2004 - 23

  24. Conclusions - Strategy • If you are making long-term plans • Plan to use WSRF • If you develop or research middleware • Engage with groups developing WSRF • If you need those or related functions • E.g. to notify or handle state without incremental resource loss • If you run distributed Grid operations • Plan to use WSRF • But only when components using it are robust • Incremental transition is possible – even necessary NA3 Induction Course Development, Edinburgh, 27 April 2004 - 24

  25. Conclusions - Tactics • Value your team’s skills & momentum • Change only if necessary • If you’re an applications researcher • Stay with what you have working • WS-I +GSI; GT2, VDT, LCG2, GT3, … • If you’re a computing researcher • If your platform serves your investigation stay on it • WS-I +GSI; GT2, VDT, LCG2, GT3, … • If you’re doing middleware R&D • Hard choices & frustrating times – keep going • Those who understand the new order will reap advantage • Therefore engage with WSRF • WSRF principles and design patterns • Are a useful guide to building distributed infrastructure • Adopt them, even while WSRF details are being agreed • To ease transition NA3 Induction Course Development, Edinburgh, 27 April 2004 - 25

  26. Conclusions - Final • WSRF • Good enough for recurrent platform requirements • Has significant commercial and technical momentum • Improves engagement with industry • Only sensible flag to rally behind • Must collaborate internationally • Scale of challenge & international virtual organisations • Discourage localised alternatives • Avoid effort fragmentation and unnecessary arguments • Coping well with transitions … Is a primary Darwinian selector! NA3 Induction Course Development, Edinburgh, 27 April 2004 - 26

More Related