A Swift Talk about Globus Technology: What Can It Do for Me?

1. A Swift Talk about Globus Technology: What Can It Do for Me? OOI Cyberinfrastructure Design Meeting, San Diego, 17-19 October The Globus Team (presented by Kate Keahey)

2. Overview • Globus Toolkit overview • What is WSRF and how does it work? • Defining community: membership and laws • Workflow management with Swift • Globus as project: how we work Virtual Workspaces: http://workspace.globus.org

3. What Is the Globus Toolkit? • The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications • Heterogeneity • To date (v1.0 - v4.0), the Toolkit has focused on simplifying heterogeneity for application developers • We are increasingly including more “vertical solutions” that implement typical application patterns • Security • The Grid Security Infrastructure (GSI) allows collaborators to share resources without blind trust • Standards • Our goal has been to capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF) • The Toolkit also includes reference implementations of new/proposed standards in these organizations Virtual Workspaces: http://workspace.globus.org

4. What’s In the Globus Toolkit? • A Grid development environment • Develop new OGSA-compliant Web Services • Develop applications using Java or C/C++ Grid APIs • Secure applications using basic security mechanisms • A set of basic Grid services • Job submission/management • File transfer (individual, queued) • Database access • Data management (replication, metadata) • Monitoring/Indexing system information • Tools and Examples • The prerequisites for many Grid community tools Virtual Workspaces: http://workspace.globus.org

5. Globus Technology Areas • Core runtime • Infrastructure for building new services • WSRF = Web Service Resource Framework • Security • Apply uniform policy across distinct systems • Execution management • Provision, deploy, & manage services • Data management • Discover, transfer, & access large data • Monitoring • Discover & monitor dynamic services Virtual Workspaces: http://workspace.globus.org

6. Globus Projects GlobusToolkit OGSA-DAI MPICH G2 Java Runtime MyProxy Data Rep Replica Location Delegation GridWay CAS C Runtime GSI- OpenSSH GridFTP MDS4 Incubation Mgmt Python Runtime C Sec GRAM Reliable File Transfer GT4 Docs Incubator Projects Swift GEMLCA MonMan GAARDS MEDICUS Cog WF Virt WkSp NetLogger GDTE GridShib OGRO UGP Dyn Acct Gavia JSC DDM Metrics Introduce PURSE HOC-SA LRMA WEEP Gavia MS SGGC ServMark Common Runtime Security Execution Mgmt Data Mgmt Info Services Other Virtual Workspaces: http://workspace.globus.org

7. Web Services Resource Framework (WSRF)

8. “Stateless” vs. “Stateful” Services FileTransferService Client move (A to B) move • Without state, how does client: • Determine what happened (success/failure)? • Find out how many files completed? • Receive updates when interesting events arise? • Terminate a request? • Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state Virtual Workspaces: http://workspace.globus.org

9. FileTransferService (without WSRF) FileTransferService Client move (A to B) : transferID move whatHappen state tellMeWhen cancel • Developer reinvents wheel for each new service • Custom management and identification of state: transferID • Custom operations to inspect state synchronously (whatHappen) and asynchronously (tellMeWhen) • Custom lifetime operation (cancel) Virtual Workspaces: http://workspace.globus.org

10. Service State representation Resource Resource Property State identification Endpoint Reference State Interfaces GetRP, QueryRPs, GetMultipleRPs, SetRP Lifetime Interfaces SetTerminationTime ImmediateDestruction Notification Interfaces Subscribe Notify ServiceGroups Resource RPs WSRF in a Nutshell Service GetRP GetMultRPs EPR EPR EPR SetRP QueryRPs Subscribe SetTermTime Destroy Virtual Workspaces: http://workspace.globus.org

11. Transfer RPs FileTransferService (w/ WSRF) FileTransferService Client createResource createResource (A to B) : EPR getRP queryRPs destroy • Developer specifies custom method to createResource and leaves the rest to WSRF standards: • State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR) • State inspected by standard interfaces (GetRP, QueryRPs) • Lifetime management by standard interfaces (Destroy) Virtual Workspaces: http://workspace.globus.org

12. The Introduce Authoring Tool • Define service • Create skeleton • Discover types • Add operations • Configure security • Modify service Generates GT4-compatible WebServices Introduce: Hastings, Saltz, et al., Ohio State University Virtual Workspaces: http://workspace.globus.org

13. A B 1 1 10 10 1 A B 1 2 1 2 16 Defining Community: Membership and Laws • Identify VO participants and roles • For people and services • Specify and control actions of members • Empower members  delegation • Enforce restrictions  federate policy Effective Access Policy of site to community Access granted by community to user Site admission-control policies Virtual Workspaces: http://workspace.globus.org

14. SAML XACML Courtsey : DOE report : LBNL-41349 : Authorization & Attribute Certificates for Widely Distributed Access Control

15. Globus Authorization Framework PERMIS VOMS Shibboleth LDAP … Authorization Decision Attributes PIP PIP PIP PDP GT4 Client GT4 Server Virtual Workspaces: http://workspace.globus.org

16. Swift System • Clean separation of logical/physical concerns • XDTM specification of logical data structures + Concise specification of parallel programs • SwiftScript, with iteration, etc. + Efficient execution on distributed resources • Karajan threading, Falkon provisioning, Globus interfaces, pipelining, load balancing + Rigorous provenance tracking and query • Virtual data schema & automated recording  Improved usability and productivity • Demonstrated in numerous applications Virtual Workspaces: http://workspace.globus.org

17. Workflow Language - SwiftScript • Goal: Natural feel to expressing distributed applications • Variables (basic, data structures) • Conditional operators (if, foreach, ) • Functions (atomic / compound) • Used to connect outputs to inputs • It does not specify invocation order, only dependencies • It can be seen as a metadata for expressing experiments Virtual Workspaces: http://workspace.globus.org

18. Execution Engine • Karajan engine (event-based execution) • Has a scheduler to map tasks to resources • Score-based planning • Recovers from failures (retries) • Falkon resource manager creates a “private cluster” • Uses Globus GRAM4 (PBS/Condor/Fork) to acquire resources from Grid systems Virtual Workspaces: http://workspace.globus.org

19. Scheduling Provisioning Execution Engine (Karajan w/ Swift Runtime) Resource Provisioners Falkon/VWS Virtual Node(s) Virtual Node(s) file1 Swift runtimecallouts launcher C C C AppF1 C Provenance data file2 launcher Status reporting AppF2 Amazon EC2 file3 Provenance data Provenance collector Dynamic Provisioning:Swift Architecture Specification Execution Abstract computation SwiftScript Compiler Virtual Data Catalog SwiftScript Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford

20. Globus Philosophy • Globus was first established as an open source project in 1996 • The Globus Toolkit is open source to: • Allow for inspection • for consideration in standardization processes • Encourage adoption • in pursuit of ubiquity and interoperability • Encourage contributions • harness the expertise of the community • The Globus Toolkit is distributed under the (BSD-style) Apache License version 2 Virtual Workspaces: http://workspace.globus.org

21. dev.globus • Governance model based on Apache Jakarta • Consensus based decision making • Globus software is organized as several dozen “Globus Projects” • Each project has its own “Committers” responsible for their products • Cross-project coordination through shared interactions and committers meetings • A “Globus Management Committee” • Overall guidance and conflict resolution Virtual Workspaces: http://workspace.globus.org

22. http://dev.globus.org Guidelines(Apache Jakarta) Infrastructure(CVS, email,bugzilla, Wiki) Projects Include … Virtual Workspaces: http://workspace.globus.org

23. Open Source != “Free time” • Globus development is well-funded • The open source model facilitates contributions • NSF and DOE sponsor Globus development at several institutions via multiple grants, totaling >$5M/yr • Non-U.S. science agencies also contribute to Globus development • Corporations also sponsor developers • NSF explicitly funds Globus improvements • CDIGS: Community-Driven Improvements to Globus Software Virtual Workspaces: http://workspace.globus.org

24. Parting Thoughts • There is a wealth of available Grid technology that can be leveraged for this project • Security mechanisms, workflow, and other services • The Globus Toolkit is funded by NSF to support scientific communities. Virtual Workspaces: http://workspace.globus.org