Services for Science

1. Services for Science Ian Foster Computation Institute Argonne National Lab & University of Chicago

2. Thanks! • DOE Office of Science • NSF Office of Cyberinfrastructure • National Institutes of Health • Colleagues at Argonne, U.Chicago, USC/ISI, OSU, Manchester, and elsewhere

3. ScientificCommunication, ~1600 Brahe Kepler

5. User Discovery tools Scientific Communication, ~2000 Analysis tools Service-Oriented Science Gateway Data Archives Figure: S. G. Djorgovski

6. Globus Services& Cancer Biology caBIG: sharing of infrastructure, applications, and data. Data Integration!

7. Services& Environmental Science RaffaeleMontella,U. NapoliParthenope

9. Service-Oriented Science People create services (data, code, instr.) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

10. Creating Services People create services (data, code, instr.) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

11. Anatomy of a Service Management Clients Clients Registry Attribute Authority Attribute Authority … (meta)data op1 opN Implementation(s) Service Persistence Service

12. State Interfaces  Get resource property  Get multiple RPs  Set RP state  Query a set of RPs  Subscribe: notifications  Manage termination time  Immediate destruction Resource Resource property Exposing Service State State representation • Resource, resource property State identification • Endpoint Reference Service GetRP GetMultRPs EPR EPR EPR SetRP QueryRPs Subscribe SetTermTime Destroy

13. Apache Tomcat Service Container Service Service Service GetRP GetRP GetRP GetMultRPs GetMultRPs GetMultRPs EPR EPR EPR EPR EPR EPR EPR EPR EPR SetRP SetRP SetRP Resource Resource Resource QueryRPs QueryRPs QueryRPs RPs RPs RPs Subscribe Subscribe Subscribe SetTermTime SetTermTime SetTermTime ResourceHome ResourceHome ResourceHome Destroy Destroy Destroy Globus Globus ToolkitWeb Services Container Security Authorization PIP PDP State Management Persistence WorkManager DB Conn Pool JNDI Directory Globus Toolkit Version 4: Software for Service-Oriented Systems, LNCS 3779, 2-13, 2005

14. Creating Services (~2005) “This full-daytutorial provides an introduction to programming Java services with the latest version of the Globus Toolkit version 4 (GT4). The tutorial teaches how to build a Java Service that makes use of GT4 mechanisms for state management, security, registry and related topics.”

15. Appln Service Create Store Advertize Discover Transfer GAR Invoke; get results Deploy Globus Creating Services in 2008Introduce and gRAVI • Introduce • Define service • Create skeleton • Discover types • Add operations • Configure security • Grid Remote Application Virtualization Infrastructure • Wrap executables Introduce Repository Service Index service Container Argonne/U.Chicago & Ohio State University

16. Creating ServicesIntroduce + gRAVI Shannon Hastings Scott Oster David Ervin Stephen Langella Kyle Chard Ravi Madduri

17. Discovering Services People create services (data or functions) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

18. B A Discovering Services Assume success Semantics Permissions Reputation  The ultimate arbiter?  Types, ontologies  Can I use it?  Billions of services

19. Globus Discovery (1):Registries

20. Globus Discovery (2):Standardized Vocabularies Core Services IndexService Queries Service Enterprise Vocabulary Services Cancer DataStandardsRepository Metadata Aggregated In Registers To Uses Terminology Described In References Subscribes to ObjectsDefined in and Aggregates Grid Discovery Service Client API Publishes Service Metadata

22. Discovery (3): Tagging& Social Networking GLOSS: Generalized Labels Over Scientific data Sources (Foster, Nestorov)

23. Discovery (3): Tagging& Social Networking David de Roure, Carole Goble, et al.

24. Composing Services People create services (data or functions) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

25. Globus Composing Services

26. Composing ServicesTaverna + GT4 Taverna team Wei Tan Ravi Madduri

27. Publishing Services People create services (data or functions) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

28. Publishing Services Description  Syntax, semantics State  Availability, load, … Policies  Who, what, when, … Hosting  Location, scalability, …

29. A B 1 1 10 10 1 A B 1 2 1 2 16 Defining Community: Membership and Laws • Identify VO participants and roles • For people and services • Specify and control actions of members • Empower members  delegation • Enforce restrictions  federate policy Effective Access Policy of site to community Access granted by community to user Site admission-control policies

30. Globus Authorization: SAML & XACML PERMIS VOMS Shibboleth LDAP … SAML Authorization Decision Attributes PIP PIP PIP PDP GT4 Client GT4 Server XACML

31. Hosting Services People create services (data or functions) … which I discover (& decide whether to use) … & compose to create a new function ... & then publish as a new service.  I find “someone else” to host services, so I don’t have to become an expert in operatingservices & computers!  I hope that this “someone else” can manage security, reliability, scalability, … ! ! “Service-Oriented Science”, Science, 2005

32. The Importance of “Hosting”and “Management” Tell me about this star Tell me about these 20K stars Support 1000sof users E.g., Sloan DigitalSky Survey, ~10 TB; others much bigger

33. Users Discovery tools Analysis tools Data Archives Fig: S. G. Djorgovski The Two Dimensions of Service-Oriented Science • Decompose across network Clients integrate dynamically • Select & compose services • Select “best of breed” providers • Publish result as new services • Decouple resource & service providers Function Resource

34. + + + + + + + = Globus Dynamic Provisioning:Falkon Applied to Stacking • Purpose • On-demand “stacks” of random locations within ~10TB dataset • Challenge • Rapid access to 10-10K “random” files • Time-varying load • Solution • Dynamic acquisition of compute, storage Sloan Data S4 Web page or Web Service Joint work with Ioan Raicu & Alex Szalay

35. Dynamic Provisioning with Falkon:Release after 15 Seconds Idle

36. Dynamic Provisioning with Falkon: Release after 180 Seconds Idle

37. Globus Building Scalable Service Implementations Functional MRI Ben Clifford, Mihael Hatigan, Mike Wilde, Yong Zhao

38. AIRSN Program Definition (Run or) reorientRun (Run ir, string direction) { foreach Volume iv, i in ir.v { or.v[i] = reorient(iv, direction); } } • (Run snr) functional ( Run r, NormAnat a, Air shrink ) { • Run yroRun = reorientRun( r , "y" ); • Run roRun = reorientRun( yroRun , "x" ); • Volume std = roRun[0]; • Run rndr = random_select( roRun, 0.1 ); • AirVector rndAirVec = align_linearRun( rndr, std, 12, 1000, 1000, "81 3 3" ); • Run reslicedRndr = resliceRun( rndr, rndAirVec, "o", "k" ); • Volume meanRand = softmean( reslicedRndr, "y", "null" ); • Air mnQAAir = alignlinear( a.nHires, meanRand, 6, 1000, 4, "81 3 3" ); • Warp boldNormWarp = combinewarp( shrink, a.aWarp, mnQAAir ); • Run nr = reslice_warp_run( boldNormWarp, roRun ); • Volume meanAll = strictmean( nr, "y", "null" ) • Volume boldMask = binarize( meanAll, "y" ); • snr = gsmoothRun( nr, boldMask, "6 6 6" ); • }

39. Scheduling Provisioning Execution Engine (Karajan w/ Swift Runtime) FalkonResource Provisioner Virtual Node(s) Virtual Node(s) file1 Swift runtimecallouts launcher C C C AppF1 C Provenance data file2 launcher Status reporting AppF2 Amazon EC2 file3 Provenance data Provenance collector Globus Dynamic Provisioning:Swift Architecture Specification Execution Abstract computation SwiftScript Compiler Virtual Data Catalog SwiftScript Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford, et al.

40. Services for Science • They’re new • A new approach to communicating • A (not-so new) approach to structuring systems • They’re real • Excellent infrastructure and tools (Globus, Introduce, gRAVI, Taverna, Swift, etc.) • Substantial numbers of services out there • They’re challenging • Sociology: incentives, rewards • Infrastructure: hosting • Provenance: justifying “results” • Scaling: services, requests