gUSE Services Remote API, DCI Bridge, Data Bridge, Robot Certificate

1. gUSE ServicesRemote API, DCI Bridge, Data Bridge, Robot Certificate Zoltán Farkas, Péter Kacsuk, Gábor Herman, István Márton, Tibor Gottdank, Ákos Balaskó MTA SZTAKI LPDS zoltan.farkas@sztaki.mta.hu

2. Outline • Remote API • DCI Bridge • Data Bridge • Robot Certificate

3. REMOTE API

4. Goals - Solutions • Submitting a WS-PGRADE/gUSE workflow using an HTTP client without the web interface of WS-PGRADE • Solution: The Remote APIweb service extension of gUSE will be used as application layer in order to communicate with the backend (gUSE) instead of WS-PGRADE portlets • The workflows to be submit should be available at the caller • As WS-PGRADE workflow definitions contain own set of input files, reference to input files must be changed, but the files of job executable, and the command line parameters can be changed as well Solution: Manual change of files to be replaced and the creation of newassociation table (coded in the „portmapping.txt” file). The mainworkflowdescriptor file <workflow>.xml should be modified only in the case whencommand line parameters belonging to the individual nodes(jobs) of theworkflow must be modified • On completion of a workflow submission session the common resources should be cleared • Solution: The servers side data belonging to the workflow (submission) will be cleared upon the successful execution of the „Get output” command of the client

5. Typical workflow development scenario in case of remote call WS PGRADE SERVER 2 Copy (modify) gUSE gUSE Eventual update Copy and rename 4. Submit the modified workflow 2. Download tested workflow 3. Reengineering of wf definition Remote API Serv. 1.Create and submit original workflow Client side Server side DCI Bridge HTTP Client GRAPH Ed Browser DCI Bridge Portal Portal <wf>.zip WS PGRADE SERVER 1 <wf>.xml <wf>.xml <file_i> <file_i> <file_i> <inputs>.zip Upd. user files Orig user files portmapping.txt tree of <wf> files Set of computational resources

6. Summary of workflow development and remote submission process • Download the original workflow from WS-PGRADE • Do some reengineering: • separate the structure description from the other parts • explore the user defined (not channel) input files connected to thenumbered ports of the named jobs and the files of executable of thenamed jobs • Pack the needed files together and create a text file which describes the association of the files and the named jobs and ports • Create the needed script files needed to habitual submission process (call, observe, download (delete)) • Execute the scripts on the client machine

7. Basic assumptions of remote call setup There is a working gUSE set of services where the Remote API Servlet extension has been installed There is a client machine containing the description (structure, input files) of a workflow The client can reach the server by the HTTP protocol There is a server-created password known by the client(s) The client has the necessary proxy certificate file if the submissions involved in the workflow are directed to such resources which need certificate bound authorization *(see installation file and documentation on Sourceforge for version <V> :Code: http://sourceforge.net/projects/guse/files/<V>/remote-3.4.tgz/download Documentation: http://sourceforge.net/projects/guse/files/<V>/Documentation/RemoteAPI_Install_Manual.pdf/download )

8. DCI Bridge

9. What is DCI Bridge - 1 A web service based application that provides standard accessto various distributed computing infrastructures (DCIs), such as: grids, desktop grids,clusters, clouds and service based computational resources (connecting through itsDCI plugins to the external DCI resources). Supported DCIs

10. What is DCI Bridge - 2 When a user submits a workflow, its jobcomponents can be submitted transparently into the various DCI systems using theOGSA Basic Execution Service 1.0 (BES) interface. As a result, the access protocoland all the technical details of the various DCI systems are totally hidden behind theBES interface. The standardized job description language of BES is JSDL. From gUSE or independently (from other workflow systems) can also use DCI Bridge for job submission Generic View

11. Administration Interface – 1 A sample view of Middleware settings page. http://LOCLAHOST:8080/dci_bridge_service/conf

12. Administration Interface – 2 Base Menus Middleware Menus

13. Architecture plugins Plugins WFI generates and submits the jobs' JSDL to the DCI-BRIDGE. The job flow in the DCI Bridge and between the components.

14. JSDL Processing JSDL <ns3:SDL_Type> <ns3:Constraints> <ns3:Middleware> <ns3:DCIName>local</ns3:DCIName> <ns3:MyProxy/> <ns3:ManagedResource>dci-bridge host(64bit)</ns3:ManagedResource> </ns3:Middleware> <ns3:Budget>0</ns3:Budget> </ns3:Constraints> </ns3:SDL_Type> WS-PGRADE WFI DCI Bridge Admin Interface (JSP pages) BES Semi-autogenerated Eventhandlers Middleware Brokering and Management Layer (Plugin Manager) DCI Bridge Middleware Class GT2 Unicore GT4 GT5 PBS CB Local gLite ... DCI Bridge accepts standardized JSDL job description documents. These documentsare based on a well-defined XML scheme containing information about the job inputs,binaries, runtime settings and output locations.

15. 2 WFI WFI sends JSDL to DCI Bridge DCI BRIDGE DCI Bridge sends back job status to WFI. 6 PLUGIN 1 WFI requests configuration data for job submission from WFS. From response WFI creates a JSDL 3 DCI Bridge gets inputs from Storage JSDL 4 PLUGIN submits the job together with inputs to a DCI, monitors the job status and gets the outputs. DCI Bridge sends outputs to the Storage 5 WFS Storage DCI Job Processing

16. Steps of Plugin Creation - 1 • Implementing of 4 methods: • submit (invoked on job submission; performs job submission • getStatus (invoked when job status need to be queried; queries job status and sets job status accordingly • abort (invoked when the job need to be aborted; aborts the execution of the job) • getOutputs(invoked when the outputs of the job need to be downloaded; downloads the outputs to a local folder) 1. Developing middleware specific parts

17. Steps of Plugin Creation – 2 NewMiddleware ….. <xsd:simpleType name="DCINameEnumeration"> <xsd:restriction base="xsd:string"> <xsd:enumeration value="glite"/> <xsd:enumeration value="arc"/> <xsd:enumeration value="unicore"/> <xsd:enumeration value="boinc"/> <xsd:enumeration value="gemlca"/> <xsd:enumeration value="pbs"/> <xsd:enumeration value="lsf"/> <xsd:enumeration value="gae"/> <xsd:enumeration value="service"/> <xsd:enumeration value="local"/> <xsd:enumeration value=”newmid"/> </xsd:restriction> </xsd:simpleType> 2.Developing configuration interface on DCI Bridge 2.1 Adding new middleware name in mb_scheduling_description_language.xsd (other fields relevant for resource (VO, DCI) selection are generic)

18. Steps of Plugin Creation - 3 2.2 Extending configuration schema with middleware-specific configuration possibilities in dci-bridge_configuration_schema_2012.xsd <xsd:element name="unicore" minOccurs="0" maxOccurs="1"> <xsd:complexType> <xsd:sequence> <xsd:element name="keystore"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="keypass"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="keyalias"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="subjectdn"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="truststore"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element> <xsd:element name="trustpass"> <xsd:simpleType> <xsd:restriction base="xsd:string"/> </xsd:simpleType> </xsd:element></xsd:sequence> </xsd:complexType> </xsd:element> 2.3 Extending existing classes that process tab/menu selection logic 2.4 Creating JSPs

19. Steps of Plugin Creation - 4 Creating new class (JobConfigUI_newmid) Implementing the getJsp and the getJobParameters methods. WS-PGRADE Newmiddleware Modification in WFI: Modify the JobConfig class: Adding the new middlewares name in mbsdlMiddleware() method(it maps the job's configuration to the middlewares configured in DCI-Bridge). 3.Developing WS-PGRADE and WFI specificparts

20. DATA BRIDGE

21. Outline Problem statement Data Bridge as independent DCI service: Data Bridge concept Use-cases Data Bridge architecture WS-PGRADE integration Data browsing portlet gUSE integration

22. Problem statement Scientific applications: Individual jobs or workflows Access data from diverse sources Science Gateways can hide the details, but… Data sources: Diverse types: HTTP, FTP, GridFTP, SRM, iRODS, … Thus, different APIs are needed to access these One possible solution is to use a service that can be used to access the sources through a unified interface

23. Data Bridge Offers a simple service that provides a generic interface above different DCI's storage services to handle the data stored The service in different use cases offers a way to browse, upload and download data, and with the help of multiple server instances it enables inter-DCI data transfer as well

24. Use cases Use case 1: Browse a single DCI data storage from WS-PGRADE, upload data Use case 2: Transfer data files between different DCIs Use case 3: Fetch input data on a DCI worker node from an other DCI Use case 4: Cloud storage usage

25. Use case 1: Storage browsing and data upload WS-PGRADE Browse and upload Storage Browsing Portlet Data Bridge Adaptor Interface Storage Adaptor Storage

26. Use case 2: Data Transfer – Using multi-level Data Bridge • Client: • Storage Browsing Portlet • Custom application • … Data Bridge Adaptor Interface Storage Adaptor1 Data Bridge Adaptor Data Bridge Adaptor Interface Storage Adaptor2 Storage1 Storage2

27. Use case 3: Fetch data on a DCI’s worker node from a „foreign” DCI’s storage Data bridge usage guidelines: First try to fetch the data using native tools Only if this fails, use the Data Bridge DCI Worker node Data Bridge Wrapper Pre-process Adaptor Interface Executable Storage Adaptor Storage Post-process

28. Use case 4: Cloud Storage access from WS-PGRADE/gUSE Currently, no S3 support in WS-PGRADE An S3 Data Bridge adaptor would fix this WS-PGRADE/gUSE DCI Worker node Job Amazon S3 Data Bridge

29. Data Bridge Architecture Public Interface HTTP servlet Adaptor Manager Temporary URL queue Worker Pool URI URI URI Thread1 Thread2 Threadn Adaptor Interface DCI Adaptor1 DCI Adaptor2 DCI Adaptor3 DCI Adaptorm jSAGA

30. Data Bridge components Interfaces: Public Interface Adaptor Interface Adaptor Manager Worker Threads DCI Adaptors

31. Data Bridge components- Interfaces Public Interface: Provides the public interface for external components (Portlets, gUSE, …) Web Service interface Adaptor Interface: A Java interface that hides the details of the different adaptors

32. Data Bridge Public Interface Operations: List Mkdir Delete Get Put Copy Move Entities: URI (either a path, an URL or some specific class) Error reports: Common exceptions

33. Data Bridge Public Interface - URI Represents an element with a given URI (a directory, a file, metadata attributes, …) Also needs to carry security credentials (if needed) Attributes: Nothing special in the base class For gLite, e.g: Path: the full path Type: directory or file Size: length of the entity (0 for directories) Attributes: optional, contains information as returned by the Adaptor Interface's Stat function

34. Data Bridge Public Interface – Get and Put Two-phase up- and download with the temporary URL queue: First, the web service interface is invoked to register the transfer request Next, a simple HTTP client may use HTTP GET or POST/PUT to down- or upload the data This way, web service invocation („heavyweight” SOAP) is separated from data transfer („lightweight” HTTP) Public Interface HTTP servlet Adaptor Manager Temporary URL queue Worker Pool URI URI URI Thread1 Thread2 Threadn Adaptor Interface DCI Adaptor1 DCI Adaptor2 DCI Adaptor3 DCI Adaptorm

35. Adaptor Manager and Worker threads Provided by JAX-WS web service API Tasks: Manage incoming requests Initialize worker threads to perform the requested operation With the help of different adaptors

36. DCI Adaptors Implement: Adaptor Interface Tasks: Perform operations requested by the Worker Threads, that is operations invoked through the web service Types: gLite (using jSAGA) GridFTP (using jSAGA) FTP (using jSAGA) … Data Bridge: special adaptor to forward requests to other Data Bridges

37. Data Bridge clients Web Service clients: Create your own based on the WSDL (or REST) Java API: Provides a convenient tool to use Data Bridge Public Interface functions Data transfer functions should accept InputStream and OutputStream objects as their arguments

38. WS-PGRADE integration A Data Browsing portlet that eases storage management

39. WS-PGRADE Workflow I/O configuration During a workflow node's IO configuration the user should be able to select files from storages The provided interface should be the same as the selected storage's Storage Browsing portlet (only with one panel)

40. Current status, future work Core Data Bridge (available as a web service) ready, working with most major protocols (FTP, GridFTP, SRM) User Interface development has been started, first version will be available as part of WS-PGRADE/gUSE shortly

41. ROBOT CERTIFICATE

42. The concept of robot certificates 42 The normal certificate is used to identify users The robot certificate is used to identify applications As a consequence the application should be trusted When the CA provides the certificate for the application, the certificate contains the identifier of the person or organization that validated the application and takes the responsibility for it It is the policy of the user community and the CA to decide whose name should be in the certificate

43. EGI VO Portal Policy I. The Portal, the VO to which the Portal is associated, the Portal manager are all individually and collectively responsible and accountable for all interactions with the Grid, unless a credential of a Strongly Identified Web User is used to interact with the Grid The Portal must be capable of limiting the job submission rate The Portal must keep audit logs for all interactions with the Grid (https://documents.egi.eu/document/81)

44. EGI VO Portal Policy II. Portal classes (in fact, these are working mode classes, i.e. the same portal can be in parameter mode from the point of view of a certain user and in the same time can be in job management mode from the point of view of another user): SZTAKI gUSE

45. EGI VO Portal Policy III. Robot certificates can be used only for the first 3 working modes of the portal Job management mode portals/applications must not use robot certificates

46. SCI-BUS portals and EGI 46 • According to the EGI classification : • WS-PGRADE/gUSE is a portal that can be used in 1,2,3,4 modes • The community portals could also work in any of the 4 modes depending of the needs of the corresponding user community • Robot certificates are needed only for the 1,2,3 modes

47. Relationship between robot certificates and WF applications in WS-PGRADE/gUSE 47 • The WF applications can have different robot certificates • Even the jobs within a workflow can have different robot certificates (this enables that different jobs of a WF can be executed in different DCIs requiring different robot certificates) • This robot certificate contains the name of the community who set up the gateway • Example: • Autodock gateway set up by SZTAKI and UoW • The robot certificate will contain the name: SCI-BUS

48. WS-PGRADE/gUSE extensions to support robot certificates 48 • The robot certificates should be hidden for the end-users but manageable for the portal developer/administrator • The WF applications with robot certificates will be stored in the internal repository of gUSE • Consequences: • The internal repository should be extended to be able to store the identification of the robot certificates for every node • The portal developer/administrator (with a new privileged role to be introduced besides the power and end-user roles) should be able to assign the robot certificates for the WF nodes

49. Suggested process to assign robot certificates to WF nodes 49 • When a WF is tested and ready to use in the community portal the next step is to assign robot certificates to the nodes of the WF • This will happen in the following way: • Portal developer imports the WF from the internal repository • Use the WF configuration facility of WS-PGRADE (this will be extended to enable the definition of robot certificates, see next slide) • Test the workflow with the assigned robot certificates • Export the WF with assigned robot certificates into the internal repository • Notice that some nodes of the WF can work with robot certificates while other nodes require user certificate. Therefore even during the execution of a WF the portal can change among working modes.

50. UI extension to assign robot certificates to WF nodes + line (robot cert opt) + checkbox + auto ModWin 50