WMPROXY usage

1. WMPROXY usage Álvaro Fernández IFIC (CSIC)

2. Contents • Introduction to WMPROXY • Architecture of WMPROXY inside WMS • Features and JDL attributes • APIs: Issues and future work

3. Introduction • gLite idea was to move towards a Service Oriented Architecture (SOA) • Imply services communicate through the exchange of messages • Moving to WS-* interfaces • Still missing a real standard. Many WS-* specifications • Activity inside OGF-GIN • WMS started moving the access services: • (“old”) Network server receives user requests from UI, using an own protocol and interface. Communication from UI based on provided cli and apis • WMProxy service is web based and provides an standard interface to manage jobs.

4. WMPROXY features • Workload Manager Proxy (WMProxy) • Developed by Datamat and included in the WMS • Web Service Interface • Simplicity/Extensibility • Accessibility: • greater community of users • Integration: • Eases interoperability among different middlewares • Stepping towards Interoperability • SOA conformance: WSDL service description • WS-I compliance: support for multi-language, multi-platform clients • Features • Provide Access to gLite Workload Management System • Serve a large number of requests • Improve performances • Improve usability • Provide new functionalities Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim 23 2007

5. WMProxy: Architecture(1/4) • Service Container: • Apache + GridSite + FastCGI • SOAP tooling • Stub generation is performed through gSOAP (C++) • Security: • SSL Authentication • DN / FQAN-based Authorization • Operations: • Job Submission and Control services implementation (C++) • Gridsite Delegation service implementation (C++) Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim 23 2007

6. WMProxy: Architecture (2/4) • gSOAP layer: • intermediate layer between gSOAP and the WMProxy core • Directory Manager: • Job reserved area Creation/Management on the server file system • LB Access: • interaction with LB components • job registration • logging for job events • job information queries • Request Delivery: • deliver request to Workload Management Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim 23 2007

7. MOD FastCGI MOD SSL Logging & Bookkeeping Client SOAP/ HTTPS LB Data Base Job Sub/Contr. Workload Manager Gridsite Delegation WMProxy: Architecture (3/4) WMProxy integration with the WMS WMProxy Server Apache LB Proxy Request Queue Local File System Server Host Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim 23 2007

8. WMProxy modules Authorization WMProxy Operations Request Delivery Directory Manager Client gSOAP Layer Delegation LB Access WMProxy: Architecture (4/4) gSOAP Independant WMProxy Server Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim 23 2007

9. Usage: Command Line Interface • Interface is described in the WSDL • Command line interface based on the that one to provide access: • glite-wms-job-submit submits a job to a WMProxy Service. It requires a JDL file as input and returns a WMS job identifier. • glite-wms-delegate-proxy allows the user to delegate her proxy credential to the WMProxy service. • This delegated credential can then be used for job submissions. • glite-wms-job-list-match lists the identifiers of jobs submitted to a WMProxy Service by the user issuing the command. • glite-wms-job-cancel cancels one or more jobs previously submitted to WMProxy Service. • glite-wms-job-output Retrieve output files of a job, when finished. After this operation the job itself is purged and no more operation can be done on it • glite-wms-job-perusal Manage the perusal functionality for a certain job such us: • Enabling perusal for one (or more) file(s) - Allowing perusal file retrieval - Disabling perusal functionality • glite-wms-job-info (new in 3.1): Allow retrieving useful information about the user delegated proxy, the delegated identification details of the user, the jdl of a previously submitted job, etc

10. Usage Features • Credential delegation (glite-wms-delegate-proxy) is used to make the security framework lighter: • Time consuming proxy delegation is done once per session • Job submission uses the delegation id to send jobs • Still can be delegated with each job-submission (as before) – not recommended • Types of Requests: • JOB, DAG, COLLECTION • Types of supported JOB: • NORMAL, INTERACTIVE, MPICH, PARTITIONABLE, CHECHKPOINTABLE, PARAMETRIC • Bulk job submission: enables submission of multiple jobs at the same time • Reduces time of submission • Single call to WMProxy server • Single Authentication and Authorization process • Sharing of files between jobs • Job’s Sandboxes: share common files • Asynchronous Job Start

11. Features: JDL Attributes • Inclusion of pre- and post- execution scripts: • PROLOGUE, EPILOGUE • PROLOGUEARGUMENTS, EPILOGUEARGUMENTS • Compression and expansion of input sandbox files: • ZippedISB = “BossArchive_1_2_1.tgz”; • AllowZippedISB = true;(overrides ZippedIsb with the input sandbox files) • Define how much time should the job be maintained in queue for execution: • ExpiryTime = 1112339655; • Job Perusal: obtain partial results of the output files before job finishes: • PerusalFileEnable = true; • PerusalTimeInterval = 10; • PerusalFilesDestURI = “gsiftp://ghemon.cnaf.infn.it/home/glite/peek”; (not mandatory, could use the glite-wms-job-perusal command to retrieve files) Examples Taken from: TEGEE-JRA1-TEC-590869-JDL-Attributes-v0-8T

12. Data Requirements JDL Attributes • Sandbox files can be stored at a https/gridftp server: • InputSandboxBaseURI = "gsiftp://matrix.datamat.it:5432/tmp"; • or multiple servers: • OutputSandbox = { "myjobOutput","run1/event1","myjobError"}; • OutputSandboxDestURI = { "gsiftp://matrix.datamat.it:5432/tmp/myjobOutput ", "gsiftp://grid003.ct.infn.it:6789/home/cms/event1", "myjobError“}; • DataRequirements contains 3 attributes: • InputData: List of InputData = { “lfn:/EO.test.file” , “lds:cms.test.file”, “guid:135b7b23-4a6a-11d7-87e7-9d101f8c8b70”, “query:select_my_DC1_files” }; • DataCatalogType (“rls”, “si” storage-index, “dli” data location interface) • DataCatalog (addess of catalog - DataCatalog = http://data.example.org/CMSDataSetCatalog ); • Matchmaking to find a CE close to the SE: • OutputSE = “grid001.cnaf.infn.it";

13. Data Requirements Example DataRequirements = { [ DataCatalogType = "DLI"; DataCatalog = "https://cms.org:8877/dli"; InputData = {"lfn:/my/test/data1", "guid:44rr44rr77hh77kkaa3", "lds:my.test.dataset", "query:my_query"}; ], [ DataCatalogType = "SI"; DataCatalog = "https://glite.org:9443/StorageIndex"; InputData = {"lfn:/eo/test.file", "guid:ddffrg5451"}; ], [ DataCatalogType = "RLS"; DataCatalog = "https://eu-datagrid.org/RLS"; InputData = {"lfn:/atlas/test.file", "guid:ggrgrg5656"}; ], [ DataCatalogType = "RLS"; InputData = {"lfn:/myvo/test.file", "guid:adbdefgilm1234"}; ], .... };

14. Other Features • Shallow resubmision: when submission fails due to a mw failure, and not application fail. • ShallowRetryCount = 3; • Deep resubmision: RetryCount = 0; • shallow retry count is reset to zero each time a deep resubmission of the job is done. • Collections: set of independent jobs • Attributes like a DAG, but no Dependencies among nodes set. nodes = { [n1], [n2], …};

15. API Usage • Command line interface • API: provided wsdl, and generated bindings: • Org.glite.wms.interface • Org.glite.wms.wmproxy-api-cpp • Org.glite.wms.wmproxy-api-java • Org.glite.wms.wmproxy-api-python • … • API functionality • delegating the credential ; • registering and submitting jobs ; • cancelling the job during its life-cycle ; • retrieving information on the location where the job input sandbox files can be stored ; • retrieving the output sandbox files list ; • retrieving a list of possible matching Computer Elements ; • getting JDL templates ; • getting information on the user disk quota on the server • Credential delegation is done by external packages. Only Java include bindings to perform it from the api. (C++ and python users should do it explicitely) • In case of file transfer needed, job should be previously registered: • jobIds = client.jobRegister(jdlString, delegationId)

16. API Usage (cont) File transfers from UI <-> WMS should be done also explicitely in the code: • JAVA: In order to allow grid user to invoke, within his Java code, the UrlCopy class to copy files from the WMProxy Server to the UI and vice versa the following Java package is requested cog-jglobus-1.4-bin.tar.gz The above package with all the needed documentation can be downloaded from this web page http://dev.globus.org/wiki/CoG_JGlobus_1.4 • New in glite3.1: getDelegatedProxyInfo(java.lang.String delegationId)           Returns the Delegated Proxy information identified by the delegationId string getJobProxyInfo(java.lang.String jobId)           Returns the information related to the proxy used to submit a job that identified by its JobId. getJDL(java.lang.String jobId, org.glite.wms.wmproxy.JdlType type)           Returns the JDL string which identifier is the input JobId

17. Issues • Performance was degrading when submitting collections, bulk submission. Solved changing hanling from DAGman to native methods. • API: Changing implementation of soap python implementation from SOAPpy to ZSI (see Patch#843) • Python bindings: Developers said Delegation functionality that is needed is only provided by other package (org.glite.data.delegation-api-c) implemented in C.

18. Thanks! • For more info: http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=Main.HomePage