Luidmila Stepanova Dubna, Jul 4, 2008

L. Stepanova WMProxy Python API tests Luidmila Stepanova Dubna, Jul 4, 2008

L. Stepanova Introduction • The WMProxy (Workload Manager Proxy) is service providing access to the WMS (Workload Management System) functionality through a Web Services based interface. • A set of API allows calling the WMProxy server. The main Python class of these APIs is called Wmproxy: it is stored inside wmproxymethods module and it allows calling all WMProxy services. • A client object can be created using the following constructor: • Wmproxy(url, ns=““, proxy=““) url=https://<host>:<port>/glite_wms_wmproxy_server Default ns is http://www.gridsite.org/namespaces/delegation-1 Default proxy is /tmp/X509up_u<UID> port is 7443

L. Stepanova How to start the test • The test contains several sub tests in directory test. • The test can be run by executing ./start_tests.sh WMSLB_HOST [-output filename] subtests listed in the file test-sequence.lst will be executed. • The test can be run by executing ./start_tests.py WMSLB_HOST [filename] The start_test.py is based on the Python's ConfigParser module. The module provides a parser for simple configuration WMProxy_API.conf file, where: if the subtest is defined as TRUE then subtest will be executed. The WMSLB_HOST is obligatory commandline argument . If the option <filename> is given the output should be directed to the file <filename>

L. Stepanova Script getversion • The subtest getversion calling client.getVersion and if version 2.2.0 and later client.getTransferProtocols methods. • getVersion - gets the version of the service. • getTransferProtocol - gets transfer protocols

L. Stepanova Example of getversion output • ######## script name is getversion ######## • time is 2008-05-13 11:09:01.820620 • | OK | getVersion • | OK | getTransferProtocol • #################################### • ====== script name is getversion ======== • WmProxy_host: grinr05.inr.troitsk.ru • ========== OK protocol version = 2.2.0=========== • ======= OK get available Tranfer Protocols: ======== • gsiftp https • **************** test getversion is OK *******************

L. Stepanova Credentials Delegation Sequence Diagram glite-proxy-cert

L. Stepanova Script delegation • This subtest use the method getProxyReq for get proxy request from the server. Proxy request signing by command: • glite-proxy-cert -e DELEGATION_REQUEST • were • -e gets the proxy request from machine environment DELEGATION_REQUEST. • The method putProxy finishes the delegation procedure by sending the signed proxy certificate to the server. • Following Wmproxy Methods are included in the test: • getVersion - gets the version of the service. • getGrstNs - gets GridSite Delegation name space string representation • proxyRequest - gets proxy request from the server. • putProxy - finishes the delegation procedure by sending the signed proxy certificate to the server. • getDelegatedProxyInfo - this operation returns the Delegated Proxy information.

L. Stepanova Examples of delegation output • ############### script name is delegation ############### • time is 2008-05-13 11:09:03.756187 • | OK | getVersion • | OK | getGrstNs • | OK | proxyRequest • | OK | putProxy • | OK | getDelegatedProxyInfo • ######################################################## • ============ script name is deligation =========== • WmProxy_host: grinr05.inr.troitsk.ru • ========== OK protocol version = 2.2.0=========== • =========== Delegation of user credential ======== • OK GridSite Delegation namespace string representation: http://www.gridsite.org/namespaces/delegation-1

L. Stepanova Examples of delegation output • ===============proxyRequest=============== • -----BEGIN CERTIFICATE REQUEST----- • MIHIMHQCADAQMQ4wDAYDVQQKEwVEdW1teTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgC • QQCyK4+iYJJTedfRzsLEkqKyxaV50cZphQXjYOiGyo4JtCDlXgvLLI7ZR/tqANR2 • qOAxNBFJbsGpIYxOsksUSw7dAgMBAAGgADANBgkqhkiG9w0BAQQFAANBAJz+3E2d • hNzdN1bZMl30xJKS+tDo9PFRwRec/XnTER5DWRwHEKC1KtpovUCsG/6TcbMa1EP/dEDG+Q+qhbkO9zs= • -----END CERTIFICATE REQUEST----- • ==== Path of X509_USER_PROXY is /tmp/x509up_u3088 ==== • glite-proxy-cert -e DELEGATION_REQUEST • ==== client.putProxy(delegationId, proxyRequest) is OK ==== • ************ test deligation is OK *****************

L. Stepanova Script jobstart_submit • The subtest jobstart_submit is the Single Job Submission. The sequence of WMProxy methods necessary for the delivery of the user's job to the WM queue is used. • Following WMProxy Methods are included in the test: • 1. the delegation procedure • getVersion • getGrstNs • proxyRequest • putProxy • getDelegatedProxyInfo • 2. jobListMatch -This method provides the list of CEs matching the requirements of the job specified in the JDL. • 3. the job submit procedure • jobSubmit - This operation submits a job. The JDL description of the job provided by the client is validated by the service, registered to the LB and finally passed to the Workload Manage.The unique identifier assigned to the job is returned to the client. • 4. the job start procedure • jobRegister - This operation registers a job for submission • getJobProxyInfo - Retrieve useful information about the user delegated proxy, the delegated identification details of the user, the jdl. • getACLItems - This operations gets a list of items from the job Access Control List.

L. Stepanova Script jobstart_submit • getSandboxDestURI -This operation returns a destination URI associated to the job, identified by the jobId provided as input, where the job input sandbox files can be uploaded by the client on the WMS node. • getSandboxBulkDestURI -This operation returns the list of destination URIs associated to a compound job. In the given test it is equivalent to the getSandboxDestURI. • getFreeQuota - This operation returns the remaining free part of available user disk quota (in bytes). • getMaxInputSandboxSize -This operation returns the maximum Input sandbox size (in bytes) a user can count-on for a job submission if using the space managed by the WM. • jobStart -Trigger the submission a previously registered job. • jobCancel -This operation cancels a previously submitted job identified by its JobId.

L. Stepanova JDL file for Single Job Submission • JDL file is created by the test. Executable = "/bin/hostname"; Arguments = ""; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"}; VirtualOrganisation = "dteam"; requirements = true; rank = 1;

L. Stepanova Examples of jobstart_submit output • ################ script name is jobstart_submit ############### • time is 2008-06-25 17:25:10.508137 • | OK | getVersion • | OK | getGrstNs • | OK | proxyRequest • | OK | putProxy • | OK | getDelegatedProxyInfo • | OK | jobListMatch • | OK | jobRegister • | OK | getJobProxyInfo • | OK | getACLItems • | OK | getSandboxDestURI • | OK | getSandboxBulkDestURI • | OK | client.getFreeQuota • | OK | client.getMaxInputSandboxSize • | OK | jobStart https://cg08.ific.uv.es:9000/8K5ltbPSFH7Sapkejt-3-g • | OK | jobCancel https://cg08.ific.uv.es:9000/8K5ltbPSFH7Sapkejt-3-g • | OK | jobSubmit https://cg08.ific.uv.es:9000/VCtGqNn9IkhODAPfTNNE5w • | OK | jobCancel https://cg08.ific.uv.es:9000/VCtGqNn9IkhODAPfTNNE5w

L. Stepanova Examples of jobstart_submit output • ======script name is jobstart_submit =============== • ============ WmProxy_host: cg08.ific.uv.es=========== • ========== protocol version = 2.2.0=========== • ============ Delegation of user credential is OK ======== • GridSite Delegation namespace string representation :http://www.gridsite.org/namespaces/delegation-1 • ============ proxyRequest is OK =============== • Path of X509_USER_PROXY is /tmp/x509up_u500 • glite-proxy-cert -e DELEGATION_REQUEST • ==== client.putProxy(delegationId, proxyRequest) is OK ==== • ======= VirtualOrganisation =dteam =========== • Count of CE Ids satisfying the job Requirements specified in the JDL: • listSize=26

L. Stepanova Scripts jobcllection and jobdag • This subtests differ in JDL attributes • A Direct Acyclic Graph (DAG) is a set of jobs (sub-nodes) where the input, output, or execution of one or more jobs depends on one or more other jobs. The jobs are graph nodes (vertices), the edges (arcs) identify the dependencies.A Job Collection is a group of jobs without dependencies. • Delegation in these tests is executed by means ofthe command: • glite-wms-job-delegate-proxy -ddelegationId --endpoint urlwere-d delegationId - the proxy that will be delegated is identified by <id_string>. • --endpoint url - the operations are performed contacting the WMProxy service represented by the given serviceURL.

L. Stepanova Wmproxy Methods are included in the jobcol and jobdag scripts Methods for Job DAG and Job Collection: getSandboxBulkDestURI -returns the list of destination URIs associated to a compound job. This feature is particularly useful when submitting compound requests: DAG, Collections. getChildren -returns the list of all sons (as a list of JobIdStruct)getJobID -returns the JobId string representation getNodeName -returns, if the istance represents a dag or collection node, the name of the node. Next methods described earlier:getDelegatedProxyInfo jobRegister getJobProxyInfo jobSubmit jobCancel

L. Stepanova JDL file for DAG Job Submission [ Type = "dag";Executable = "/bin/hostname";VirtualOrganisation = "dteam";Requirements = true;Rank = other.GlueCEPolicyMaxRunningJobs - \ other.GlueCEStateRunningJobs;FuzzyRank = true;nodes= [nodeA = [Description = [Executable = "/bin/hostname" ;StdOutput = "jobA.out";StdError = "jobA.err";OutputSandbox = {"jobA.err", "jobA.out" };];];nodeB = [Description = [Executable ="/bin/hostname";];];dependencies = { { nodeA, nodeB}};];]

L. Stepanova JDL file for Collection Job Submission [ Type = "collection";Executable = "/bin/hostname";VirtualOrganisation = "dteam";Requirements = other.GlueCEUniqueid == \ "grce001.inr.troitsk.ru:2119/jobmanager-lcgpbs-dteam";Rank = other.GlueCEPolicyMaxRunningJobs - \ other.GlueCEStateRunningJobs;Nodes = { [JobType = "normal";Executable = "/bin/hostname" ;Arguments = "";StdOutput = "job1.out";StdError = "job1.err";OutputSandbox = {"job1.err", "job1.out" };],[JobType = "normal";Executable = "/bin/hostname";Arguments = "";StdOutput = "job2.out";StdError = "job2.err";OutputSandbox = {"job2.err", "job2.out" };] };]

L. Stepanova Examples of jobdag output ############## script name is jobdag ####### time is 2008-06-25 17:45:58.944178 | OK | getDelegatedProxyInfo | OK | jobRegister | OK | getJobProxyInfo | OK | jobSubmit https://cg08.ific.uv.es:9000/EfGNzHNRuzTis5vKr4q9rQ | OK | getSandboxBulkDestURI | OK | getChildren | OK | getNodeName = nodeB | OK | getNodeName = nodeA | OK | jobCancel https://cg08.ific.uv.es:9000/EfGNzHNRuzTis5vKr4q9rQ########################################====== script name is jobdag========= =====WmProxy_host: cg08.ific.uv.es===== =======VirtualOrganisation =dteam=====

L. Stepanova Examples of jobdag output • ==the list of destination URIs associated to a compound job=== https://cg08.ific.uv.es:9000/z64WSVXjl97TG_hwhAsszggsiftp://cg08.ific.uv.es:2811/var/glite/SandboxDir/z6/https_3a_2f_2fcg08.ific.uv.es_3a9000_2fz64WSVXjl97TG_5fhwhAsszg/inputhttps://cg08.ific.uv.es:7443/SandboxDir/z6/https_3a_2f_2fcg08.ific.uv.es_3a9000_2fz64WSVXjl97TG_5fhwhAsszg/inputhttps://cg08.ific.uv.es:9000/f-X9wumIHqmxJCb8--lCoQgsiftp://cg08.ific.uv.es:2811/var/glite/SandboxDir/f-/https_3a_2f_2fcg08.ific.uv.es_3a9000_2ff-X9wumIHqmxJCb8--lCoQ/inputhttps://cg08.ific.uv.es:7443/SandboxDir/f-/https_3a_2f_2fcg08.ific.uv.es_3a9000_2ff-X9wumIHqmxJCb8--lCoQ/inputhttps://cg08.ific.uv.es:9000/dA-MMnBPzCkHDY0Qn65VZggsiftp://cg08.ific.uv.es:2811/var/glite/SandboxDir/dA/https_3a_2f_2fcg08.ific.uv.es_3a9000_2fdA-MMnBPzCkHDY0Qn65VZg/inputhttps://cg08.ific.uv.es:7443/SandboxDir/dA/https_3a_2f_2fcg08.ific.uv.es_3a9000_2fdA-MMnBPzCkHDY0Qn65VZg/input

L. Stepanova Examples of jobdag output • =========task.getChildren======NodeJobId = https://cg08.ific.uv.es:9000/tzoZKNuozlrvmUEaqi8j5QNodeName = nodeBNodeJobId = https://cg08.ific.uv.es:9000/d06-FTMNcpgs5MeCkLOkYANodeName = nodeA OK: Job https://cg08.ific.uv.es:9000/EfGNzHNRuzTis5vKr4q9rQ was cancel****** test jobdag is OK ******

L. Stepanova Examples of jobcol output • ####### script name is jobcollection ####### time is 2008-06-25 17:49:40.724095 | OK | getDelegatedProxyInfo | OK | jobRegister | OK | getJobProxyInfo | OK | jobSubmit https://cg08.ific.uv.es:9000/uQuVCwr1SjOCA_XALHynpQ | OK | getSandboxBulkDestURI | OK | getChildren | OK | getNodeName = Node_0 | OK | getNodeName = Node_1 | OK | jobCancel https://cg08.ific.uv.es:9000/uQuVCwr1SjOCA_XALHynpQ#################################======= script name is jobcollection ==============WmProxy_host: cg08.ific.uv.es======= =======VirtualOrganisation =dteam=====

L. Stepanova Examples of jobcol output • ==== task.getChildren ====NodeJobId = https://cg08.ific.uv.es:9000/q4RfnL5MiugmCQUueoFZnA NodeName = Node_0NodeJobId = https://cg08.ific.uv.es:9000/aiQqrFj4b1sXYCS3PtWyoA NodeName = Node_1 OK: Job https://cg08.ific.uv.es:9000/uQuVCwr1SjOCA_XALHynpQ was cancel***** test jobdcollection is OK ******

L. Stepanova Further information • See also: https://twiki.cern.ch/twiki/bin/view/LCG/SAMTest#WMProxy_Python_API http://glite.cvs.cern.ch/cgi-bin/glite.cgi/org.glite.testsuites.ctb/WMS-api/python

Luidmila Stepanova Dubna, Jul 4, 2008