Grid programming basics

1. Grid programming basics Meteo VO Training, Belgrade 24-25. June 2008 Branko Marovic RCUB - UoB The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

2. Software/Job Management Software installation management using ESM tools • Experiment Software Installation in LCG-2 • http://goc.grid.sinica.edu.tw/gocwiki/Experiments_Software_Installation • SEEGRID Wiki “SEE-GRID ESM Software Installation Guide” • http://wiki.egee-see.org/index.php/SEE-GRID_ESM_Software_Installation_Guide • Site configuration • Using ESM tools • Example JDL and shell scripts for installation, validation, and removal Job management • EDG WMS API (LCG-2) – C++ & Java • http://www.to.infn.it/grid/workload_management/apiDoc/edg-wms-api-index.html • Exec/spawn of shell scripts that create and submit JDL – simple, but no direct feedback on jobs • gLite • http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/glite-wms-api-index.shtml Application Gridification 2/27

3. WMProxy Java API ver 3.1.0 • Overview of the WMProxy Java API • Software Requirements • Class org.glite.wms.wmproxy.WMProxyAPI • An use case Application Gridification 3/27

4. Workload Management System • Helps the user accessing computing resources • resource brokering • management of input and output • management of complex workflows • Support for MPI job even if the file system is not shared between CE and Worker Nodes (WN) – easy JDL extensions • Web Service interface via WMProxy Application Gridification 4/27

5. WMProxy WMProxy is a SOAP Web service providing access to the Workload Management System (WMS) Job characteristics specified via JDL jobRegister create id map to local user and create job dir register to L&B return id to user input files transfer jobStart register sub-jobs to L&B map to local user and create sub-job dirs unpack sub-job files deliver jobs to WM Application Gridification 5/27

6. The WMProxy The WMProxy is the service responsible to provide access to the WMS functionality through a Web Service Interface • The gLite WMProxy Server can be either accessed directly through the published WSDL, the C++ command line interface, or the API • In these two links you can find a guide about how to interact with the main services exposed by the WMProxy through the provided API Java and C++ APIhttps://grid.ct.infn.it/twiki/bin/view/GILDA/ApiJavaWMProxyhttps://grid.ct.infn.it/twiki/bin/view/GILDA/WMProxyCPPAPI Application Gridification 6/27

7. org.glite.wms.wmproxy-api-java • The main Java class of this API is: org.glite.wms.wmproxy.WMProxyAPI; • A client object can be created using one of these following constructor(s): public WMProxyAPI(String url, String proxyFile) public WMProxyAPI(String url, String proxyFile, String CAcerts) public WMProxyAPI(String url, InputStream proxyStream) public WMProxyAPI(String url, InputStream proxyStream, String CAcerts) Application Gridification 13/27

8. org.glite.wms.wmproxy-api-java where: • url: the WMProxy server URL to be contacted (e.g. https://<host>:<port>/glite_wms_wmproxy_server); • proxyFile: the pathname to a valid user proxy; for the default value (/tmp/x509up_u<UID>) • CAcerts: the path of a local CA directory • proxyStream: a valid proxy passed as input stream; Application Gridification 14/27

9. org.glite.wms.wmproxy-api-java Therefore, using one of the four constructors WMProxyAPI Client = new WMProxyAPI( … ); a new instance is created. This instance will be created to invoke any other WMProxy services. For example: WMProxyAPI client = new WMProxyAPI( “https://trinity.datamat.it:7443/glite_wms_wmproxy_server”, “/x509up_u504”); proxy = client.grstGetProxyReq (delegationID); client.grstPutProxy(delegationID, proxy); jobIDs = client.jobSubmit(jdlString, delegationID); Application Gridification 15/27

10. Basic Requirements • The JDL to be matched must contains some mandatory Requirements and Rank attributes; • when using command line interface these are added by default by the UI, but in our case user has to take care to check if his/her JDL file contains them: Requirements = (other.GlueCEStateStatus == "Production"); Rank = (-other.GlueCEStateEstimatedResponseTime); Application Gridification 17/27

11. Authorization • The client must be properly authorized when interacts with the WMProxy service. • This means that either the FQAN or the DN (in case of globus-style proxies) of the client must be properly listed and authorized in the glite_wms_wmproxy.gacl file on the WMProxy machine. [root@glite-rb2 etc]# cat glite_wms_wmproxy.gacl <gacl version='0.0.1'> <entry> <voms><fqan>bio/Role=NULL</fqan></voms> <allow><exec/></allow> </entry> </gacl> Application Gridification 18/27

12. Delegation of user credential • Before calling submission or list match services, client proxy credentials are requested to be transferred from the client to the server through a delegation process. • The following Java code performs this delegation process : String delegationId = “larocca”; WMProxyAPI client = new WMProxyAPI wmproxy_url,user_proxy,CAcertsPath); /*Get a proxy identified by the delegationID String*/ proxy = client.grstGetProxyReq (delegationId); /* Allows delegating user’s credential to the WMProxy */ client.grstPutProxy(delegationId, proxy); The obtained delegation identifier can be reused for several calls to the WMProxy server; Application Gridification 19/27

13. Difference between submission and registration /1 • Submission of simple jobs, DAGs, collections and parametric jobs to the WMProxy service requires as input a job description file in which job characteristics and requirements are expressed by means of Condor class-ad-like expressions. • In this description the users can specify some files that are needed by the submitted jobs during the execution on the remote Computing Elements. These files are listed in the JDL InputSandbox attribute specifying for each of them the complete URI of its location. • If the job does NOT have any file in the InputSandbox to be transferred from the submitting machine to the WMProxy node, the submission can be performed also calling the jobSubmit service: jobIds = client.jobSubmit(jdlString, delegationId); Application Gridification 20/27

14. Difference between submission and registration /2 Otherwise, these following steps are needed: • a preliminary server registration: jobId = client.jobRegister(jdlString, delegationId); • transfer of files in the InputSandbox from the client machine to the WMS node; • call the jobStart service to trigger the submission: client.jobStart(jobId); Application Gridification 21/27

15. Submit a new request to WMProxy String jobID_String = (String) jobID.getId(); InputSandboxURI = client.getSandboxDestURI(jobID_String, "gsiftp"); //Convert listURI into a String. String[] reduced_path = InputSandboxURI.getItem(); // Creation of the "toURL" link to copy the file(s). int pos = (reduced_path[0]).indexOf("2811"); int length = (reduced_path[0]).length(); String front = (reduced_path[0]).substring(0 , pos); String rear = (reduced_path[0]).substring(pos + 4 , length); for (int index = 0; index < InputSandboxFiles.length; index++) { String toURL = front + “2811” + rear; toURL = toURL + "/" + InputSandboxFiles[index]; fromURL = "file:///" + InputSandboxFiles[index]; try { GlobusURL from = new GlobusURL(fromURL); GlobusURL to = new GlobusURL(toURL); UrlCopy uCopy = new UrlCopy(); uCopy.setDestinationUrl(to); uCopy.setSourceUrl(from); uCopy.setUseThirdPartyCopy(true); uCopy.copy(); } catch (Exception e) {System.err.println(e.getMessage());} } UrlCopy class performs the copy of the InputSandbox files to the WMS Specify the Destination and Source URL(s) Copy file(s) from the UI to the Resource Broker Application Gridification 25/27

16. Retrieve output file(s) //Use the "gsiftp" transfer protocols to retrieve the list of files produced by the jobs. result = client.getOutputFileList(jobId, "gsiftp"); if ( result != null ) { // list of files+their size list = (StringAndLongType[ ]) result.getFile(); size = list.length; if (list != null) { for (int i=0; i<size ; i++){ // Creation of the "fromURL" link from where download the file(s). int pos = (list[i].getName()).indexOf("2811"); String front = (list[i].getName()).substring(0 , pos); String rear = (list[i].getName()).substring(pos + 4 , (list[i].getName()).length()); String fromURL = front + "2811/" + rear; int pos_output = (list[i].getName()).indexOf("output"); String filename = (list[i].getName()).substring(pos_output + 7 , length); // Creation of the "toURL" link where to download the file(s). toURL = toURL = "file:///" + dirPath + filename; try { // Retrieve the file(s) from the WMProxy Server. GlobusURL from = new GlobusURL(fromURL); GlobusURL to = new GlobusURL(toURL); UrlCopy uCopy = new UrlCopy(); uCopy.setDestinationUrl(to); uCopy.setSourceUrl(from); uCopy.setUseThirdPartyCopy(true); uCopy.copy(); } catch (Exception e) {System.err.println(e.getMessage());} } } else System.out.println ("No output files for this job!"); } else System.out.println ("An empty list has been received"); // Purge output file(s) from the WMProxy Server. client.jobPurge(jobId); Application Gridification 27/27

17. References • API Documentation http://trinity.datamat.it/projects/EGEE/wiki/apidoc/3.1/htmljava/index.html • Datamat – WMProxy quickstart http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=WMProxyClient.QuickStart • JDL Attributes guide for WMProxy https://edms.cern.ch/document/590869/1 • WMProxy user guide https://edms.cern.ch/document/674643/1 Application Gridification 28/27