Practical on special jobs and WMProxy

1. Practical on special jobsand WMProxy GILDA TEAM INFN EELA tutorial Madrid 24 February 2006

2. Outline of exercises • MPI on gLite • DAG job • Job Collection • Parametric jobs Madrid, EELATutorial, 24 February 2006

3. MPI exercise Create the following “mpiglite.jdl” file on your home. [ Type = "Job"; JobType = "MPICH"; Executable = “cpi"; NodeNumber = 2; StdOutput = “cpi.out"; StdError = “cpi.err"; InputSandbox = {"cpi"}; OutputSandbox = {“cip.err",“cip.out"}; Requirements = Member("GLITE-1.4", other.GlueHostApplicationSoftwareRunTimeEnvironment); RetryCount = 0; ] Madrid, EELATutorial, 24 February 2006

4. MPI submission [glite-tutor] /home/vardizzo > glite-job-submit -o id mpi_glite.jdl Selected Virtual Organisation name (from proxy certificate extension): gilda Connecting to host glite-rb.ct.infn.it, port 7772 Logging to host glite-rb.ct.infn.it, port 9002 ========== glite-job-submit Success ========================== The job has been successfully submitted to the Network Server. Use glite-job-status command to check job current status. Your job identifier is: - https://glite-rb.ct.infn.it:9000/bsrbbzbcXZWSzU3iUYlm6g The job identifier has been saved in the following file: /home/vardizzo/id ========================================================== Madrid, EELATutorial, 24 February 2006

5. MPI status and output Query the status of the job using the following command: [glite-tutor] /home/vardizzo > glite-job-status -i id ……………………………………………. When the status of the job is “DONE”, you can retrieve output with the following command: [glite-tutor] /home/vardizzo > glite-job-output -i id …………………………………………… Madrid, EELATutorial, 24 February 2006

6. WMProxy : submission & monitoring • In order to submit job with WMProxy, it’s mandatatory credentials delegation • The submission/monitoring commands are slightly different, but the most of the “old” options are supported glite-wms-job-delegate-proxy -d del_ID_01 glite-wms-job-submit -d del_ID_01 collection.jdl glite-wms-job-output \ https://glite-rb.ct.infn.it:9000/LHIIGaCVdl7Olm sz0jpI_g Madrid, EELATutorial, 24 February 2006

7. DAG job nodeA nodeB nodeC NodeF nodeD • DAG job is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other ones • Dependencies are represented through Directed Acyclic Graphs, where the nodes are graphs, and the edges identify the dependencies Madrid, EELATutorial, 24 February 2006

8. DAG jdl [ type = "dag"; max_nodes_running = 4; nodes = [ nodeA = [ file ="nodes/nodeA.jdl" ; ]; nodeB = [ file ="nodes/nodeB.jdl" ; ]; nodeC = [ file ="nodes/nodeC.jdl" ; ]; nodeF = [ file ="nodes/nodeF.jdl"; ]; dependencies = { {nodeA, nodeB}, {nodeA, nodeC}, { {nodeB,nodeC}, nodeF } } ]; ; ] Node description could be done also here, instead of using separate file Madrid, EELATutorial, 24 February 2006

9. Job Collection • Job collection is a set of independent jobs that user wants to submit and monitor as a single request • Jobs of a collection are submitted as DAG nodes, without dependencies • JDL is a list of classad, which describes the subjobs [ Type = "collection"; VirtualOrganisation = “gilda"; nodes = { [ <job descr 1 >], [ <job descr 2 >], … }; ] Madrid, EELATutorial, 24 February 2006

10. gridftp basic use • Upload TO a gridftp server • Download FROM a gridftp server globus-url-copy file:<local absolute path> gsiftp://<gsiftp server>/<remote absolute path> Ex: globus-url-copy file:/home/madrid01/file.txt gsiftp://glite-rb.ct.infn.it/tmp/file.txt globus-url-copy gsiftp://<gsiftp server>/<remote absolute path> file:<local absolute path> Ex: globus-url-copy gsiftp://glite-rb.ct.infn.it/tmp/file.txt file:/home/madrid01/file.txt Madrid, EELATutorial, 24 February 2006

11. Job collection examples [ Type = "collection"; InputSandbox = {"start_hostname.sh"}; RetryCount = 0; nodes={ [ Executable = "/bin/sh"; StdOutput = "host.out"; StdError = "host.err"; InputSandbox = root.InputSandbox; OutputSandbox = {"host.err","host.out"}; OutputSandboxURI={"gsiftp://glite-rb.ct.infn.it:2811/tmp/host.out", "host.err"}; Arguments = "start_hostname.sh"; ],[ Executable = "/bin/sh"; StdOutput = "test.out"; StdError = "test.err"; InputSandbox={"starter.sh","gsiftp://glite-rb.ct.infn.it:2811/tmp/t01.txt"}; OutputSandbox = {"test.err","test.out"}; Arguments = "starter.sh"; ],[ file = "hostname.jdl"; ] }; ] Madrid, EELATutorial, 24 February 2006

12. Parametric Job • A parametric job is a job where one or more of its attributes are parametrized • Value of attributes vary according to parameter • Job monitoring / managing is always done through an unique jobID, as if the job was single (see submission of collection [ JobType = "Parametric"; Executable = “/bin/echo"; Arguments = “PARAM”; #InputSandbox = “input_PARAM_.txt"; StdOutput = "myoutput_PARAM_.txt"; StdError = "myerror_PARAM_.txt"; Parameters = 2500; ParameterStep = 100; ParameterStart = 1000; OutputSandbox = {“myoutput_PARAM_.txt”}; ] Madrid, EELATutorial, 24 February 2006

13. Parametric job / 2 • Parameter can be also a list of string • InputSandbox (if present) has to be coherent with parameters [ui-test] /home/giorgio/param>cat param2.jdl [ JobType = "Parametric"; Executable = “/bin/cat"; Arguments = “input_PARAM_.txt”; InputSandbox = "input_PARAM_.txt"; StdOutput = "myoutput_PARAM_.txt"; StdError = "myerror_PARAM_.txt"; Parameters = {earth,moon,mars}; OutputSandbox = {“myoutput_PARAM_.txt”}; ] [ui-test] /home/giorgio/param > ls inputEARTH.txt inputMARS.txt inputMOON.txt param2.jdl Madrid, EELATutorial, 24 February 2006

14. References • JDL attributes specification for WM proxy • https://edms.cern.ch/document/590869/1 • WMProxy quickstart • http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wmproxy_client_quickstart.shtml • WMS user guides • https://edms.cern.ch/document/572489/1 Madrid, EELATutorial, 24 February 2006