1 / 30

Ejecuci ón con MPI e interactividad en el Grid Ejercicios Pr ácticos

Ejecuci ón con MPI e interactividad en el Grid Ejercicios Pr ácticos. Download the examples. wget http: / /devel.ifca.es/ ~enol / mpi-examples.tgz tar xzf mpi-examples.tgz. Finding where to run (I). Sites publish in their tags: MPI-START (if they have mpi -start installed)

nan
Download Presentation

Ejecuci ón con MPI e interactividad en el Grid Ejercicios Pr ácticos

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ejecución conMPI e interactividad en el GridEjercicios Prácticos

  2. Download the examples • wgethttp://devel.ifca.es/~enol/mpi-examples.tgz • tar xzfmpi-examples.tgz

  3. Finding where to run (I) Sites publish in their tags: • MPI-START (if they have mpi-start installed) • Then for each MPI flavour supported: • <MPI flavour> • <MPI flavour>-<MPI version> • <MPI flavour>-<MPI version>-<Compiler> • And interconnect info (optional and mostly absent): • MPI-<interconnect> • Interconnects: Ethernet, Infiniband, SCI, Myrinet

  4. Finding where to run (II) • $ lcg-info–vobiomed--list-ce --query 'Tag=MPICH2’ • - CE: ce-02.roma3.infn.it:8443/cream-pbs-grid • - CE: ce-grid.obspm.fr:2119/jobmanager-pbs-biomed • ….. • $ lcg-info --vobiomed --list-ce --query 'Tag=OPENMPI-1.4.4' • CE: ce01.kallisto.hellasgrid.gr:2119/jobmanager-pbs-biomed • CE: cream01.kallisto.hellasgrid.gr:8443/cream-pbs-biomed • CE: egeece01.ifca.es:2119/jobmanager-sge-biomed • CE: egeece02.ifca.es:2119/jobmanager-sge-biomed • CE: egeece03.ifca.es:2119/jobmanager-sge-biomed • CE: gridce01.ifca.es:8443/cream-sge-biomed • CE: gridce02.ifca.es:8443/cream-sge-biomed • CE: ngiescream.i3m.upv.es:8443/cream-pbs-biomed

  5. Finding where to run (III) • $ lcg-info --vobiomed --list-ce --query 'Tag=MPI-INFINIBAND' • - CE: ce.reef.man.poznan.pl:2119/jobmanager-pbs-biomed • - CE: ce002.ipp.acad.bg:2119/jobmanager-pbs-biomed • - CE: cr1.ipp.acad.bg:8443/cream-pbs-biomed • CE: creamce.reef.man.poznan.pl:8443/cream-pbs-biomed • $ lcg-info --vobiomed --list-ce --query 'Tag=OPENMPI,Tag=MPI-START,Tag=MPI-ETHERNET' • - CE: glite.univ.kiev.ua:2119/jobmanager-pbs-grid • - CE: glite.univ.kiev.ua:8443/cream-pbs-grid • - CE: grid-lab-ce.ii.edu.mk:2119/jobmanager-pbs-biomed

  6. Finding where to run (IV) • Use the JDL requirements to match to sites supporting your needs: Requirements= Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);

  7. Paralleljob.jdl Basic example JobType = "Normal"; nodeNumber = 4; Executable = "starter.sh"; Arguments = "hello.sh OPENMPI"; InputSandbox = {"starter.sh", "hello.sh"}; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"}; Requirements = Member("MPI-START”, other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("OPENMPI”, other.GlueHostApplicationSoftwareRunTimeEnvironment);

  8. starter.sh #!/bin/bash MY_EXECUTABLE=$1 MPI_FLAVOR=$2 # Convertflavortolowercaseforpassingtompi-start. MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'` # Pull out thecorrectpathsfortherequestedflavor. evalMPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH` # Ensuretheprefixiscorrectly set. Don'trelyonthedefaults. eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH export I2G_${MPI_FLAVOR}_PREFIX # Touchtheexecutable. Itexistmustfortheshared file system check. # Ifitdoesnot, thenmpi-start may trytodistributetheexecutable # whenitshouldn't. touch $MY_EXECUTABLE chmod +x $MY_EXECUTABLE # Setupformpi-start. export I2G_MPI_APPLICATION=$MY_EXECUTABLE export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER #export I2G_MPI_APPLICATION_ARGS="./ 0 0 1" # optionalhooks #export I2G_MPI_PRE_RUN_HOOK=hooks.sh #export I2G_MPI_POST_RUN_HOOK=hooks.sh # Ifthese are set thenyouwillget more debugginginformation. #export I2G_MPI_START_VERBOSE=1 #export I2G_MPI_START_TRACE=1 #export I2G_MPI_START_DEBUG=1 # Invokempi-start. $I2G_MPI_START EGI UF. Vilnius, Apr 2011.

  9. hello.sh #!/bin/bash echo "helloworldfrom `hostname`”

  10. List-match $ glite-wms-job-list-match -a job.jdl Connectingtotheservicehttps://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server ========================================================================== COMPUTING ELEMENT IDs LIST ThefollowingCE(s) matchingyourjobrequirementshavebeenfound: *CEId* - gridce01.ifca.es:2119/jobmanager-sge-ngifor - ngiesce.i3m.upv.es:2119/jobmanager-pbs-ngies - ce2.egee.cesga.es:2119/jobmanager-lcgsge-GRID_ngifor ========================================================================== EGI UF. Vilnius, Apr 2011.

  11. Submission $ glite-wms-job-submit–ajob.jdl Connectingtotheservicehttps://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server ====================== glite-wms-job-submitSuccess ====================== Thejob has beensuccessfullysubmittedtotheWMProxy Yourjobidentifieris: https://wms01.egee.cesga.es:9000/q5BP68lwwUH4-YFzrgHBhQ ========================================================================== glite-wms-job-status https://wms01.egee.cesga.es:9000/q5BP68lwwUH4-YFzrgHBhQ ======================= glite-wms-job-status Success ===================== BOOKKEEPING INFORMATION: Status infoforthe Job : https://wms01.egee.cesga.es:9000/q5BP68lwwUH4-YFzrgHBhQ Current Status: Running Status Reason: Job successfullysubmittedtoGlobus Destination: gridce01.ifca.es:2119/jobmanager-sge-ngifor Submitted: WedApr 6 10:12:19 2011 CEST ========================================================================== EGI UF. Vilnius, Apr 2011.

  12. Output $ glite-wms-job-output --dir ./ https://wms01.egee.cesga.es:9000/q5BP68lwwUH4-YFzrgHBhQ Connectingtotheservicehttps://wms01.egee.cesga.es:7443/glite_wms_wmproxy_server ================================================================================ JOB GET OUTPUT OUTCOME Output sandbox files forthejob: https://wms01.egee.cesga.es:9000/q5BP68lwwUH4-YFzrgHBhQ havebeensuccessfullyretrievedandstored in thedirectory: /gpfs/csic_projects/grid/tmp/jobOutput/enol_q5BP68lwwUH4-YFzrgHBhQ ================================================================================ $ cat /gpfs/csic_projects/grid/tmp/jobOutput/enol_q5BP68lwwUH4-YFzrgHBhQ/* helloworldfrom gcsic142wn helloworldfrom gcsic142wn helloworldfrom gcsic142wn helloworldfrom gcsic142wn

  13. … things may go wrong • Errors happen frequently, we need a way to figure out what went wrong • Use the MPI-START debugging variables: • Basic information: • I2G_MPI_START_VERBOSE=1 • Debug information: • I2G_MPI_START_DEBUG=1 • Full trace of the execution (normally too much, but useful for mpi-start developers) • I2G_MPI_START_TRACE=1 • Set them in the starter.sh or in the JDL: Environment = {“I2G_MPI_START_VERBOSE=1”,…} EGI UF. Vilnius, Apr 2011.

  14. Debugging output: job-debug.jdl ************************************************************************ UID = ngifor080 HOST = gcsic177wn DATE = TueJul 6 12:07:15 CEST 2010 VERSION = 0.0.66 ************************************************************************ mpi-start [DEBUG ]: dumpconfiguration mpi-start [DEBUG ]: => I2G_MPI_APPLICATION=hello.sh mpi-start [DEBUG ]: => I2G_MPI_APPLICATION_ARGS= mpi-start [DEBUG ]: => I2G_MPI_TYPE=openmpi […] mpi-start [INFO ]: searchforscheduler mpi-start [DEBUG ]: source /opt/i2g/bin/../etc/mpi-start/lsf.scheduler mpi-start [DEBUG ]: checkingforschedulersupport : sge mpi-start [DEBUG ]: checkingfor $PE_HOSTFILE mpi-start [INFO ]: activatesupportforsge mpi-start [DEBUG ]: convertPE_HOSTFILEintostandardformat mpi-start [DEBUG ]: dumpmachinefile: mpi-start [DEBUG ]: => gcsic177wn.ifca.es mpi-start [DEBUG ]: => gcsic177wn.ifca.es mpi-start [DEBUG ]: => gcsic177wn.ifca.es mpi-start [DEBUG ]: => gcsic177wn.ifca.es mpi-start [DEBUG ]: startingwith 4 processes.

  15. A more real MPI job: cpi • Compile your application $ mpicccpi.c –o cpi • Change your JDL: InputSandbox= {"starter.sh", "cpi"}; Arguments= "cpi OPENMPI”; • Submit

  16. Output $ glite-wms-job-output --dir ./ https://wms01.ncg.ingrid.pt 9000/3hXhJD3eRRgiSLPqW3vSKg Connectingtotheservicehttps://wms01.ncg.ingrid.pt:7443/glite_wms_wmproxy_server ================================================================================ JOB GET OUTPUT OUTCOME Output sandbox files for the job: https://wms01.ncg.ingrid.pt:9000/3hXhJD3eRRgiSLPqW3vSKg have been successfully retrieved and stored in the directory: /home/enol/tests/eciencia/cpi/enol_3hXhJD3eRRgiSLPqW3vSKg ================================================================================ $ cat/home/enol/tests/eciencia/cpi/enol_3hXhJD3eRRgiSLPqW3vSKg/* Process 3 on gcsic093wn: n=1 Process 2 on gcsic091wn: n=1 Process 1 on gcsic091wn: n=1 Process 0 on gcsic026wn: n=1 Using 16384 intervals pi isapproximately 3.1415926539002341, Error is 0.0000000003104410 wallclock time = 0.003501

  17. Remote compilation • What if your binary does not match the site environment? • Remote compilation is needed • Mpi-start hooks can help here!

  18. Compilationusinghooks #!/bin/sh pre_run_hook () { # Compile theprogram. echo "Compiling ${I2G_MPI_APPLICATION}" # Actually compile theprogram. cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c" $cmd if [ ! $? -eq 0 ]; then echo "Error compilingprogram. Exiting..." return 1 fi # Everything's OK. echo "Successfullycompiled ${I2G_MPI_APPLICATION}" return 0 }

  19. Compilationusinghooks JobType = "Normal"; nodeNumber = 4; Executable = "starter.sh"; Arguments = ”cpi OPENMPI"; InputSandbox = {"starter.sh", “cpi.c”, “hooks.sh”}; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"}; Requirements = Member("MPI-START”, other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("OPENMPI”, other.GlueHostApplicationSoftwareRunTimeEnvironment); Environment = {“I2G_MPI_PRE_RUN_HOOK=hooks.sh”};

  20. Compilationusinghooks […] mpi-start [DEBUG ]: Trytorunprehooks at hooks.sh mpi-start [DEBUG ]: callpre_runhook -<START PRE-RUN HOOK>------------------------------------------- Compilingcpi Successfullycompiledcpi -<STOP PRE-RUN HOOK>-------------------------------------------- […] =[START]======================================================== […] piisapproximately 3.1415926539002341, Error is 0.0000000003104410 wallclock time = 0.008393 =[FINISHED]======================================================

  21. Gettingouputusinghooks #!/bin/sh exportOUTPUT_PATTERN="host_*.txt" exportOUTPUT_ARCHIVE=myout.tgz # thefirstparamteristhenameof a host in the copy_from_remote_node() { if [[ $1 == `hostname` || $1 == 'hostname -f' || $1 == "localhost" ]]; then echo "skip local host" return 1 fi CMD="scp–r $1:\"$PWD/$OUTPUT_PATTERN\" .” $CMD } post_run_hook () { echo "post_run_hookcalled" if [ "x$MPI_START_SHARED_FS" == "x0" -a "x$MPI_SHARED_HOME" != "xyes" ] ; then echo "gather output from remote hosts" mpi_start_foreach_hostcopy_from_remote_node fi echo "pack the data" tarcvzf $OUTPUT_ARCHIVE $OUTPUT_PATTERN return 0 }

  22. Gettingouputusinghooks JobType = "Normal"; nodeNumber = 4; Executable = "starter.sh"; Arguments = ”cpi2 OPENMPI"; InputSandbox = {"starter.sh", “cpi2.c”, “hooks.sh”}; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"}; Requirements = Member("MPI-START”, other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("OPENMPI”, other.GlueHostApplicationSoftwareRunTimeEnvironment); Environment = {“I2G_MPI_PRE_RUN_HOOK=hooks.sh”, “I2G_MPI_PRE_RUN_HOOK=hooks.sh”};

  23. Gettingouputusinghooks […] mpi-start [DEBUG ]: Trytorun post hooks at hooks.sh mpi-start [DEBUG ]: call post-runhook -<START POST-RUN HOOK>------------------------------------------- post_run_hookcalled pack the data host_0.txt host_1.txt host_2.txt host_3.txt -<STOP POST-RUN HOOK>--------------------------------------------

  24. ETHERNET v. INFINIBANDIMBenchmark #!/bin/sh pre_run_hook () { echo "Compilation of the code" tar xzvfimb.tar.gz cdsrc gmake ls -lrt cp IMB-MPI1 ../. cd .. } • Compare the bandwidht and latency of two different interconnects • imb.jdl Ethernet • imb-INF.jdl INFINIBAND #--------------------------------------------------- # Benchmarking PingPong # #processes = 2 # ( 6 additional processes waiting in MPI_Barrier) #--------------------------------------------------- #bytes #repetitions t[usec] Mbytes/sec 0 1000 2.36 0.00 1 1000 2.44 0.39 2 1000 2.44 0.78 4 1000 2.49 1.53 8 1000 2.74 2.78 16 1000 2.74 5.57 32 1000 2.96 10.30 #--------------------------------------------------- # Benchmarking PingPong # #processes = 2 # ( 6 additional processes waiting in MPI_Barrier) #--------------------------------------------------- #bytes #repetitions t[usec] Mbytes/sec 0 1000 23.29 0.00 1 1000 23.10 0.04 2 1000 23.21 0.08 4 1000 22.84 0.17 8 1000 22.11 0.35 16 1000 22.78 0.67 32 1000 23.44 1.30

  25. Compiling & Using the Storage Element • The Exercise SE_hooks: • Compiles the code “testprogram.tgz” • Executes the parallel MPI code on Grid resources • Copies the output to a Storage Element. • This is important when the output • is too large to be brought to the user in • the Sandbox (>50 MB) testprogram_hooks.sh #!/bin/sh export OUTPUT_PATTERN=L8 export OUTPUT_ARCHIVE=resultado.tar.gz export OUTPUT_HOST=srm01.ncg.ingrid.pt export OUTPUT_SE=lfn:/grid/biomed/tutorial-Valencia export OUTPUT_VO=biomed pre_run_hook () { echo "Compilation of the code" tar xzvftestprogram.tgz cd o3parallel make ls -lrt cp o3sg_8 ../. cd .. } DEFINITION OF ENVIRONMENT VARIABLES COMPILATION OF THE SOURCES

  26. Compiling & Using the Storage Element • Copies the output to a Storage Element. • This is important when the output • is too large to be brought to the user in • the Sandbox (>50 MB) # the first parameter is the name of a host in the copy_from_remote_node() { if [[ $1 == `hostname` || $1 == 'hostname -f' || $1 == "localhost" ]]; then echo "skip local host" return 1 fi # pack data CMD="scp -r $1:\"$PWD/$OUTPUT_PATTERN\" ." echo $CMD $CMD } post_run_hook () { echo "post_run_hook called" if [ "x$MPI_START_SHARED_FS" == "x0" ] ; then echo "gather output from remote hosts" mpi_start_foreach_hostcopy_from_remote_node fi echo "pack the data" tar cvzf $OUTPUT_ARCHIVE $OUTPUT_PATTERN echo "delete old file if it is thre" lcg-del --vo $OUTPUT_VO -a lfn:/grid/$OUTPUT_VO/$OUTPUT_ARCHIVE echo "upload the data" lcg-cr --vo $OUTPUT_VO -d $OUTPUT_HOST -l $OUTPUT_SE/$OUTPUT_ARCHIVE file://$PWD/$OUTPUT_ARCHIVE return 0 } testprogram_hooks.sh

  27. Interactive Job: getting a shell Starti2glogin inone shell ofthe UI Everyoneshould use a different TCP portbetween 20000 and 25000 i2glogin waits for theconnection to beestablished... Build a JDL for aninteractive JOB Submitthejobinanother shell oftheUI JobType= "normal"; Executable = "i2glogin"; Arguments = "-r -p 20784:193.146.75.73 -t -c /bin/sh"; STDOUTPUT = "std.out"; STDERROR = "std.err"; InputSandbox = {"i2glogin"}; OutputSandbox = {"std.err", "std.out"}; Submityour JOB in a 2nd UI terminal... Port:IPofthemachineexpectingtheconnection [tut02@i2gui01 ~]$ ./i2glogin -p20784 -p20784:10.10.0.22 [tut02@i2gui01 ~]$glite-wms-job-submit -a interactive.jdl

  28. MPI interactive JOB Needs the adition of “i2glogin as PRECOMMAND using I2G_MPI_PRECOMMAND mpi-interactive.jdl JobType = "Normal"; nodeNumber = 2; Executable = "starter.sh"; Arguments = "IMB-MPI1 OPENMPI"; InputSandbox = {"imb.tar.gz", "imb_hooks.sh", "starter.sh", "i2glogin.64"}; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"}; Requirements = Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment) && Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment); Environment = {"I2G_MPI_PRE_RUN_HOOK=imb_hooks.sh", "I2G_MPI_PRECOMMAND='./i2glogin.64 -r -p 20717:193.146.75.73 -t -c'"}; Rank = 0;

  29. MPI interactive JOB The PRECOMMAND needs to have execution permissions in the starter.sh if [ "x$I2G_MPI_PRECOMMAND" != "x" ] ; then PRE=`mktemp` cat > $PRE << EOF #!/bin/bash ARGS="\$*" set `echo $I2G_MPI_PRECOMMAND` chmod +x \$1 exec $I2G_MPI_PRECOMMAND "\$ARGS" EOF export I2G_MPI_PRECOMMAND=$PRE chmod +x $PRE fi starter.sh glite-wms-job-submit -a -r ngiescream.i3m.upv.es:8443/cream-pbs-biomedmpi-interactive.jdl

  30. More information • MPI-Start: http://devel.ifca.es/mpi-start • Documentation: http://grid.ifca.es/wiki/Middleware/MpiStart/ • EGI MPI-Start installation and configuration: • https://wiki.egi.eu/wiki/MAN03_MPI-Start_Installation_and_Configuration • TCG MPI wiki: http://www.grid.ie/mpi/wiki/

More Related