1 / 10

Using the Grid

Using the Grid. Steve Lloyd Queen Mary, University of London. Getting Started. 1. Get a digital certificate. Authentication – who you are. http://ca.grid-support.ac.uk/. 2. Join a Virtual Organisation (VO) For LHC join LCG and choose a VO. Authorisation – what you are allowed to do.

tomas
Download Presentation

Using the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using the Grid Steve LloydQueen Mary, University of London GridPP13 Durham July 2005

  2. Getting Started 1. Get a digital certificate Authentication – who you are http://ca.grid-support.ac.uk/ 2. Join a Virtual Organisation (VO) For LHC join LCG and choose a VO Authorisation – what you are allowed to do http://lcg-registrar.cern.ch/ 3. Get access to a local User Interface Machine (UI) and copy your files and certificate there Lxplus: > source /afs/cern.ch/project/gd/LCG-share/2.4.0/sl3/etc/profile.d/grid_env.sh GridPP13 Durham July 2005

  3. Overview Resource Broker Input Sandbox Output Sandbox Script you want to run Output files (Ntuples, Logs...) Storage Element Storage Element Other files (Job Options, Source...) Compute Element Grid UI Machine Job Description Proxy Certificate Job Output Sandbox Input Data Output Data GridPP13 Durham July 2005

  4. Job Preparation Prepare a file of Job Description Language (JDL): #############Hello World################# Executable = "/bin/echo"; Arguments = "Hello Steve welcome to the Grid "; StdOutput = "hello.out"; StdError = "hello.err"; OutputSandbox = {"hello.out","hello.err"}; ######################################### Simplest Case? Input files Script to run Job Options My C++ Code Realistic Case: ############# athena.jdl ################# Executable = "athena.sh"; StdOutput = "athena.out"; StdError = "athena.err"; InputSandbox = {"athena.sh", "MyJobOptions.py", "MyAlg.cxx", "MyAlg.h", "MyAlg_entries.cxx", "MyAlg_load.cxx", "login_requirements", "requirements", "Makefile"}; OutputSandbox = {"athena.out","athena.err", "ntuple.root", "histo.root", "CLIDDBout.txt"}; Requirements = Member("VO-atlas-release-10.0.1", other.GlueHostApplicationSoftwareRunTimeEnvironment); ################################################ Output Files Choose ATLAS Software Version GridPP13 Durham July 2005

  5. Job Preparation A Job Script (ATLAS Specific): #!/bin/bash # Script to run Athena Analysis on the Grid # Steve Lloyd source $VO_ATLAS_SW_DIR/software/10.0.1/setup.sh source $SITEROOT/dist/10.0.1/Control/AthenaRunTime/*/cmt/setup.sh # Set up the correct directory structure for my files (?) cmt create UserAnalysis UserAnalysis-00-02-02 # Fails to create these for some reason mkdir UserAnalysis/UserAnalysis-00-02-02/UserAnalysis mkdir UserAnalysis/UserAnalysis-00-02-02/src/components mv AnalysisSkeleton.cxx UserAnalysis/UserAnalysis-00-02-02/src mv AnalysisSkeleton.h UserAnalysis/UserAnalysis-00-02-02/UserAnalysis mv UserAnalysis_load.cxx UserAnalysis/UserAnalysis-00-02-02/src/components mv UserAnalysis_entries.cxx UserAnalysis/UserAnalysis-00-02-02/src/components mv requirements Makefile UserAnalysis/UserAnalysis-00-02-02/cmt # Build my code export CMTPATH=`pwd`:${CMTPATH} cd UserAnalysis/UserAnalysis-00-02-02/cmt cmt config source setup.sh gmake # Back to base cd ../../.. # Botch required to find PDGTABLE.MeV: #cp $SITEROOT/dist/10.0.1/InstallArea/share/PDGTABLE.MeV . # Run the job: athena.py AnalysisSkeleton_jobOptions.py GridPP13 Durham July 2005

  6. Job Submission Make a copy of your certificate to send out (~ once a day): [lloyd@lcgui ~/atlas]$ grid-proxy-init Your identity: /C=UK/O=eScience/OU=QueenMaryLondon/L=Physics/CN=steve lloyd Enter GRID pass phrase for this identity: Creating proxy .............................. Done Your proxy is valid until: Thu Mar 17 03:25:06 2005 [lloyd@lcgui ~/atlas]$ Submit the Job: VO File to hold job IDs JDL [lloyd@lcgui ~/atlas]$ edg-job-submit --vo atlas -o jobIDfile athena.jdl Selected Virtual Organisation name (from --vo option): atlas Connecting to host lxn1188.cern.ch, port 7772 Logging to host lxn1188.cern.ch, port 9002 ================================ edg-job-submit Success ==================================== The job has been successfully submitted to the Network Server. Use edg-job-status command to check job current status. Your job identifier (edg_jobId) is: - https://lxn1188.cern.ch:9000/0uDjtwbBbj8DTRetxYxoqQ The edg_jobId has been saved in the following file: /home/lloyd/atlas/jobIDfile ============================================================================================ [lloyd@lcgui ~/atlas]$ GridPP13 Durham July 2005

  7. [lloyd@lcgui ~/atlas]$ edg-job-status -i jobIDfile------------------------------------------------------------------1 : https://lxn1188.cern.ch:9000/tKlZHxqEhuroJUhuhEBtSA2 : https://lxn1188.cern.ch:9000/IJhkSObaAN5XDKBHPQLQyA3 : https://lxn1188.cern.ch:9000/BMEOq90zqALvkriHdVeN7A4 : https://lxn1188.cern.ch:9000/l6wist7SMq6jVePwQjHofg5 : https://lxn1188.cern.ch:9000/wHl9Yl_puz9hZDMe1OYRyQ6 : https://lxn1188.cern.ch:9000/PciXGNuAu7vZfcuWiGS3zQ7 : https://lxn1188.cern.ch:9000/0uDjtwbBbj8DTRetxYxoqQa : allq : quit------------------------------------------------------------------Choose one or more edg_jobId(s) in the list - [1-7]all:7*************************************************************BOOKKEEPING INFORMATION:Status info for the Job : https://lxn1188.cern.ch:9000/0uDjtwbBbj8DTRetxYxoqQCurrent Status: Done (Success)Exit code: 0Status Reason: Job terminated successfullyDestination: lcg00125.grid.sinica.edu.tw:2119/jobmanager-lcgpbs-shortreached on: Wed Mar 16 17:45:41 2005*************************************************************[lloyd@lcgui ~/atlas]$ Job Status Find out its status: Ran at: Valencia RAL CERN Taiwan Taiwan GridPP13 Durham July 2005

  8. Job Retrieval Retrieve the Output: [lloyd@lcgui ~/atlas]$ edg-job-get-output -dir . -i jobIDfile Retrieving files from host: lxn1188.cern.ch ( for https://lxn1188.cern.ch:9000/0uDjtwbBbj8DTRetxYxoqQ ) ********************************************************************************* JOB GET OUTPUT OUTCOME Output sandbox files for the job: - https://lxn1188.cern.ch:9000/0uDjtwbBbj8DTRetxYxoqQ have been successfully retrieved and stored in the directory: /home/lloyd/atlas/lloyd_0uDjtwbBbj8DTRetxYxoqQ ********************************************************************************* [lloyd@lcgui ~/atlas]$ ls -lt /home/lloyd/atlas/lloyd_0uDjtwbBbj8DTRetxYxoqQ total 11024 -rw-r--r-- 1 lloyd hep 224 Mar 17 10:47 CLIDDBout.txt -rw-r--r-- 1 lloyd hep 69536 Mar 17 10:47 ntuple.root -rw-r--r-- 1 lloyd hep 5372 Mar 17 10:47 athena.err -rw-r--r-- 1 lloyd hep 11185282 Mar 17 10:47 athena.out GridPP13 Durham July 2005

  9. Reading Data Either specify in the JDL (then resource broker will try and find a CE 'close' to the SE): Logical file names ############# athena.jdl ################# ... InputData = {"lfn:rome.004201.recov10.ZeeJimmy._00018.AOD.pool.root", "lfn:rome.004201.recov10.ZeeJimmy._00018.AOD.pool.root"}; DataAccessProtocol = {"gsiftp"}; ... ################################################ "rome" stands for ATLAS Rome Workshop Production not the location of the file! Or pull the files from inside the job script: #!/bin/bash # Get the data lcg-cp -v --vo atlas lfn:rome.004201.recov10.ZeeJimmy._00018.AOD.pool.root file://`pwd`/Zee_18.pool.root lcg-cp -v --vo atlas lfn:rome.004201.recov10.ZeeJimmy._00019.AOD.pool.root file://`pwd`/Zee_19.pool.root ... GridPP13 Durham July 2005

  10. Conclusions • The Grid is here – it works! • Currently difficult to install and maintain the middleware and the experiment’s software • It is VERY straightforward to use if all you have is a binary executable • It is straightforward to use if someone has installed your experiment's software somewhere • There are huge resources available: On 29 June 5789 out of 14079 CPUs free (UK: 1548 out of 2966) Use it! GridPP13 Durham July 2005

More Related