1 / 43

AliEn Tutorial

AliEn Tutorial. MODEL 2009 4 th May , 2009. Outline. Installation of the AliEn software AliEn and the GRID Authentication File Catalogue Submitting jobs. Installation. Please, do: w get http://alien.cern.ch/alien-torrent-installer c hmod +x alien-torrent-installer

jadzia
Download Presentation

AliEn Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AliEnTutorial MODEL 2009 4th May, 2009

  2. Outline • Installation of the AliEnsoftware • AliEn and the GRID • Authentication • File Catalogue • Submitting jobs

  3. Installation • Please, do: • wgethttp://alien.cern.ch/alien-torrent-installer • chmod +x alien-torrent-installer • ./alien-torrent-installer • Default installation • First two commands fast • Let’s leave the 3rd running...

  4. AliEn, AliEn2 • ALICE ENvironment on the GRID • GRID middleware • Developed as single entry point to the GRID for ALICE (CERN) • All the components necessary to build a GRID and interact with other GRIDs • File System with metadata • Authorization, authentication, job optimization and execution, storage management • Audit, quotas, monitoring • Interfaces to various GRID implementations • Used since: • 2002 for centrally managed productions • 2006 for user analysis • Used by several Virtual Organizations: • ALICE, PANDA, CBM, GPCALMA • http://alien.cern.ch

  5. Pablo Saiz CERN

  6. Pablo Saiz LHC 27km particle accelerator 100 meters underground About to start collecting data

  7. Pablo Saiz A Large Ion ColliderExperiment

  8. Why do we need a catalogue? • Loads of data (several PB per year!!) • Distributed community • Same structure from everywhere • Contains location of the file • It does not contain the file • Possibility of several copies of files • Select the closest one • Metadata information • We will see this later on...

  9. Tier1 |--./ | |--cern.ch/ | | |--user/ | | | |--a/ | | | | |--admin/ | | | | | | | | | |--aliprod/ | | | | | | | |--f/ | | | | |--fca/ | | | | | | | |--p/ | | | | |--psaiz/ | | | | | |--as/ | | | | | | | | | | | |--dos/ | | | | | | | | | | | |--local/ | | | | | | | |--b/ | | | | |--barbera/ | |--36/ | | |--stderr | | |--stdin | | |--stdout | | | |--37/ | | |--stderr | | |--stdin | | |--stdout | | | |--38/ | | |--stderr | | |--stdin | | |--stdout |--simulation/ | |--2001-01/ | | |--V3.05/ | | | |--Config.C | | | |--grun.C AliEn2 File Catalogue • Mapping from LFN to SE and PFN • UNIX-like file system • Metadata • GUID • Used by all other services ALICE REMOTE ALICE USERS ALICE DATA

  10. LFN and GUID catalogues AliEn File & Metadata Catalogue GUID Catalogue LFN Catalogue LFN GUID GUID PFN Index Index LFN: Logical File Name (what the user sees) GUID: Global Unique Identifier (another way of identifying files) PFN: Physical File Name

  11. Who uses the catalogue? • Everybody IF A FILE IS NOT IN THE CATALOGUE, IT DOES NOT EXIST ON THE GRID • Users User data, user executables ... • Job execution Executable, input, output • Packages Source code, binary code, post installation

  12. Back to the exercises... • Did the installation finish? • Let’s get ready to access the GRID • X509 Certificate authentication • Public vs Private key • Certificate Authority • Proxy certificates

  13. Preparing certificates • In $HOME/.globus: • userkey.pem • usercert.pem • Readable only by the user (chmod 400) • Make alien use those files: • mkdir $HOME/.alien • ln –s $HOME/.globus $HOME/.alien/globus

  14. Environment files • $ALIEN_ROOT/.Environment • $HOME/.alien/Environment • Bash files sourced before starting alien • cat <<EOF > $HOME/.alien/Environment export ALIEN_ORGANISATION=PANDA export ALIEN_USER=<username> EOF • Put alien in the PATH • mkdir –p $HOME/bin • ln –s $HOME/alien/bin/alien $HOME/bin/

  15. Getting a GRID proxy • alien proxy-init, alien proxy-info *****

  16. Authentication problems • Globus related: • Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pem • Your certificate authority is exotic and not known to the server. • Your certificate has expired. • Clock skew: • Your local computer time is in the future with respect to the server's time. • Your local computer time is more in the past than the certificate life time.

  17. Try it out !! • Check the certificates in your machines: • Store them under e.g: $HOME/.globus/ • Check that the key file is read only • Setup the environment • $HOME/.alien/Environment • Put alien in the path • Get a valid alien token • alien proxy-init

  18. Accessing the shell Standard bash shell with grid comands Main bash features are available. Not all shell helper programs are available. Some local commands (like ls, cat etc) are overwritten with the corresponding GRID commands. File/path tab completion in the virtual GRID directory.

  19. Basic commands I Tab completion working!!!

  20. Basic commands II

  21. whereiscommand Gives (list of) SE and pfns for a file

  22. Viewing the files

  23. Shell – Copying files (add/get)

  24. Shell – Try it out I • Access the alien shell. • alien • Check your user name: • whoami • List the contents of your home directory • ls • Print the working directory • pwd • Create the following directory structure in your AliEn space: • $HOME/bin (if it is not there) • $HOME/macros and $HOME/jdl • Get the information of the file: • whereis/bin/date

  25. Shell – Try it out II • See the file /bin/date • cat /bin/date • Copy the file to your local machine • cp /bin/date /tmp/my_file • Exit the shell and check the file

  26. Job & Transfer models fts JA File catalogue JA bbftp FTD JA xrdcp LFN GUID Meta data SE CE Transfer Manager Job Manager Packman MonALISA Job Broker Transfer Broker TransferQueue TaskQueue Site A Jobs Site B CE MonALISA Splitting Expired Zombies Packman SE Central services Priorities Merging Site services FTD Job optimizers Transfer optimizers

  27. ALICE sites Europe Africa Asia North America See Costin’s presentation

  28. Running jobs More than 25 CPU/years per day

  29. AliEn ARC/ NORDUGRID Interconnecting GRIDs AliEn user interface VTD EDG CONDOR LCG /GLITE Nice! I STILL do nothavetoworryabouteverchanging GRID environment…

  30. Job status

  31. Submitting jobs Your job is submitted and a <JOBID>.is assigned to it.

  32. Checking the job status

  33. Getting the output

  34. Other useful commands • jobListMatch • List of CE that can execute the job • Spy • Output of the job while it is running • Masterjob • Give information about

  35. Try it out!! • Connect to alien • alien login • Submit the job • submit <<EOF Executable=“date”; EOF • Follow the execution • top –id, ps trace, psjdl • Get the output • cp /proc/<user>/<id>/job-output/stdout /tmp/out

  36. JDL fields I • Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin. • Arguments: They will be passed to the executable. • Packages: Type packages in the shell to see what kind of packages are installed. • InputFile: The files that will be transported to the node where the job will run. • InputData: It will require that the job will be executed in a site close to the files specified here. • InputDataList: The filename in which the Job Agent will write the InputData list. • InputDataListFormat: The format of the InputData list.

  37. JDL fields II • OutputFile: The files that will be registered in the catalog once the job finishes. • OutputArchive: What files will be archived in a zip file. • Validationcommand: Specifies the script to be used as a validation script. • Email: Receive a mail when the job finishes. • TTL: The maximum run time of your job. • Split: Split the jobs in several sub jobs.

  38. Packages • packman • gives a list of the packages defined/installed • installs packages

  39. Try it out !! • Add: • Packages=“pbarprod@pandaroot::nov08“; • Try other commands: • Executable=“pandaroot”; • Multiple submission • Split=“production:1-5”; • Create your own executables

  40. Conclusion • AliEn • Interface to the GRID • Developed by ALICE • Also used by PANDA, CBM, GPCALMA • In production since 2002 • File Catalogue • File and Metadata • Job submission model • Data movement and replication

  41. References I • Registration – Certificates: • http://alien.cern.ch/twiki/bin/view/Alice/UserRegistration • https://ca.cern.ch/ca/ • AliEn: • http://alien.cern.ch • Gshell: • http://alien.cern.ch/twiki/bin/view/AliEn/GAPI • User's guide: • http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/AA-UserGuide-0.0m.pdf

  42. References II • aliensh Grid Command Online Reference V1.0 • http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/guide/guide-1.0.htm • Previous tutorials: • http://aliceinfo.cern.ch/Offline/Analysis/Tutorial/ • Event Tag System: • http://pcaliweb02.cern.ch/Offline/Analysis/RunEventTagSystem/EventTags.html#Event%20tag%20system • https://edms.cern.ch/document/788315/1 (INTERNAL NOTE)

  43. http://alien.cern.ch Thank you very much!! Any questions?

More Related