1 / 11

Přístup k datům ATLAS registrovaným na gridu

Přístup k datům ATLAS registrovaným na gridu. Jiří Chudoba FzÚ AV Č R 6.5.2005. Hromadné simulace. velké nároky na CPU i diskový prostor – distriubované výpočty ATLAS produkční systém používá 3 gridy LCG, Nordugrid, GRID3 plus možnost submitování do lokálního dávkového systému

garin
Download Presentation

Přístup k datům ATLAS registrovaným na gridu

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Přístup k datům ATLAS registrovaným na gridu Jiří Chudoba FzÚ AV ČR 6.5.2005

  2. Hromadné simulace • velké nároky na CPU i diskový prostor – distriubované výpočty • ATLAS produkční systém používá 3 gridy • LCG, Nordugrid, GRID3 • plus možnost submitování do lokálního dávkového systému • DC2, Rome production • Výsledné soubory mohou být na různých SE v různých gridech Jiri.Chudoba@cern.ch

  3. Přístup do GRIDu • nutný certifikát, požádat na http://www.cesnet.cz/pki/doc/user-req-guide/ - ukázka • kroky: žádost, ověření, registrace, konverze certifikátu, import do browseru • registrace v ATLAS VO - http://lcg-registrar.cern.ch/ • celý proces lze zvládnout do 1 týdne (mimo období dovolených ) • přístup k UI – lxplus, ui.farm.particle.cz, lokální instalace Jiri.Chudoba@cern.ch

  4. User Interface • nahrát certifikát a klíč do ~/.globus/usercert.pem a userkey.pem • dále podle http://grid-deployment.web.cern.ch/grid-deployment/documentation/UI-lxplus/ • source ...; grid-proxy-init • instalace UI na lokálním stroji (linux): • http://grid-it.cnaf.infn.it/packages/UIPnP/ • update revocation list .../etc/grid-security/certificates Jiri.Chudoba@cern.ch

  5. Kopírování souborů v LCG • lcg-infosites –vo atlas ... seznam CE a SE, adresy RLS a RMC • export RMC_ENDPOINT=http://rlsatlas.cern.ch:7777/atlas/v2.2/edg-replica-metadata-catalog/services/edg-replica-metadata-catalogexport LRC_ENDPOINT=http://rlsatlas.cern.ch:7777/atlas/v2.2/edg-local-replica-catalog/services/edg-local-replica-catalog • edg-lrc mappingsByPfn '*dc2.003017.evgen.A7_w4jets._00010.pool.root*' --endpoint $LRC_ENDPOINTguid:202154C1-B4E7-D811-982B-00E081022B84, sfn://castorgrid.cern.ch/castor/cern.ch/grid/atlas/datafiles/dc2/evgen/dc2.003017.evgen.A7_w4jets/dc2.003017.evgen.A7_w4jets._00010.pool.root.1 edg-lrc pfnsForGuid guid:202154C1-B4E7-D811-982B-00E081022B84 --endpoint $LRC_ENDPOINT • edg-lrc pfnsForGuid guid:202154C1-B4E7-D811-982B-00E081022B84 --endpoint $LRC_ENDPOINT sfn://castorgrid.ific.uv.es/castor/ific.uv.es/grid/atlas/generated/2004-08-09/file629bb8c9-c61a-4ba6-a7d8-59767a5d4f49, sfn://lcgse03.ifae.es/castor/ifae.es/lcg/atlas/generated/2004-08-09/file8e5f4f8c-2ff9-4eb7-a65a-f0a24528b096, sfn://lcg00123.grid.sinica.edu.tw/flatfiles/SE00/atlas/generated/2004-08-09/file8aaaae78-f86d-4ae3-a7e1-5a422eb5d2fc, sfn://castorgrid.cern.ch/castor/cern.ch/grid/atlas/datafiles/dc2/evgen/dc2.003017.evgen.A7_w4jets/dc2.003017.evgen.A7_w4jets._00010.pool.root.1, sfn://t2-se-02.lnl.infn.it/data1/atlas/generated/2004-08-09/filef523a662-0377-4e72-9e5f-c48cd9eedad1, sfn://griditse01.na.infn.it/flatfiles/SE00/atlas/generated/2004-08-10/fileae9f8113-92da-46c4-9278-40a145d8e2cf, sfn://grid01.uibk.ac.at/flatfiles/SE00/atlas/generated/2004-08-12/filea84342de-add0-4049-a61b-debd589d4937, sfn://lcgse01.gridpp.rl.ac.uk/flatfiles/lcg-data/atlas/generated/2004-08-11/filee94f44d3-f244-4356-ba02-efb49c84714e, sfn://cclcgseli01.in2p3.fr/grid/atlas/generated/2004-08-11/filed09515fd-1096-4c72-9193-91676611b3fc, sfn://testbed003.phys.sinica.edu.tw/flatfiles/SE00/atlas/generated/2004-08-14/fileb45e2545-8502-44b9-ab62-29143881337d, sfn://se01.gridpp.shef.ac.uk/data/atlas/generated/2004-08-17/filea1bc76e7-129c-4e9f-9deb-b9c0df5685ea, sfn://atlasse.lnf.infn.it/flatfiles/SE00/atlas/generated/2005-01-15/filefb4e249f-119e-41e1-bbc1-ac496761603d, sfn://grid009.to.infn.it/flatfiles/SE00/atlas/generated/2005-01-16/filef5b8eaff-1a05-4bcd-8f42-7c237e24d9a7 • globus=url=copy gsiftp://atlasse.lnf.infn.it/flatfiles/SE00/atlas/generated/2005-01-15/filefb4e249f-119e-41e1-bbc1-ac496761603d file:/tmp/file.1.pool.root NEVHODNÉ PRO BĚŽNOU PRÁCI, POUŽÍT JEN PŘI HLEDÁNÍ CHYB Jiri.Chudoba@cern.ch

  6. Don Quijote • přístup do všech 3 gridů pod jedním interfacem • metadata • Users guide: https://uimon.cern.ch/twiki/bin/view/Atlas/DonQuijoteDms3 • python2/afs/cern.ch/atlas/offline/external/DQClient/dms3/dms3.py Jiri.Chudoba@cern.ch

  7. Z přednášky Miguela Branca na plenární schůzi během ATLAS SW Workshop http://agenda.cern.ch/fullAgenda.php?ida=a036309#s0 Jiri.Chudoba@cern.ch

  8. Organization of DC-2 data • Typical DC-2 Logical File Names (LFNs): • dc2.003002.evgen.A2_Z_mumu._00010.pool.root • dc2.003026.simul.A0_top._00005.pool.root • In addition, files have a Logical Collection Name. The usage of the LCN is optional: • /datafiles/dc2/evgen/dc2.003002.evgen.A2_z_mumu/dc2.003002.evgen.A2_z_mumu._00002.pool.root • Physical files have a name which is usually similar to the LFN, but with an attempt number in the end: • So, for LFN: dc2.003002.evgen.A2_Z_mumu._00010.pool.root • We can have a physical copy: dc2.003002.evgen.A2_Z_mumu._00010.pool.root.2 • Never read directly a physical file (e.g. from Castor) without querying the Grid Replica Location Service using dms2.py client tool: • you don’t know if it is the latest attempt or not!

  9. Common question • I see some DC-2 data at CERN. How do I find the GUID for the Pool files and how do I generate the XML catalog? • The fact you see files on Castor@CERN under /castor/cern.ch/grid/atlas/datafiles/dc2/… • Does not mean these are good files. • Always use the client tool to find files that have been validated by the DC-2 Production System • All other files will be removed SOON • So, if you find a DC-2 file directly in a SE, you are never sure whether this is or not the latest file: • LFN : dc2.003002.evgen.A2_Z_mumu._00010.pool.root • Physical File at CERN: dc2.003002.evgen.A2_Z_mumu._00010.pool.root.1 • Physical File in Valencia: dc2.003002.evgen.A2_Z_mumu._00010.pool.root.2 Attempt 1. BAD

  10. Examples • ./dms3.py search 'dc2.003002.evgen.A2_z_mumu._0000*.pool.root' • ./dms3.py get dc2.003002.evgen.A2_z_mumu._00001.pool.root --catalog=PoolFileCatalog.xml Jiri.Chudoba@cern.ch

  11. Brzké změny • RLS katalog není spolehlivý • LFC a FIREMAN pro gLite • DQ interface by mohl zůstat stejný Jiri.Chudoba@cern.ch

More Related