1 / 32

ATLAS DDM Operations Issues

This document discusses the operations issues, T1/T2 functional test, and conclusions of the ATLAS Distributed Data Management (DDM) workshop held at BNL. It also provides information on the roles and responsibilities of Tier-1s and Tier-2s in DDM operations.

rscott
Download Presentation

ATLAS DDM Operations Issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ATLAS DDM Operations Issues Alexei Klimentov, BNL US ATLAS DDM and Production Workshop BNL, Sep 29th 2006

  2. DDM Operations issues • DDM Operations (highlights from ATLAS SW week) • DDM T1/T2 functional test • Conclusions and Action items US ATLAS DDM WS. A.Klimentov

  3. ATLAS Distributed Data Management Operations Note (ATL-COM-SOFT-2006-008) ATLAS Distributed Data Management Operations D.Barberis, J.Chudoba, S.Jezequel, J.Kennedy, A.Klimentov, D.Liko, P.Nevski, A.Olszewski, L.Perini, G.Poulard DDM day-by-day operations Operations team organization, T2 reps joined DDM ops team roles and responsibilities of Tier-1s and Tier-2s coordinators https://twiki.cern.ch/twiki/bin/viewfile/Atlas/DDMOperations?rev=2;filename=DDMops_Note.pdf US ATLAS DDM WS. A.Klimentov

  4. DDM Ops team • ASGC (Jason Shih, Suijian Zhou) • BNL (Wensheng Deng, Hironori Ito, Xin Zhao) • CERN (MB,DC,AK) • CNAF (Guido Negri, Paolo Veronesi and Giuseppe Lo Re ) • FZK (John Kennedy, Jiri Chudoba, Andrzej Olszewski ) • IN2P3 Lyon (Stephane Jezequel, Ghita Rahal) • NG (Alex Read, Oxana Smirnova) • PIC (Xavier Espinal) • RAL (Catalin Condurache, Frederic Brochu) • SARA (Jiri Chudoba , grid.support@sara.nl) • TRIUMF (Rod Walker, Denice Deatrich, Reda Tafirout) • Central Team : • AK, Cunfeng Feng, Zhijun Liang, Pavel Nevski and Nectarios Benekos • Stephane, Oxana, Jiri • Yuri Smirnov, Tomasz Wlodek (DB support and monitoring) Many new names and also people from Tier-2s, who actively participating in DDM operations US ATLAS DDM WS. A.Klimentov

  5. DDM Operations (US ATLAS) • Alexei Klimentov and Wensheng Deng • BNL : Wensheng Deng, Hironori Ito, Xin Zhao • GLTier2 : UM : Shawn Mckee, • NETier2 : BU : Saul Youssef, • WTier2 : Wei Yang, Stephen Gowdy • SWTier2 : Patrick McGuigan • OU : Horst Severini, Karthik Arunachalam • UTA : Patrick McGuigan, Mark Sosebee • MWTier2 : Dan Schrager • IU : Kristy Kallback-Rose, Dan Schrager, • UC : Robert Gardner, Greg Cross http://www.usatlas.bnl.gov/twiki/bin/view/Projects/ATLASDDM US ATLAS DDM WS. A.Klimentov

  6. DDM Operations Team Responsibilities (Central Team) • day-by-day data transfer between Tier-0 and Tier-1s • data transfer monitoring and control • data rerouting in case of Tier-1/Tier-2 or transfer channels instability • resolving data transfer errors and providing first line expertise • reporting data transfer problems to the Computing Operations Coordinator • deployment of DDM/DQ2 releases • 24/7 support of central DDM operations facilities • production server at CERN • DQ2 client • central databases • Support DDM Savannah users requests portal • Keep data integrity, in particular clean obsolete datasets and files entries from LFC/LRC and DDM catalogues • Help ATLAS users to transfer data • Communication with services developers and providers (both ATLAS and WLCG) and CERN Networking personnel (NetOps) US ATLAS DDM WS. A.Klimentov

  7. DDM Operations Team Responsibilities (US ATLAS Regional Team) • day-by-day data transfer between BNL and Tier-2s • data transfer monitoring and control • resolving data transfer errors and providing first line expertise • day-by-day data transfer between BNL and Tier-0 (together with central DDM team) • data exchange between BNL and Tier-1s • DDM/DQ2 SW installation and maintenance on VO boxes • keep data integrity in particular clean obsolete datasets and files records from DDM and LRC • Help to ATLAS users to transfer data, in particular for the physicists from geographically closed Universities and Laboratories US ATLAS DDM WS. A.Klimentov

  8. Tier-2: GRIF CEA/DAPNIA LAL (LLR) LPNHE (IPNO) Tier-2: (Subatech Nantes) Tier-2: LPC Clermont-Ferrand Centre d’Analyse - Lyon Tier-3: CPPM Marseille Tier-3: LAPP Annecy Tier-1: CC-IN2P3 Lyon LYON CLOUD Non french sites: • TOKYO • BEIJING (Upgraded SE end august) List of T1/T2/T3 Ready since end August Preparation and follow-up Meeting each 3-4 weeks of french sites + Tokyo (I. Ueda) • Fast and effective interactions with responsibles in each T2/T3 • Tests of plain FTS made before SC4 US ATLAS DDM WS. A.Klimentov S.Jezequel, G.Rahal : Computing Operations Session

  9. DDM Operations at SARA Cloud • Site Issues • vobox failure (local hard disk) • restored from backup • FTS server • problems since August • local BDII stability – solved • user mappings, channels for atlas vs atlassgm • incidents logged on wiki: https://twiki.cern.ch/twiki/bin/view/Main/AtlasDDMatSARA Jiri Chudoba talk on Computing Operations Sep 12 CERN->SARA data transfer failed during Functional test. Problem was understood and fixed Sep 27th US ATLAS DDM WS. A.Klimentov

  10. Funding and Manpower Issues • The formation of a Data Management team within a cloud (Tier-1/Tier-2s system), although a central ATLAS service, should be seen as the responsibility of the cloud members themselves and it is expected that the member sites (or associated funding bodies) provide the majority of the required manpower. • It is estimated that initially 1.7-1.9 FTE equivalents would be needed to support DDM operations for each Tier-1 cloud; these operations include supporting and monitoring all previously described services and also database replication. This manpower level will decrease to ~1 FTE (over several people to guarantee full coverage) after a couple of years, as the automation and robustness of the tools will improve. • As an example a team may be formed from an expert (data-guru) who would be 75-100% dedicated to data management and O~(3) other members who would give 30% of their time to DDM matters. • The formation of such a cloud based DDM team would ensure that the cloud runs smoother and is more configurable than would be possible if the data management was provided by one ATLAS wide service. John Kennedy proposal US ATLAS DDM WS. A.Klimentov

  11. DDM Ops Main activities since June 1/2 • ATLAS Distributed Data Management Operations Note : ATL-COM-SOFT-2006-008(https://twiki.cern.ch/twiki/bin/viewfile/Atlas/DDMOperations?rev=2;filename=DDMops_Note.pdf • Opening up FTS site-to-site access– Dan Schrager (https://twiki.cern.ch/twiki/bin/viewfile/Atlas/DDMOperations?rev=1;filename=D.S) • FTS channels set up (BNL-Tier-1s) – Hironori Ito • DDM ops Savannah page – Z.Liang et al • DDM ops (and DDM developers) HyperNews • US ATLAS DDM operations team as a part of DDM ops • DQ2 installation procedure (Patrick, Horst, Dan) • OSG specifics datasets/files cleaning script (Wensheng, Patrick) US ATLAS DDM WS. A.Klimentov

  12. DDM Ops Main activities since June 2/2 • DDM/ProdSys integration (also part of SWING activity) – AK, PN • CSC11 AOD datasets consolidation (together with D.Liko et al) at CERN – Cunfeng Feng, Zhijun Liang, Pavel Nevski at LYON - Stephane Jezequel at BNL - David Adams, Wensheng Deng, Xin Zhao at FZK - John Kennedy But the goal to consolidate all AODs at 3 centers not reached yet. • Nordic Grid/DQ2 integration – Alex Read Automatic datasets book-keeping Automatic datasets copying to CERN 1st step to full integration • T1/T2 data transfer functional test – AK, Pavel Nevski US ATLAS DDM WS. A.Klimentov

  13. ATLAS Tier1-Tier2s Data Transfer Functional Test (Sep 11 - Sep21) Alexei Klimentov, Pavel Nevski

  14. Tier-1/Tier-2s Data Transfer Functional Test 1/3 • The test was proposed by Zhongliang Ren within ATLAS Computing Operations Group • DDM ops was asked to define the scope of the test and to conduct it The goal is to have system test (*) UC, IU, OU, UTA, SLAC (BU,UM) US ATLAS DDM WS. A.Klimentov

  15. Functional test is a first phase of DDM regular system check-up Sites have been asked to run the test without any special actions. Exceptions : ASGC (the site was used by developers just before the test to debug DDM software) LYON and RAL (the latest version of DQ2 was installed. CC & AK) A coherent data transfer test between Tier-1 and Tier-2s for all ‘clouds’, using existing SW to generate data, replicate them to ATLAS sites, monitor and control data transfer. Data generated by ATLAS MC production SW Organized in DQ2 datasets Performance evaluation is not the goal of the test We want to know how functional is our system Tier-1/Tier-2s Data Transfer Functional Test 2/3 US ATLAS DDM WS. A.Klimentov

  16. Tier-1/Tier-2s Data Transfer Functional Test 3/3 • Step 1 • 1a : Data distribution from Tier-0 to Tier-1 • 1b : Data distribution from Tier1 to Tier-2s within the cloud • 1c : Bulk data transfer from CERN to Tier-1 • Step 2 • For Tier-2s passed step 1b, consolidate data from Tier-2s on Tier-1 • Step 2a • Access data located at Tier-2s in “foreign cloud” from Tier-1 (BNL). Opening up FTS site-to-site access (Dan’s proposal) US ATLAS DDM WS. A.Klimentov

  17. Test Statistics • 8 sites passed Step 1a (CERN -> Tier-1) • 3.5 sites passed Step 1b (BNL, CNAF, GRIDKA and LYON) • Severe SW errors have been found during Step 1b • The test for several sites was postponed : • ASGC, GRIDKA, PIC and RAL • DDM developers starts investigated the problems • Step 1c was executed for 4 sites (CNAF, BNL, GRIDKA and LYON) • 2 sites completed the test (LYON and CNAF) • The bulk data transfer (from CERN to BNL) wasn’t succeeded (slow subscription). The problem is understood and fixes will be provided by DDM core team. US ATLAS DDM WS. A.Klimentov

  18. Typical DDM problems observed • Extremely long subscription servicing cycle (20 and more hours between the moment dataset subscription request and the first action to process the request) • Obscure subscription history, lack of monitoring and control • DDM/DQ2 SW and Computing Facilities interference (DQ2/Storage and DQ2/FTS interference - data replication requests from PIC and ASGC blocked BNL storage system. Also LYON was blocked by T2 data transfer) • Inconsistencies in DDM monitoring • Site FTS related errors : ‘0’ length files on CASTOR, files overwrite on dCache, keeping FTS/LFC/etc busy and blocking data transfer to the particular site • Dataset location. Though there are no data on site or data transfer failed, site is shown as dataset replicas holder. • Rare 100% data replication, no way for automatic check, control and retransfer. US ATLAS DDM WS. A.Klimentov

  19. Errors observed : Site related HW failures and power outages FTS misconfiguration Authentication failures Problems with local SE Problems with SW components (non-ATLAS, f.e. ORACLE) It takes from 2h to 3 days to locate and fix problems. All sites have people responding to error report US ATLAS DDM WS. A.Klimentov

  20. Errors observed : Data Transfer Errors • Network and Storage system problems • timeouts, SRM put/get, etc • Problems caused by Software (ATLAS and GRID) • LFC busy, slow dataset subscription • Data transfer in ‘VALIDATED’ state • Data to Tier-2 are transferred via srmcp from CERN, though expected via FTS from the parent Tier-1 Practically for all sites Regional DDM team is able to check computing facilities and to run FTS tests locally. US ATLAS DDM WS. A.Klimentov

  21. Errors observed : Data Transfer Failures • No data transfer to or/and from site The most severe errors. REQUIRE ACTION FROM DDM DEVELOPERS Typical cases : files transfer is in ‘ASSIGNED’ state for days dataset subscription isn’t processed data transfer agent crashed Impossible to reliably monitor the above errors from DDM monitoring. To find information one must dig in local log files Fixes required manual intervention and manipulation with local DQ2 database and VO box. US ATLAS DDM WS. A.Klimentov

  22. Test Results • In general, the situation is very unstable. It is difficult to identify one source of failures. Manual interventions (on VO boxes) are required to fix problems and to control the data transfer. • To run such a test is a full time job now. • Two weeks of running • Sep 11 – Sep 15 • Some problems with sites (fixed). Only one site was able to get data at the beginning (TRIUMF) • Data transfer from CERN to sites with reasonable failure rate, after 2-3 attempts • Sep 15 – Sep 21 • Take 4 sites out of the test (ASGC, GRIDKA, PIC and RAL. GRIDKA is back Sep 19th. ) • Manual interventions for other sites to resume data transfer. • Bulk data transfer test : ~45%-65% data transfer efficiency • Situation deteriorated US ATLAS DDM WS. A.Klimentov

  23. Test Results (sites summary) US ATLAS DDM WS. A.Klimentov

  24. Near Term and Strategic Plans Strategic plan is needed to bring DDM to the production quality • To define the long-term plans we need to agree about action items and to define the priorities and common strategy. • If the collaboration won’t define the strategic plan. We must have one for US ATLAS • US ATLAS Computing needs to define DDM requirements and priorities (as it is seen from our side) and submit them to Computing Management. • This will work only in case of joint effort of all groups • Developers • Operations • Analysis • Production • AND COORDINATION US ATLAS DDM WS. A.Klimentov

  25. Near Term Plans (DDM ops) • DDM Operations Issues • AOD and ESD datasets consolidation at CERN, BNL, LYON, FZK C.Feng, S.Jezequel, D.Adams, X.Zhao, P.Nevski, J.Kennedy • Check Data Integrity • BNL (Wensheng) • generic script to check LRC/”storage”/DQ2 • Pre-DQ data consolidation (Cunfeng) • Check for obscure subscriptions, files, datasets and start clean up (TBD : Dietrich, Cunfeng,Liang,Pavel, AK) • Database replication to ATLAS sites (P.Nevski together with A.Vanyashine et al) • Support user’s requests for data transfer • Hiro, Xin, Tadashi, Pavel, David, AK • Maintain DDM facilities US ATLAS DDM WS. A.Klimentov

  26. Near Term Plans (Functional Test) • Continue the test after SC4 test (mid of October). The test scope will be updated. • Include functional test as a part of scheduled tests coordinated by Gilbert Poulard. • Consider functional test as a part of DDM ops activity and run the test regularly (GP) US ATLAS DDM WS. A.Klimentov

  27. Near Term Plans (installation and deployment) • DQ2 new releases and deployment • DDM ops and T1/T2 sites Coordinators will take responsibility for DDM deployment. • Major releases should be beta-tested by DDM ops group (as it was done for 0.2.9). • GRID specifics features must be tested in advance • Coherent deployment for all sites • Priority must be given to ATLAS production needs • DDM/DQ2 “full-scale” test bed (anyway we will need it very soon) US ATLAS DDM WS. A.Klimentov

  28. Near Term Plans (integration with Prodsys) • DDM/ProdSys Integration • NordicGrid/DQ2 Integration (A.Read, Z.Liang, AK) • ‘Final’ Job/Files state (P.Nevski, AK) • “input” files repository (problems with inter-grid files replication) • Missing files in MC datasets (P.Nevski) • Need to discuss with Rod, Kaushik, Simone (probably can be done within ‘Final’ Job check). • Production datasets tables, task definitions, integration with AKTR (P.Nevski,AK) US ATLAS DDM WS. A.Klimentov

  29. Conclusions 1/2 • Improvements during last 3 months are evident • DDM operations team is set up • More people are involved in DDM operations and technical discussion • GRID and Prodsys Integration with DQ2 • Users work with datasets and DQ2 subscription mechanism • DQ2 release 0.2.1x and monitoring are dramatic step forward. We need to keep it running for a while to find out as much as possible correlations and bottlenecks of computing elements and DDM/DQ2 SW • Some bottlenecks are already observed even for the modest data rates we have now (many of them are known from Panda experience, many are new and related to inter-site communication) US ATLAS DDM WS. A.Klimentov

  30. Conclusions 2/2 Many problems related not only to DQ2, but to the late DQ2/prodsys integration, GRID tools, local settings, computing storage facilities, Sites files catalogs for the end-user it is a problem with ATLAS DDM SW WE NEED TO DRAW THE PLAN HOW TO IMPROVE THE SITUATION US ATLAS DDM WS. A.Klimentov

  31. Action Items • SWING, DDM ops and DDM developers must discuss and prepare the list of priorities for the next DQ2 release(s) to bring ATLAS DDM system to the production quality • DDM production and DDM developers computing facilities must be kept separated. The SW upgrade on production facilities can be done only after testing and debugging. No last minute upgrades. • GRID tools has to be evaluated. We cannot rely on believe that the next release of XYZ will satisfy our needs. We have to select and use tools we can run in production today. The clear understanding of all problems is needed. • LFC performance and robustness look like the most critical for the whole system. LFC and LRC evaluation is one of the first priorities. • ATLAS dedicated DDM functional and performance tests are required. The tests must be considered as a part of computing facilities commissioning and the participation must be mandatory for ATLAS sites. • We need more people to work on DDM operations and monitoring. Probably the DDM ops model need to be tuned, and central team will assign people to help particular Tier-1s. • DDM Operational model must be reviewed after October’s tests • DDM priorities and strategy need to be defined US ATLAS DDM WS. A.Klimentov

  32. DDM Operations (US ATLAS) • Alexei Klimentov and Wensheng Deng • BNL : Wensheng Deng, Hironori Ito, Xin Zhao • GLTier2 : UM : Shawn Mckee, • NETier2 : BU : Saul Youssef, • WTier2 : Wei Yang, Stephen Gowdy • SWTier2 : Patrick McGuigan • OU : Horst Severini, Karthik Arunachalam • UTA : Patrick McGuigan, Mark Sosebee • MWTier2 : Dan Schrager • IU : Kristy Kallback-Rose, Dan Schrager, • UC : Robert Gardner, Greg Cross What fraction of your time you can dedicate to DDM Operations What is task you are interested in US ATLAS DDM WS. A.Klimentov

More Related