ATLAS Analysis Use

1. ATLAS Analysis Use Dietrich Liko

2. Credits • GANGA Team • PANDA DA Team • ProdSys DA Team • ATLAS EGEE/LCG Taskforce • EGEE Job Priorities WG • DDM Team • DDM Operations Team

3. Overview • ATLAS Computing Model • AOD & ESD analysis • TAG based analysis • Data Management • DDM and Operation • Workload Management • EGEE, OSG & Nordugrid • Job Priorities • Distributed Analysis Tools • GANGA • pathena • Other tools: Prodsys, LJSF • Distributed Analysis activities during SC4 • Tier-2 Site requirements

4. Tier-2 in the Computing Model • Tier-2 centers have an important role • Calibration • Simulation • Analysis • Tier-2 provide analysis capacity for the physics and detector groups • In general chaotic access patterns • Typically a Tier-2 center will host … • Full TAG samples • One third of the full AOD sample • Selected RAW and ESD data • Data will be distributed according to guidelines given by the physics groups

5. Analysis models • For efficient processing it is necessary to analyze data locally • Remote access to data is discouraged • To analyses the full AOD and the ESD data it is necessary to locate the data and send the jobs to the relevant sites • TAG data at Tier-2 will be file based • Analysis uses same pattern as AOD analysis

6. AOD Analysis • The assumption is that the users will perform some data reduction and generate Athena aware ntuples (AANT) • There are also other possibilities • This development is steered by the Physics Analysis Tools group (PAT) • AANT are then analyzed locally

7. Aims for SC4 & CSC • The main data to be analyzed are the AOD and the ESD data from the CSC production • The total amount of data is still small …few 100 GB • We aim first at a full distribution of the data to all interested sites • Perform tests of the computing model by analysis these data and measurement of relevant indicators • Reliability • Scalability • Throughput • Simply answer the questions … • How long does it take to analyze the expected 150 TB of AOD data corresponding to one year of running of the LHC • And what happens if several of you try to do it at the same time

8. In the following …. • I will discuss the technical aspects necessary to achieve these goals • ATLAS has three grids with different middleware • LCG/EGEE • OSG • Nordugrid • Data Management is shared between the grids • But there are grid specific aspects • Workload Management is grid specific

9. Data Management • I will review only some aspect related to Distributed Analysis • The ATLAS DDM is based on Don Quijote 2 (DQ2) • See the tutorial session on Thursday for more details • Two major tasks • Data registration • Datasets are used to increase the scalability • Tier-1’s are providing the necessary services • Data replication • Based on the gLite File Transfer Service (FTS) • Fallback to SRM or gridftp possible • Subscription are used to manage the actual file transfer

10. How to access data on a Tier-2 Dataset catalog http CE Tier 0 rfio dcap gridftp nfs lrc protocol VOBOX SE LRC FTS Tier 2 Tier 1

11. To distributed the data for analysis … • Real data • Data recorded and processed at CERN (Tier-0) • Data distribution via Tier-1 • Reprocessing • Reprocessing at Tier-1 • Distribution via other Tier-1 • Simulation • Simulation at Tier-1 and associated Tier-2 • Collection of data at Tier-1 • Distribution via other Tier-1

12. For analysis … • The challenge is not the amount of data, but the management of the overlapping flow patterns • For SC4 we have a simpler aim … • Obtain a equal distribution for the current available simulated data • Data from the Tier-0 exercise is not useful for analysis • We will distribute only useful data

13. Grid specific aspects • OSG • DDM fully in production since January • Site services also at Tier-2 centers • LCG/EGEE • Only dataset registration in production • New deployment model addresses this issue • Migration to new version 0.2.10 on the way • Nordugrid • Up to now only dataset registration • Final choice of file catalog still open

14. New version 0.2.10 • Many essential features for Distributed Analysis • Support for the ATLAS Tier structure • Fallback from FTS to SRM and gridftp • Support for disk areas • Parallel operation of production and SC4 • And many more …. • Deployment is on the way • We hope to see in production very soon • OSG/Panda has to move to it asap • We should stay with this version until autumn • The success of Distributed Analysis during SC4 is crucially depending on the success of this version

15. Local throughput • An Athena job has in the ideal case about 2MB/sec data throughput • The limit given by the persistency technology • Storegate-POOL-ROOT • Up to 50% of the capacity of a site is dedicated to analysis • We plan to access data locally via the native protocol (rfio, dcap, root etc) • Local network configuration should take that into account

16. Data Management Summary • Data Distribution is essential for Distributed Analysis • DQ2 0.2.10 has the required features • There is a lot of work in front of us to control and validate data distribution • Local configuration determined by Athena I/O

17. Workload management • Different middleware • Different teams • Different submission tools • Different submission tools are confusing to our users .. • We aim to obtain some common ATLAS UI following the ideas of pathena tool (see later) • But …. the priority for Distributed Analysis in the context of SC4 is to solve the technical problems within each grid infrastructure

18. LCG/EGEE Sites Sites gLite UI gLite Resource Broker Sites Sites Dataset Location Catalog BDII

19. Advantages of the gLite RB • Bulk submission • Increased performance and scalability • Improved Sandbox handling • Shallow resubmission • If you want to use a Resource Broker for Distributed Analysis, you want to use finally the gLite RB • Status • Being pushed into deployment with gLite 3.0 • Has not yet the same maturity as the LCG RB • Turing the gLite RB into production quality has evidently a high priority

20. ATLAS EGEE/LCG Taskforce gLite LCG Bulk submission Multiple threads Submission: 0.3 sec/job 2 sec/job Matchmaking: 0.6 sec/job Job submission is not the limit any more (there are other limits ….)

21. Plan B: Prodsys + CondorG Sites ProdDB Sites CondorG Executor Sites Sites Dataset Location Catalog CondorG Negotiator BDII

22. Advantages • Profits from the experiences of the production • Proven record from the ongoing production • Better control on the implementation • CondorG by itself is also part of the Resource Broker • Performance • ~ 1 sec/job • Status • Coupled to the evolvement of the production system • Work on GANGA integration has started

23. OSG • On OSG ATLAS uses PANDA for Workload management • Concept is similar to DIRAC and Alien • Fully integrated with the DDM • Status • In production since January • Work to optimize the support for analysis users is ongoing

24. PANDA Architecture

25. Advantages for DA • Integration with DDM • All data already available • Jobs start only when the data is available • Late job binding due to pilot jobs • Addresses grid inefficiencies • Fast response for user jobs • Integration of production and analysis activities

26. Nordugrid • ARC middleware • Compact User Interface • 14 MB vs • New version has very good job submission performance • 0.5 to 1 sec/job • Status • Open questions: Support for all ATLAS users, DDM integration • Analysis capability seems to be coupled with the planned NG Tier-1

27. ARC Job submission ARC UI Sites RLS Nordugrid IS

28. WMS Summary • EGEE/LCG • gLite Resource Broker • Prodsys & CondorG • OSG • PANDA • Nordugrid • ARC • Different system – different problems • All job submission systems need work to optimize user analysis

29. Job Priorities • Different infrastructures – different problems • EGEE • Job Priority WG • OSG/Panda • Separate cloud of analysis pilots • Nordugrid • Typically a site has several queues

30. EGEE Job Priority WG • TCG working group • ATLAS & CMS • LHCb & Diligent observer • JRA1 (developers) • SA3 (deployment) • Several sites (NIKHEF, CNAF)

31. ATLAS requirements • Split site resources in several shares • Production • Long jobs • Short jobs • Other jobs • Objectives • Production should not be pushed from a site • Analysis jobs should bypass production jobs • Local fairshare

32. Proposed solution Role=Production Production 70% CE Long 20 % Role=Software Software 1 % CE Short 9 %

33. Status • Based on VOMS Roles • Role=Production • Role=Software • No new middleware • A patch to the WMS has to be back ported • Test Installation • NIKHEF, CNAF • TCG & WLCG MB have agreed to the proposed solution • We are planning the move to the preproduction service • Move to the production sites in the not so far future • In the future • Support for physics groups • Dynamic settings • Requires new middleware (as GPBOX)

34. PANDA • Increase the number of analysis pilots • Fast pickup of user jobs • First job can start in few seconds • Several techniques are being studied • Multitasking pilots • Analysis queues

35. Job Priorities Summary • EGEE/LCG • New site configuration • OSG/Panda • Addressed by PANDA internal developments

36. DA User Tools • pathena • PANDA tool for Distributed Analysis • Close to the Physics Analysis Tools group (PAT) • GANGA • Common project between LHCb & ATLAS • Used for Distributed Analysis on LCG

37. pathena • Developed in close collaboration between PAT and PANDA • Local job athena ttbar_jobOptions.py • Grid job pathena ttbar_jobOptions.py –inDS csc11.005100.ttbar.recon.AOD… –split 10

38. GANGA • Framework for job submission • Based on plugins for • Backends • LCG, gLite, CondorG, LSF, PBS • Applications • Athena, Executable • GPI abstraction layer • Python Command Line Interface (CLIP) • GUI

39. GANGA GUI

40. Other Tools • LJSF (Light Job submission framework) • Used for ATLAS software installations • Runs ATLAS transformations on the grid • No integration with DDM yet • DA with Prodsys • Special analysis transformations • Work to interface with GANGA has started

41. DA Tools Summary • Main tools • pathena with PANDA/OSG • GANGA with Resource Broker/LCG • Integration with Athena as demonstrated by pathena is a clear advantage • GANGA plug-in mechanism allows in principle to obtain a unique interface • Priority for the GANGA team is to deliver a robust solution on LCG first

42. Distributed Analysis in SC4 • Data distribution • Ongoing activity with the DDM operations team • Site configuration • We will move soon to the preproduction service • In few weeks we will then move to the production sites • Exercising the job priority model • Analysis in parallel to production • Scaling tests of the computing infrastructure • Measurement of the turnaround time for analysis of large datasets

43. SC4 Timescale • We plan to perform DA tests in August and then later in autumn • The aim is to quantify the current characteristics of the Distributed Analysis systems • Scalability • Reliability • Throughput • Simply answer the questions … • How long does it take to analyze the expected 150 TB of AOD data corresponding to one year of running of the LHC • And what happens if several of you try to do it at the same time

44. Tier-2 Site Requirements • Configuration of the batch system to support the job priority model • gLite 3.0 • Analysis and production in parallel • Data availability • Connect to DDM • Disk area • Sufficient local throughput