250 likes | 391 Views
The ATLAS Computing Model. Roger Jones Lancaster University CHEP06 Mumbai 13 Feb. 2006. Overview. Brief summary ATLAS Facilities and their roles Growth of resources CPU, Disk, Mass Storage Network requirements CERN↔ Tier 1 ↔ Tier 2 Operational Issues and Hot Topics.
E N D
The ATLAS Computing Model Roger Jones Lancaster University CHEP06 Mumbai 13 Feb. 2006
Overview • Brief summary ATLAS Facilities and their roles • Growth of resources • CPU, Disk, Mass Storage • Network requirements • CERN↔ Tier 1 ↔ Tier 2 • Operational Issues and Hot Topics RWL Jones 13 Feb. 2006 Mumbai
Computing Resources • Computing Model fairly well evolved, documented in C-TDR • Externally reviewed • http://doc.cern.ch//archive/electronic/cern/preprints/lhcc/public/lhcc-2005-022.pdf • There are (and will remain for some time) many unknowns • Calibration and alignment strategy is still evolving • Physics data access patterns MAY be exercised from June • Unlikely to know the real patterns until 2007/2008! • Still uncertainties on the event sizes , reconstruction time • Lesson from the previous round of experiments at CERN (LEP, 1989-2000) • Reviews in 1988 underestimated the computing requirements by an order of magnitude! RWL Jones 13 Feb. 2006 Mumbai
ATLAS Facilities • Event Filter Farm at CERN • Located near the Experiment, assembles data into a stream to the Tier 0 Center • Tier 0 Center at CERN • Raw data Mass storage at CERN and to Tier 1 centers • Swift production of Event Summary Data (ESD) and Analysis Object Data (AOD) • Ship ESD, AOD to Tier 1 centers Mass storage at CERN • Tier 1 Centers distributed worldwide (10 centers) • Re-reconstruction of raw data, producing new ESD, AOD • Scheduled, group access to full ESD and AOD • Tier 2 Centers distributed worldwide (approximately 30 centers) • Monte Carlo Simulation, producing ESD, AOD, ESD, AOD Tier 1 centers • On demand user physics analysis • CERN Analysis Facility • Analysis • Heightened access to ESD and RAW/calibration data on demand • Tier 3 Centers distributed worldwide • Physics analysis RWL Jones 13 Feb. 2006 Mumbai
Processing • Tier-0: • Prompt first pass processing on express/calibration physics stream • 24-48 hours later, process full physics data stream with reasonable calibrations • Implies large data movement from T0 →T1s • Tier-1: • Reprocess 1-2 months after arrival with better calibrations • Reprocess all resident RAW at year end with improved calibration and software • Implies large data movement from T1↔T1 and T1 → T2 RWL Jones 13 Feb. 2006 Mumbai
Analysis model Analysis model broken into two components • Scheduled central production of augmented AOD, tuples & TAG collections from ESD • Derived files moved to other T1s and to T2s • Chaotic user analysis of augmented AOD streams, tuples, new selections etc and individual user simulation and CPU-bound tasks matching the official MC production • Modest job traffic between T2s RWL Jones 13 Feb. 2006 Mumbai
Inputs to the ATLAS Computing Model (1) RWL Jones 13 Feb. 2006 Mumbai
Inputs to the ATLAS Computing Model (2) RWL Jones 13 Feb. 2006 Mumbai
Data Flow Tier 0 view • EF farm T0 • 320 MB/s continuous • T0 Raw data Mass Storage at CERN • T0 Raw data Tier 1 centers • T0 ESD, AOD, TAG Tier 1 centers • 2 copies of ESD distributed worldwide • T1 T2 • Some RAW/ESD, All AOD, All TAG • Some group derived datasets • T2 T1 • Simulated RAW, ESD, AOD, TAG • T0 T2 Calibration processing? Tier 2 view RWL Jones 13 Feb. 2006 Mumbai
RAW ESD2 AODm2 0.044 Hz 3.74K f/day 44 MB/s 3.66 TB/day RAW ESD (2x) RAW RAW AODm (10x) 1.6 GB/file 0.02 Hz 1.7K f/day 32 MB/s 2.7 TB/day 1.6 GB/file 0.02 Hz 1.7K f/day 32 MB/s 2.7 TB/day 1 Hz 85K f/day 720 MB/s OtherTier-1s OtherTier-1s EachTier-2 ESD2 ESD2 ESD2 ESD1 ESD2 AOD2 AOD2 AODm1 AODm2 AODm1 AODm2 AODm2 AODm2 AODm2 T1 T1 T1 T1 T1 T1 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day 10 MB/file 0.2 Hz 17K f/day 2 MB/s 0.16 TB/day 10 MB/file 0.2 Hz 17K f/day 2 MB/s 0.16 TB/day 500 MB/file 0.036 Hz 3.1K f/day 18 MB/s 1.44 TB/day 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day 500 MB/file 0.036 Hz 3.1K f/day 18 MB/s 1.44 TB/day 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day 500 MB/file 0.004 Hz 0.34K f/day 2 MB/s 0.16 TB/day 500 MB/file 0.004 Hz 0.34K f/day 2 MB/s 0.16 TB/day ATLAS partial &“average” T1 Data Flow (2008) Tape Tier-0 diskbuffer Plus simulation and analysis data flow CPUfarm diskstorage RWL Jones 13 Feb. 2006 Mumbai
Total ATLAS Requirements in for 2008 RWL Jones 13 Feb. 2006 Mumbai
Important points: • Discussion on disk vs tape storage at Tier-1’s • Tape in this discussion means low-access secure storage • No ‘disk buffers’ included except input to Tier 0 • Storage of Simulation data from Tier 2’s • Assumed to be at T1s • Need partnerships to plan networking • Must have fail-over to other sites • Commissioning • Requirement of flexibility in the early stages • Simulation is a tunable parameter in T2 numbers! • Heavy Ion running still under discussion. RWL Jones 13 Feb. 2006 Mumbai
ATLAS T0 Resources RWL Jones 13 Feb. 2006 Mumbai
ATLAS T1 Resources RWL Jones 13 Feb. 2006 Mumbai
ATLAS T2 Resources RWL Jones 13 Feb. 2006 Mumbai
Required Network Bandwidth • Caveats • No safety factors • No headroom • Just sustained average numbers • Assumes no years/datasets are ‘junked’ • Physics analysis pattern still under study… RWL Jones 13 Feb. 2006 Mumbai
T1 ↔ CERN Bandwidth I+O The projected time profile of the nominal bandwidth required between CERN and the Tier-1 cloud. Mainly outward data movement RWL Jones 13 Feb. 2006 Mumbai
T1 ↔ T1 Bandwidth I+O About half is scheduled analysis The projected time profile of the nominal bandwidth required T1 and T1 cloud. RWL Jones 13 Feb. 2006 Mumbai
T1↔ T2 Bandwidth I+O Dominated by AOD The projected time profile of the nominal aggregate bandwidth expected for an average ATLAS Tier- 1 and its three associated Tier-2s. RWL Jones 13 Feb. 2006 Mumbai
Issues 1: T1 Reprocessing • Reprocessing at Tier 1s is understood in principle • In practice, requires efficient recall of data from archive and processing • Pinning, pre-staging, DAGs all required? • Requires the different storage roles to be well understood RWL Jones 13 Feb. 2006 Mumbai
Issues 2: Streaming • This is *not* a theological issue • All discussions are about optimisation of data access • TDR has 4 streams from event filter • primary physics, calibration, express, problem events • Calibration stream has split at least once since! • At AOD, envisage ~10 streams • ESD streaming? • Straw man streaming schemes (trigger based) being agreed • Will explore the access improvements in large-scale exercises • Will also look at overlaps, bookkeeping etc RWL Jones 13 Feb. 2006 Mumbai
TAG Access • TAG is a keyed list of variables/event • Two roles • Direct access to event in file via pointer • Data collection definition function • Two formats, file and database • Now believe large queries require full database • Restricts it to Tier1s and large Tier2s/CAF • Ordinary Tier2s hold file-based TAG corresponding to locally-held datasets RWL Jones 13 Feb. 2006 Mumbai
Conclusions • Computing Model Data Flow understood for placing Raw, ESD and AOD at Tiered centers • Still need to understand data flow implications of Physics Analysis • SC4/Computing System Commissioning in 2006 is vital. • Some issues will only be resolved with real data in 2007-8 RWL Jones 13 Feb. 2006 Mumbai
Backup Slides RWL Jones 13 Feb. 2006 Mumbai
Heavy Ion Running RWL Jones 13 Feb. 2006 Mumbai