An Example of CAF Workflow (CMS Tracker Alignment)

1. Computing Infrastructure Mostly 8 core WNs with 2GB memory/core Express queue for exteme priority work Memory intensive jobs use slots with 4GB/job 700 Cores, accessed via LSF Multiple Queues Fair share Priority Queues CASTOR Pool : 216 nodes 1.2 Petabytes Storage disk Disk only (CASTOR) Manual Space Mgmt Distrib. Analysis User list managed via web interface by stakeholders Standard CMS analysis tool available for CAF usage Interactive access Special batch queue which allows dedicated interactive sessions (up to 20 cores per user) User storage Group share 2TB AFS space Commissioning dedicated CASTOR user pool (50TB) An Example of CAF Workflow (CMS Tracker Alignment) Alignment & Calibration results CMS Detector HLT + Storage Manager • Step 1: track-level analysis & track-by-track matrix elements: • Step 2: global fit of alignment parameters 450MB/s • dedicated millepede server • to support memory-intensive fit CAF TIER-0 • parallelised across many CPUs High priority users only The CMS CERN Analysis Facility (CAF)Peter Kreuzer (RWTH Aachen) -peter.kreuzer@cern.ch Stephen Gowdy (CERN), Jose Afonso Sanches (UERJ Brazil)on behalf of the CMS Offline & Computing project CAF Primary Alignment Producer AlCaReco Condensed Track data Resource Ramp up 2008 • Alignment and Calibration • Trigger/detector diagnostics, monitoring and performance analysis • Physics monitoring, analysis of express streams, fast-turnaround high-priority analysis 600MB/s Global Millepede fit Alignment Producer … … AlCaReco TIER-1 TIER-1 TIER-1 TIER-1 [Job Slots] AlCaReco TIER-2 TIER-2 TIER-2 TIER-2 Misaligned Geometry Aligned Geometry Constants [Month 08] Max rate-in : 3.5 GB/s Plateau rate-out : ~2GB/s [TB] [Month 08] • Ramp up and Commissioning in Spring 08 • Cosmic data taking in Fall 08 • Factor x1.8 additional CPU in 2009 CAF in Prompt Data Flow CAF Jobs and Users 2008 AlCaReco: data for alignment and calibration [Nb Jobs] 268 Users CMS Centre [Nb Users] Monitoring Monitoring [Month 08] [Month 08] • Reached >500k jobs/month during Fall 08 data taking • Dedicated to high-priority workflows (not useable by every CMS user) • Today nearly 300 active CAF users • Monitor/Control user activity is non-trivial T0 Store T1 Sites * Express reconstruction on O(10%) quickly, then full pass after 24h CAF Utilisation CAF pool data transfers [Jobs] [Tbytes] [Mbytes/s] [Mbytes/s] cmscaf LSF queue: 635 job slots [h] [h] • Free Space on CAF CASTOR pool • Disk-only CASTOR pool for fast data access • Dynamic disk space monitoring/alarming needed • Data deletions triggered by CAF Data Managers, using central CMS Data Management tools [Days] [h] • Running/Pending Jobs batch queue • Job statistics from the cmscaf LSF queue during Fall 08 data taking • Average data transfer rate • CAF can receive transfers from the T0 and from T1 sites. • During Fall 08 run, reached an average input rate of 112 MB/s • Peak data transfer rate • Regularly reaching input rates >2.5 GB/s sustained during 1 hour • This is a Disk to Disk rate max jobs running : 635 average job slots usage: 67% • In Fall 2008 CMS ran 4 weeks continuously • Acquired ~300M Cosmic events with magnetic field at B=3.8 Tesla • Good opportunity to test CAF workflows • Example: mean of residual distributions in Tracker Primary Alignment Producer Condensed Track data 26 mm Primary Alignment Producer Condensed Track data CHEP’09 Conference, PRAGUE (Czech Republic) Mar 2009