LCG Status and Plans

1. LCG Status and Plans GridPP13 Durham, UK 4th July 2005 Ian Bird IT/GD, CERN

2. Overview • Introduction • Project goals and overview • Status • Applications area • Fabric • Deployment and Operations • Baseline Services • Service Challenges • Summary

3. We are here LCG – Goals • The goal of the LCG project is to prototype and deploy the computing environment for the LHC experiments • Two phases: • Phase 1: 2002 – 2005 • Build a service prototype, based on existing grid middleware • Gain experience in running a production grid service • Produce the TDR for the final system • Phase 2: 2006 – 2008 • Build and commission the initial LHC computing environment LCG and Experiment Computing TDRs completed and presented to the LHCC last week

4. Project Areas & Management Project Leader Les Robertson Resource Manager – Chris Eck Planning Officer – Jürgen Knobloch Administration – Fabienne Baud-Lavigne • Distributed Analysis - ARDA • Massimo Lamanna • Prototyping of distributed • end-user analysis using • grid technology Joint with EGEE Applications Area Pere Mato Development environment Joint projects, Data management Distributed analysis Middleware Area Frédéric Hemmer Provision of a base set of grid middleware (acquisition, development, integration)Testing, maintenance, support CERN Fabric AreaBernd Panzer Large cluster management Data recording, Cluster technology Networking, Computing service at CERN Grid Deployment Area Ian Bird Establishing and managing the Grid Service - Middleware, certification, security operations, registration, authorisation,accounting

5. Applications Area

6. Application Area Focus • Deliver the common physics applications software • Organized to ensure focus on real experiment needs • Experiment-driven requirements and monitoring • Architects in management and execution • Open information flow and decision making • Participation of experiment developers • Frequent releases enabling iterative feedback • Success defined by experiment validation • Integration, evaluation, successful deployment

7. Validation Highlights • POOL successfully used in large scale production in ATLAS, CMS, LHCb data challenges in 2004 • ~400TB of POOL data produced • Objective of a quickly-developed persistency hybrid leveraging ROOT I/O and RDBMSes has been fulfilled • Geant4 firmly established as baseline simulation in successful ATLAS, CMS, LHCb production • EM & hadronic physics validated • Highly stable: 1 G4-related crash per O(1M) events • SEAL components underpin POOL’s success, in particular the dictionary system • Now entering a second generation with Reflex • SPI’s Savannah project portal and external software service are accepted standards inside and outside the project

8. Current AA Projects • SPI – Software process infrastructure (A. Aimar) • Software and development services: external libraries, savannah, software distribution, support for build, test, QA, etc. • ROOT – Core Libraries and Services (R. Brun) • Foundation class libraries, math libraries, framework services, dictionaries, scripting, GUI, graphics, etc. • POOL – Persistency Framework (D. Duellmann) • Storage manager, file catalogs, event collections, relational access layer, conditions database, etc. • SIMU - Simulation project (G. Cosmo) • Simulation framework, physics validation studies, MC event generators, participation in Geant4, Fluka.

9. SEAL and ROOT Merge • Major change in the AA has been the merge of the SEAL project with ROOT project • Details of the merge are being discussed following a process defined by the AF • Breakdown into a number of topics • Proposals discussed with the experiments • Public presentations • Final decisions by the AF • Current status • Dictionary plans approved • MathCore and Vector libraries proposals have been approved • First development release of ROOT including these new libraries

10. Ongoing work • SPI • Porting LCG-AA software to amd64 (gcc 3.4.4) • Finalizing software distribution based on Pacman • QA tools: test coverage and savannah reports • ROOT • Development version v5.02 released last week • Including new libraries: mathcore, reflex, cintex, roofit • POOL • Version 2.1 released including new file catalog implementations: LFCCatalog (lfc), GliteCatalog (glite, Fireman), GTCatalog (globus toolkit) • New version of Conditions DB (COOL) 1.2 • Adapting POOL to new dictionaries (Reflex) • SIMU • New Geant4 public minor release 7.1 is being prepared • Public release of Fluka expected by end July • Intense activity in the combined calorimeter physics validation with ATLAS, report in September. • New MC generators being added (CASCADE , CHARYBDIS, etc.) into the already long list of generators provided • Prototyping persistency of Geant4 geometry with ROOT

11. Fabric Area

12. Extremely Large Fabric management system configuration, installation andmanagement of nodes lemon LHC Era Monitoring - system & service monitoring LHC Era Automated Fabric – hardware / state management Includes technology developed by European DataGrid CERN Fabric • Fabric automation has seen very good progress • The new systems for managing large farms are in production at CERN

13. CERN Fabric • Fabric automation has seen very good progress • The new systems for managing large farms are in production at CERN • New CASTOR Mass Storage System • Was deployed first on the high throughput cluster for the recent ALICE data recording computing challenge • Agreement on collaboration with Fermilab on Linux distribution • Scientific Linux based on Red Hat Enterprise 3 • Improves uniformity between the HEP sites serving LHC and Run 2 experiments • CERN computer centre preparations • Power upgrade to 2.5 MW • Computer centre refurbishment well under way • Acquisition process started

14. Preparing for 7,000 boxes in 2008

15. High Throughput Prototype openlab/LCG • Experience with likely ingredients in LCG: • 64-bit programming • next generation I/O (10 Gb Ethernet, Infiniband, etc.) • High performance cluster used for evaluations, and for data challenges with experiments • Flexible configuration • components moved in and out of production environment • Co-funded by industry and CERN

16. Alice Data Recording Challenge • Target – one week sustained at 450 MB/sec • Used the new version of Castor mass storage system • Note smooth degradation and recovery after equipment failure

17. Deployment and Operations

18. Computing Resources: June 2005 Number of sites is already at the scale expected for LHC - demonstrates the full complexity of operations • Country providing resources • Country anticipating joining • In LCG-2: • 139 sites, 32 countries • ~14,000 cpu • ~5 PB storage • Includes non-EGEE sites: • 9 countries • 18 sites

19. Operations Structure • Operations Management Centre (OMC): • At CERN – coordination etc • Core Infrastructure Centres (CIC) • Manage daily grid operations – oversight, troubleshooting • Run essential infrastructure services • Provide 2nd level support to ROCs • UK/I, Fr, It, CERN, + Russia (M12) • Hope to get non-European centres • Regional Operations Centres (ROC) • Act as front-line support for user and operations issues • Provide local knowledge and adaptations • One in each region – many distributed • User Support Centre (GGUS) • In FZK – support portal – provide single point of contact (service desk)

20. RC RC RC RC ROC RC RC RC RC RC ROC RC RC RC CIC CIC RC ROC RC CIC OMC CIC CIC CIC RC RC RC ROC RC RC RC Grid Operations • The grid is flat, but • Hierarchy of responsibility • Essential to scale the operation • CICs act as a single Operations Centre • Operational oversight (grid operator) responsibility • rotates weekly between CICs • Report problems to ROC/RC • ROC is responsible for ensuring problem is resolved • ROC oversees regional RCs • ROCs responsible for organising the operations in a region • Coordinate deployment of middleware, etc • CERN coordinates sites not associated with a ROC RC = Resource Centre It is in setting up this operational infrastructure where we have really benefited from EGEE funding

21. GIIS Monitor + Monitor Graphs Sites Functional Tests GOC Data Base Scheduled Downtimes Live Job Monitor GridIce – VO + fabric view Certificate Lifetime Monitor Grid monitoring • Operation of Production Service: real-time display of grid operations • Accounting information • Selection of Monitoring tools:

22. LCG-2 (=EGEE-0) 2004 prototyping prototyping product 2005 product LCG-3 (=EGEE-x?) Operations focus • Main focus of activities now: • Improving the operational reliability and application efficiency: • Automating monitoring  alarms • Ensuring a 24x7 service • Removing sites that fail functional tests • Operations interoperability with OSG and others • Improving user support: • Demonstrate to users a reliable and trusted support infrastructure • Deployment of gLite components: • Testing, certification  pre-production service • Migration planning and deployment – while maintaining/growing interoperability • Further developments now have to be driven by experience in real use

23. Recent ATLAS work ~10,000 concurrent jobs in the system Number of jobs/day • ATLAS jobs in EGEE/LCG-2 in 2005 • In latest period up to 8K jobs/day • Several times the current capacity for ATLAS at CERN alone – shows the reality of the grid solution

24. Baseline Services & Service Challenges

25. Baseline Services: Goals • Experiments and regional centres agree on baseline services • Support the computing models for the initial period of LHC • Thus must be in operation by September 2006. • Expose experiment plans and ideas • Timescales • For TDR – now • For SC3 – testing, verification, not all components • For SC4 – must have complete set • Define services with targets for functionality & scalability/performance metrics. • Very much driven by the experiments’ needs – • But try to understand site and other constraints Not done yet

26. Storage management services Based on SRM as the interface Basic transfer services gridFTP, srmCopy Reliable file transfer service Grid catalogue services Catalogue and data management tools Database services Required at Tier1,2 Compute Resource Services Workload management VO management services Clear need for VOMS: roles, groups, subgroups POSIX-like I/O service local files, and include links to catalogues Grid monitoring tools and services Focussed on job monitoring VO agent framework Applications software installation service Reliable messaging service Information system Baseline services • Nothing really surprising here – but a lot was clarified in terms of requirements, implementations, deployment, security, etc

27. Preliminary: Priorities A: High priority, mandatory service B: Standard solutions required, experiments could select different implementations C: Common solutions desirable, but not essential

28. 2005 2006 2007 2008 SC3 First physics cosmics First beams Full physics run SC4 LHC Service Operation Service Challenges – ramp up to LHC start-up service June05 - Technical Design Report Sep05 - SC3 Service Phase May06 – SC4 Service Phase Sep06 – Initial LHC Service in stable operation Apr07 – LHC Service commissioned See Jamie’s talk for more details SC2 SC2 – Reliable data transfer (disk-network-disk) – 5 Tier-1s, aggregate 500 MB/sec sustained at CERN SC3 –Reliable base service – most Tier-1s, some Tier-2s – basic experiment software chain – grid data throughput 500 MB/sec, including mass storage (~25% of the nominal final throughput for the proton period) SC4 –All Tier-1s, major Tier-2s – capable of supporting full experiment software chain inc. analysis – sustain nominal final grid data throughput LHC Service in Operation –September 2006 – ramp up to full operational capacity by April 2007 – capable of handling twice the nominal data throughput

29. Baseline Services, Service Challenges, Production Service, Pre-production service, gLite deployment, … … confused?

30. Services … • Baseline services • Are the set of essential services that the experiments need to be in production by September 2006 • Verify components in SC3, SC4 • Service challenges • The ramp up of the LHC computing environment – building up the production service, based on results and lessons of the service challenges • Production service • The evolving service putting in place new components prototyped in SC3, SC4 • No big-bang changes, but many releases!!! • gLite deployment • As new components are certified, will be added to the production service releases, either in parallel with or replacing existing services • Pre-production service • Should be literally a preview of the production service, • But is a demonstration of gLite services at the moment – this has been forced on us by many other constraints (urgency to “deploy” gLite, need for reasonable scale testing, … )

31. Releases and Distributions • We intend to maintain a single line of production middleware distributions • Middleware releases from [JRA1, VDT, LCG, …] • Middleware distributions for deployment from GDA/SA1 • Remember: announcement of a release is months away from a deployable distribution (based on last 2 years experience) • Distributions still labelled “LCG-2.x.x” • Would like to change to something less specific to avoid LCG/EGEE confusion • Frequent updates for Service challenge sites • But only needed for SC sites • Frequent updates as gLite is deployed • Not clear if all sites will deploy all gLite components immediately • This is unavoidable • A strong request from LHC experiment spokesmen to the LCG POB: • “early, gradual and frequent releases of the [baseline] services is essential rather than waiting for a complete sets” Throughout all this, we must maintain a reliable production service, which gradually improves in reliability and performance

32. Summary • We are at end of LCG Phase 1 • Good time to step back and look at achievements and issues • LCG Phase 2 has really started • Consolidation of AA projects • Baseline services • Service challenges and experiment data challenges • Acquisitions process starting • No new developments  make what we have work absolutely reliably, and be scaleable, performant • Timescale is extremely tight • Must ensure that we have appropriate levels of effort committed