1 / 20

CMS Status & Requirements

CMS Status & Requirements. Topics covered: CMS Grid Status CMSUK approach to Grid work First stab at high-level requirements Proposed UK work areas Other sources of info: Our EB/TAG submission – this talk summarises it. CMS Note 2001/037 (Holtmann et al)

hateya
Download Presentation

CMS Status & Requirements

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CMS Status & Requirements • Topics covered: • CMS Grid Status • CMSUK approach to Grid work • First stab at high-level requirements • Proposed UK work areas • Other sources of info: • Our EB/TAG submission – this talk summarises it. • CMS Note 2001/037 (Holtmann et al) • Do NOT read as a fixed statement of CMS approach or priorities • Our approach has already evolved since then… • DataGrid WP8 validation proposal – public at end of this week. • DMN at CERN for CMS software week – apologies sent. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  2. CMS Grid Status • Many active Grid groups within CMS • US (Caltech, FNAL, UCSD, …); INFN; CERN • Some Grid technologies already used ‘in production’ • Data replication via GDMP (for 12 months) • Data movement across WAN using bbcp, and now GridFTP • …and some have been tried in large-scale tests • CONDOR / DAGman • Formal CMS Grid Project just started • Coordinator: C. Grandi (Bologna) • Oriented toward coordination of existing work areas + manpower • Many of the ‘obvious’ work areas already well covered D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  3. 28 Pages D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  4. CMS is developing a detailed vision of the Grid. CMS Overview and Requirements Document “The document contains a snapshot, taken in 2001, of the vision of the intended software capabilities of the production system in 2003, and the expected scaling towards 2007” Document admits that the “vision” will evolve…. But contains architectural choices and constraints. There is a detailed “Division of labour” between tasks for grid-projects components and tasks for CMS components. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  5. 2003 CMS Grid-Job (Vision) D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  6. Pythia CMSIM (Geant3) C) Pythia C) HEPEVT Ntuples Zebra files with Hits DataBase ORCA Add Pile-up ORCA ooHit Formatter CMS has considerable experience already of distributed computing on large scales. ORCA User Analysis Histograms Ntuples Root files... CMS Software Today Pythia Production presently at about 10 regional centres - UK one of the largest - has produced 10TB of data (15TB by eoy) D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  7. CMS Production Status • MC Production aims: • Satisfy (increasing) requests for samples by physics/HLT groups; will get worse as HLT + Physics TDRs approach • Investigate scalability of hardware, software, people (!) • Put in place the correct communication mechanisms + policies • Status: • We are now/soon at the 10’s of TB scale • Production runs quasi-continuously, worldwide. • We are continually find (sometimes even solve!) the expected scaling, communication problems… • CMS physicists: • Gripe a lot • Demand lots of support + a high level of reliability • Are highly clued-up, and provide very useful input + steerage D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  8. Can these milestones contribute to the success of GridPP, and vice versa? CMS Milestones Dec-01: 2x10*7 MC events (reduced processing chain). Jun-02: 2x10*7 MC events (full chain and storage). Late-02: Specification of final CMS software system for LHC era. Dec-02: Higher-level-trigger TDR completed. Dec-03: Computing TDR completed; includes 5% challenge. Late-04: 20% data challenge. Dec-04: Physics TDR completed. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  9. CMS in the UK Main UK contribution to date has been the development of MC production software and its exploitation to meet ongoing simulation requirements. The UK has provide over 25% of the data to date. Plan is to maintain this high profile (much greater than our size). Plan to contribute via GridPP in two ways: Assist with the Grid-enabling of our current world-wide MC production system. Deploy test and operate production software at prototype Tier-1, 2, 3 centres in the UK to test the scalability and performance of the developing software. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  10. Requirements: protoT1 • What do we need? • A rapidly-deployed protoT1 at RAL • Scale attained should be ~0.5 of a Hoffmann T1 centre by end 2003 (all available to CMS for short periods) • Good + responsive hardware and operations support / expertise • Direct support for the tools we need to use there • Large, fast database systems • Commercial software installations (e.g. LHC++, Objy, Oracle) • Is this a ‘technical requirement’? Yes (IMHO). • There is more to building / operating a T1 than stacking boxes (i.e. we need a real programme of R&D) • Industry does NOT yet know how to do this in a cost-effective way • CMS has valuable experience + real plans for this area. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  11. Requirements: Middleware • Middleware / uppermiddleware req’d by CMS includes: • Data management • Covered (for flat files) by GDMP • Workload management • Lots of work being done in US, DataGrid projects, INFN • Monitoring / control • A UK contribution? • How to integrate middleware? • No plans to ‘Gridify’ our core applications (OSCAR, ORCA, etc) • We take a ‘toolkit’ approach for our production (-> analysis) framework; will start to introduce Grid middleware under the toolkit, where it makes life easier. • Integration + testing will be a primary technical task in the UK. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  12. Requirements: ooStuff (I) • Object data management (from a non-expert): • Traditionally, HEP experiments have used flat-file storage • Flat files are ‘easy’ to deal with, but often inefficient, reliant upon huge and complex metadata system, not amenable to ‘virtual data’ approach. • CMS plans to use a true object-based data model, and a corresponding persistent object store. • The problem: • OS / Mass storage only knows about flat files • Worse: data replication tools only know about flat files • Vendor-supplied data management tools are inadequate / insecure for use on the WAN • Files will get very big soon (remember: 2GB limit will not exist for much longer) – no longer a good unit of data transfer. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  13. Requirements: ooStuff (II) • A short-term solution: • CMS will need to implement an object clustering/declustering layer • Stuffs object collections into flat files when required for: Archiving; Data Movement; Creation of compact ‘final stage’ datasets from tagged collections • Destuffs them for efficiency when required. • Long-term solution: • Our data management / metadata tools need to understand object granularity (or at least, object collections and their properties). • This is not a well-understood problem (to say the least); lots of interesting computer science here • Presumably (?) very interesting to industry. • A role for GridPP / UK ? D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  14. Requirements: Interoperability • We have middleware coming from all over the place • Probably a more visible problem within CMS than anywhere else right now. • Please please please: • Coordinate at a high level between projects • Make sure the tools work together wherever possible • Avoid architectural mismatches. • Otherwise: • CMS will have to ‘make a choice’ • GridPP / UK T1 will have to support it. • We will all have to test / integrate twice as much stuff D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  15. CMS-UK: Areas of interest DataGrid deliverables within the WP8 programme (installation, testing, organisation and support of data challenges). MC Grid development: Introducing more grid-elements into the CMS data challenges. Development and integration of object database into regional centres. Monitoring tools for CMS applications and T1 hardware. Object-level replication tool development. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  16. UK Proposed Work Areas (I) • NOT in any priority order • protoT1 database management/support • Primary goal: to enable and support use of last, fast, modern object database systems at the protoT1 • Some ‘operations’ tasks • Some technical development tasks (which can be monitored / managed): e.g. integration with Grid StorageElement. • Also a very important liaison role with • CERN / SLAC / whoever • Vendors • This is a necessary condition for use of the Tier-A/protoT1 by CMS and BaBar • We intend to collaborate in this area. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  17. UK Proposed Work Areas (II) • Production / analysis Grid framework: • The enhancement of our existing MC production toolkit (IMPALA) using Grid technology • This project is starting NOW • Integrates with DataGrid WP8 work to some extent • Will lead directly to Grid analysis framework development (no acronym yet assigned!) • UK contribution: • Use our experience to guide detailed requirements defn • Deploy and test first tools at our protoT1/T2/T3 sites • Investigate possible collaborations with other expts • Also: provide input from working physicists who need the tools! D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  18. UK Proposed Work Areas (III) • Monitoring / T1 management + simulation • Development of tools to allow effective: • Monitoring of workflow • Monitoring of application performance • Analysis of bottlenecks + computing system performance • Comparison of T1 performance with simulation • Infra-T1 workflow scheduling and work pattern optimisation • Builds upon UK expertise in monitoring, info services, data publication • Builds upon MONARC work towards ‘real’ simulation of a local or worldwide computing system • Builds upon UK interest (incl. computer science) in network performance, application optimisation, workflow management. • First tools required (Real Soon) for optimisation of work at T1. D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  19. UK Proposed Work Areas (IV) • Object data management / replication: • Provision of short-term solutions to this problem • Provision of a ‘clustering layer’ • A good example of an indirect contribution by Grid projects to solve a real ‘core software’ problem • Investigation of possible long-term approaches • Lots of potential collaboration with • US projects • Replication tool developers in DataGrid (etc) • Computer scientists • Industry • This work should not be tied to one DB product or approach D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

  20. Summary • CMS and the Grid: • We have considerable experience already. We are already “pushing-the-envelope” with our MC productions. • We have uncovered many problems, issues, impedance mismatches already. CMS is developing a grid plan. • The UK contribution: • Build upon our practical experience / expertise. • Make use of the large protoT1 we will have • Focus upon: • integration / deployment / testing • interesting development projects needed by CMS • There are real, interesting, CMS-related things to do • GridPP can make a big contribution to CMS • CMS UK data challenges can make a big contribution to GridPP D. Newbold / D. Britton GridPP Collaboration Meeting, 5/11/2001

More Related