210 likes | 335 Views
DØ Grid PP Plans – SAM, Grid, Ceiling Wax and Things. Iain Bertram Lancaster University Monday 5 November 2001. SAM, DØ, and the Grid. DØ Basics What is SAM? History Current Deployment Collaborators SAM, Grid and Future Developments Overview of Plans UK Plans CDF and SAM.
E N D
DØ Grid PP Plans –SAM, Grid, Ceiling Wax and Things Iain Bertram Lancaster University Monday 5 November 2001
SAM, DØ, and the Grid • DØ Basics • What is SAM? • History • Current Deployment • Collaborators • SAM, Grid and Future Developments • Overview of Plans • UK Plans • CDF and SAM Iain A Bertram - Lancaster
The DØ Experiment • Detector Data • 1,000,000 Channels • Event size 250KB • Event rate ~50 Hz • On-line Data Rate 12 MBps • Est. 2 year totals (incl Processing and analysis): • 1 x 109 events • ~0.5 PB • Monte Carlo Data • 5 remote processing centers • Estimate ~300 TB in 2 years. Iain A Bertram - Lancaster
~500 Physicists 72 institutions 18 Countries Collaboration Iain A Bertram - Lancaster
SAM (DØ & FNAL Project) • SAM is Sequential Access to data via Meta-data • Project started in 1997 to handle DØ ’s needs for Run II data system. • SAM is a data-grid • No fully functional Grid currently exists • SAM does have many GRID functionalities • Stations – logical collection of computers, networks, storage • Transparent access and transport of data between stations • Data Cataloguing – Replica Management • Fabric management • Job Submission on Local Station Only http://d0db.fnal.gov/sam Iain A Bertram - Lancaster
Deployment Iain A Bertram - Lancaster
Deployment II WAN LAN Datalogger MSS MSS Reco-farm Central Analysis ClueD0 (Others) • Current active stations • FNAL (several) • Lyon FR (IN2P3), • Amsterdam NL (NIKHEF) • Lancaster UK • Imperial College UK • Others in US Stations at FNAL MSS Interconnected network of primary cache stations Communicating and replicating data where it is needed. Iain A Bertram - Lancaster
Statistics 25TB • Number of registered users: 360 • Data in the system: 25 TB • 160k files • Acessing > 3 TBa day • Goal 13 TB/day • Fully integrated into DØ Analysis Framewrok Iain A Bertram - Lancaster
SAM Collaborators • PPDG - Particle Physics Data Grid • DØ Participation • Condor • Globus • Fermilab CMS Computing Group • iVDGL International Virtual Data-Grid Laboratory. • IGMB – InterGrid Management Board Iain A Bertram - Lancaster
Future Plans SAM is an operational GRID • Ideal platform for demonstrating Grid technologies on the time scale of 2 years • Modular design allows integration of modern Grid Tools • Ideal Testing Ground for LHC scale experiments • Full Scale Test of Grid Middleware Iain A Bertram - Lancaster
DØ Goals • DØ is fully committed to making SAM a fully functional GRID on the timescale of 2 years. • DØ is committed to using standard GRID tools wherever possible. • DØ has/is committing significant resources to the GRID. Iain A Bertram - Lancaster
General Goals • Use of standard middleware to promote interoperability • Globus Security infrastructure, Interoperability with Fermilab Kerberos security infrastructure • GridFTP as one of the supported file transfer protocols • Globus job submission. • Condor and extensions job submission • Publish availability and status of SAM station resources • Publish catalog of data files and their replicas using standard or standards emerging from PPDG and DataGrid Iain A Bertram - Lancaster
General Goals II • Additional Grid functionality for Job specification, submission and tracking. • Use of full Condor services for migration and checkpointing of jobs – as much as is possible with DØ software and the DØ software framework. This may require work on both Condor software to achieve full functionality • Building incrementally enhanced Job specification language and job submission services that ensure co-location of job execution and data files and reliably execute a chain of job processing, with dependencies between job steps. The first step in this is expected to be work in conjunction with the Condor team to provide for specification and execution of a Directed Acyclic Graph of jobs using an extended version of the DAGMAN product that CMS is testing for their MC job execution. Iain A Bertram - Lancaster
General Goals III • Enhancing Monitoring and Diagnostic capabilities • Extensions to existing system of logging all activities in the system to both local and central log files - as demanded by robustness and increased use of system. • Incorporation of emerging Grid Monitoring Architecture and monitoring tools. Little exists on this at this point and this work will involve working with other Grid projects and participating in Global Grid Forum working groups Iain A Bertram - Lancaster
Proposed Applications • Monte Carlo Production System • To upgrade the distributed Monte Carlo production system as a short term use case for demonstrating, in an incremental fashion, essential components of the Grid. In particular this involves demonstrating transparent job submission to a number of DØ processing centres (SAM stations), reliable execution of jobs, transparent data access and data storage and an interface for users to understand and monitor the state of their MC requests and the resultant MC jobs and data that satisfy their requests Iain A Bertram - Lancaster
Proposed Applications • General User Applications • To demonstrate analysis of data products (both MC and Detector Data) on desktop systems in the UK, using an enhanced version of the DØ SAM Grid system that incrementally incorporates Grid middleware components. This will not only demonstrate active use of a Grid for analysis but will also eventually demonstrate interoperability between the DØ Grid and other emerging Grid testbeds. Iain A Bertram - Lancaster
Milestones Iain A Bertram - Lancaster
CDF and SAM • Rick St Denis Talk • CDF to use SAM for data access • UK setting up test facilities • Combined DØ and CDF Proposal Iain A Bertram - Lancaster
Proposal • Request 6 FTEs • Four of the additional FTEs will work on integrating CORE Grid functionality into SAM (this talk) • One FTE for DØ applications (MC and Analysis) • One FTE integrate CDF software with SAM Iain A Bertram - Lancaster
Conclusions • SAM is an operational GRID • Offers great opportunity for testing GRID Middleware • Goal: Fully functional test-bed in two years. Iain A Bertram - Lancaster