1 / 22

Meta-Computing at D Ø

Meta-Computing at D Ø. Igor Terekhov, for the D Ø Experiment Fermilab, Computing Division, PPDG ACAT 2002 Moscow, Russia June 28, 2002. Overview. Overview of the D0 Experiment Introduction into Computing and the paradigm of Distributed Computing SAM – the advanced data handling system

cedric
Download Presentation

Meta-Computing at D Ø

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Meta-Computing at DØ Igor Terekhov, for the DØ Experiment Fermilab, Computing Division, PPDG ACAT 2002 Moscow, Russia June 28, 2002

  2. Overview • Overview of the D0 Experiment • Introduction into Computing and the paradigm of Distributed Computing • SAM – the advanced data handling system • Global Job And Information Management (JIM) – the current Grid project • Collaborative Grid work

  3. The DØ Experiment • P-pbar collider experiment 2TeV • Detector (Real) Data • 1,000,000 Channels (793k from Silicon Microstrip Tracker), 5-15% read at a time • Event size 250KB (25% increase in RunIIb) • Recorded event rate 25 Hz RunIIa, 50 Hz (projected) RunIIb • On-line Data Rate0.5 TB/day, Total 1TB/day • Est. 3 year totals (incl Processing and analysis): • Over 109 events, 1-2 PB • Monte Carlo Data • 6 remote processing centers • Estimate ~300 TB in next 2 years.

  4. The Collaboration • 600+ people • 78 Institutions • 18 countries • Is a large Virtual Organization whose members share resources for solving common problems

  5. Analysis Assumptions

  6. Data Storage • The Enstore Mass Storage System, http://www-isd.fnal.gov/enstore/index.html • All data is stored on tape in Automated Tape Library (ATL) – robot, including derived datasets • Enstore is attached to the network, accessible via a cp-like command • Other, remote MSS’s may be used (the distributed ownership paradigm – grid computing)

  7. Data Handling - SAM • Responds to the above challenges in: • Amounts of data • Rate of access (processing) • The degree to which the user base is distributed • Major goals and requirements • Reliably store (real and MC) produced data • Distribute the data globally to remote analysis centers • Catalogue the data – contents, status, locations, processing history, user datasets etc • Manage resources

  8. SAM Highlights • SAM is Sequential data Access via Meta-data • http://d0db.fnal.gov/sam • Joint project between D0 and Computing Division started in 1997 to meet the Run II data handling needs • Employs a centrally managed RDBMS (Oracle) for meta-data catalog • Processing takes place at stations • Actual data is managed by a fully distributed set of collaborating servers (see architecture later)

  9. SAM Advanced Features • Uniform interfaces for data access modes • Online system, reconstruction farm, Monte-Carlo farm, analysis server are all subclasses of the station. • Uniform capabilities for processing at FNAL and remote centers • On-demand data caching and forwarding (intra-cluster and global) • Resource management: • Co-allocation of compute and data resources (interfaces with batch system abstraction) • Fair share allocation and scheduling

  10. Components of a SAM Station Producers/ /Consumers Project Masters Temp Disk Cache Disk MSS or Other Station MSS or Other Station File Storage Server Station & Cache Manager File Storage Clients File Stager(s) Data flow Control eworkers

  11. WAN SAM as a Distributed System Data Site User Station Master Station Master Station Master Station Master Station 1 FNAL Shared Globally (standard): Shared locally, optional Logger Database Server optimizer MSS Logger optimizer

  12. Routing+Caching=Replication Data Site WAN Data Flow User Station Master Station Master Station Master Station Master Station Master Station Master Mass Storage System Mass Storage System User User

  13. SAM as a Data Grid • Provides high-level collective services of reliable data storage and replication • Embraces multiple MSS’s (Enstore , HPSS, etc) local resource management systems (LSF, FBS, PBS, Condor), several different file transfer protocols (bbftp, kerberos rcp, grid ftp, …) • Optionally uses Grid technologies and tools • Condor as a Batch system (in use) • Globus FTP for data transfers (ready for deployment) • From de facto to de jure…

  14. Client Applications Web Command line Python codes, Java codes D0 Framework C++ codes Request Formulator and Planner Request Manager Storage Manager Cache Manager Job Manager Collective Services “Dataset Editor” “Project Master” “Station Master” “Station Master” “File Storage Server” Batch Systems - LSF, FBS, PBS, Condor Job Services SAM Resource Management Data Mover “Optimiser” “Stager” Significant Event Logger Naming Service Catalog Manager Database Manager Mass Storage systems protocols e.g. encp, hpss CORBA UDP Catalog protocols File transfer protocols - ftp, bbftp, rcp GridFTP Connectivity and Resource GSI SAM-specific user, group, node, station registration Bbftp ‘cookie’ Authentication and Security Fabric Resource and Services Catalog Meta-data Catalog Tape Storage Elements Disk Storage Elements Replica Catalog LANs and WANs Compute Elements Code Repository Indicates component that will be replaced enhanced or added using PPDG and Grid tools Name in “quotes” is SAM-given software component name

  15. Dzero SAM Deployment Map Processing Center Analysis site

  16. SAM usage statistics for DZero • 497 registered SAM users in production • 360 of them have at some time run at least one SAM project • 132 of them have run more than 100 SAM projects • 323 of them have run a SAM project at some time in the past year • 195 of them have run a SAM project in the past 2 months • 48 registered stations, 340 registered nodes • 115TB of data on tape • 63,235 cached files currently (over 1 million entries total) • 702,089 physical and virtual data files known to SAM • 535,048 physical files (90K raw, 300K MC related) • 71,246 “analysis” projects ever ran • http://d0db.fnal.gov/sam_data_browsing/ for more info

  17. SAM + JIM  Grid • So we can reliably replicate a TB of data, what’s next? • It is handling of jobs, not data, that constitutes the top of the services pyramid • Need services for job submission, brokering and reliable execution • Need resource discovery and opportunistic computing (shared vs dedicated resources) • Need monitoring of the global system and jobs • Job and Information Management (JIM) emerged

  18. JIM and SAM-Grid • (NB: Please hear Gabriele Garzoglio’s talk) • Project started in 2001 as part of the PPDG collaboration to handle D0’s expanded needs. • Recently included CDF • These are real Grid problems and we are incorporating (adopting) or developing Grid solutions • http://www-d0.fnal.gov/computing/grid • PPDG, GridPP, iVDGL, DataTAG and other Grid Projects

  19. SAMGrid Principal Components • (NB Please come to Gabriele’s talk) • Job Definition and Management: The preliminary job management architecture is aggressively based on the Condor technology provided by through our collaboration with University of Wisconsin CS Group. • Monitoring and Information Services: We assign a critical role to this part of the system and widen the boundaries of this component to include all services that provide, or receive, information relevant for job and data management. • Data Handling: The existing SAM Data Handling system, when properly abstracted, plays a principal role in the overall architecture and has direct effects on the Job Management services.

  20. SAM-Grid Architecture Monitoring and Information Job Handling (All) Job Status Updates JH Client Logging and Bookkeeping Info Processor And Converter Request Broker AAA Condor MMS Resource Info GSI Job Scheduler MDS-2 Condor Class Ads Replica Catalog Condor-G Grid RC Site Gatekeeper GRAM DH Resource Management Compute Element Resource Data Delivery and Caching SAM Batch System Grid sensors Data Handling Information Principal Component Implementation Or Library Service

  21. SAMGrid: Collaboration of Collaborations • HEP Experiments are traditionally collaborative • Computing solutions in the Grid era: new types of collaboration • Sharing solution within experiment – UTA MCFarm software etc • Collaboration between experiments – D0 and CDF joining forces an important event for SAM and FNAL • Collaboration among the grid players: Physicists, Computer Scientists (Condor and Globus teams), Physics-oriented computer professionals (such as myself)

  22. Conclusions • The Dzero experiment is one of the largest currently running experiments and presents computing challenges • The advanced data handling system, SAM, has matured. It is fully distributed, its model is proven sound and we expect to scale to meet RunII needs for both D0 and CDF • Expanded needs are in the area of job and information management • The recent challenges are typical of the Grid Computing and D0 engages actively, in collaboration with Computer scientists and other Grid participants • More in Gabriele Garzoglio’s talk

More Related