1 / 25

Scientific Data Management: An Incomplete Experimental HENP Perspective

Scientific Data Management: An Incomplete Experimental HENP Perspective. D. Olson, LBNL 26 March 2002 SDM-ISIC Meeting Gatlinburg. Particle Physics Data Grid. Coordinators: Pordes, Olson. PI’s: Mount, Livny, Newman. www.ppdg.net. Contents. Quick overview of HENP data Generic data flow

hoang
Download Presentation

Scientific Data Management: An Incomplete Experimental HENP Perspective

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scientific Data Management:An Incomplete Experimental HENP Perspective D. Olson, LBNL 26 March 2002SDM-ISIC MeetingGatlinburg

  2. Particle Physics Data Grid Coordinators:Pordes,Olson PI’s: Mount, Livny, Newman www.ppdg.net

  3. Contents • Quick overview of HENP data • Generic data flow • Sizes, timescales • Average physicist view • What’s hard • Making technology work in production • A clear view for average physicist • Analysis of large datasets • Other things as well • Today, many issues wrapped in hopes for “Data Grid”

  4. Experimental HENP event data • Basic character of data is “event” • May be few particles

  5. BaBar event http://www.slac.stanford.edu/BFROOT/

  6. Experimental HENP event data • Basic character of data is “event” • May be few particles • May be MANY particles

  7. STAR event, Au + Au www.star.bnl.gov

  8. Experimental HENP event data • Basic character of data is “event” • May be few tracks • May be MANY tracks • Detector characteristics, beam types, triggers effect the type of events recorded • Physics analysis is a statistical analysis of many (1000’s, M’s, B’s, T’s) independent events

  9. Generic data flow in HENP “Skims”, “microDST production”, … Filtering chosen to make this a convenient size

  10. A collaboration of people Free?, 10 yr, 20 people $100M, 10 yr, 100 people Free?, 1 yr, 10 people, 5x/yr Free?, 1 mo, 1 person, 50x/yr (“Typical” example today, LHC is larger)

  11. Example: CMS Tiers

  12. List of major accelerator-based HENP experiments

  13. Size / frequency of basic activities

  14. Average physicist view • Mythology, culture, terminology varies a lot from one experiment to another. • BaBar • Object view or primary event store (Objectivity) • Event collection objects give primary access points to data • Event collection has list of references to all event components of interest • With 100,000 collections, how to organize them? • Ntuples & PAW for final data format, analysis tool • STAR (first year data, getting started) • A “production, trigger” is all reconstructed events for a trigger type with a certain version of code, (P00hg, central) • Access point is list of directory path’s below which all data are stored on disk • WZ will be setting up STACS • ROOT for data format and analysis tool • …

  15. What’s hard I, living with technology • Typical computer center today • A couple STK Powderhorn tape silos, HPSS or home-grown MSS • 1000 linux processors • Assortment of 100/1000 Gbps network • 50 TB disk (1000 spindles) • Network s/w for I/O (NFS, Objy AMS, RFIO, …) • AFS for distributed collaboration • Can make large RAID filesystems w/ network access • Faults can affect many nodes • stale NFS file handles • AFS faults affects nodes across country, work • Large RAID is $$$ • Desire to reduce effect of faults • Fewer faults • More tolerance • …

  16. What’s hard II,A clear view for average physicist What’s going on in this box?

  17. What’s hard II,A clear view for average physicist • What data is available? • “data” means • List of files? (like STAR) • Collection object w/ pointers to all events? (like BaBar) • “available” means • On disk? Where? • Exists? • Does it really have the filters and calibrations I need? • Is it the “official” version of the data? • …

  18. What’s hard III,Analysis of large datasets Dataset does not fit on disk, or requires parallel processing, oris large enough operation that chance of fault is high

  19. What’s hard III,Analysis of large datasets • Dataset does not fit on disk • Needs access s/w to couple w/ processing • SAM, STACS • Does performance meet demand?

  20. SAM (Sequential data Access via Meta-data) http://d0db.fnal.gov/sam/

  21. STACS http://sdm.lbl.gov/projectindividual.php?ProjectID=STACS

  22. What’s hard III,Analysis of large datasets • Dataset does not fit on disk • Needs access s/w to couple w/ processing • SAM, STACS • Does performance meet demand? • Needs parallel processing (not very hard) • Can not do analysis on private/personal machine • Schedule access to shared resource (CPU and disk) • Operation for a single analysis is large enough that faults occur • Need exception handling • Need workflow management to complete failed tasks or, at least, accurately report status

  23. Example shared nothing cluster http://www.ihep.ac.cn/~chep01/paper/4-026.pdf

  24. PPDG

  25. Summary • Faulty technology sets boundary conditions • Fault tolerant will expand boundaries of capabilities • Data management is coupled with processing • Visualization (access w/o processing) is minor in HENP • Need access to data when & where it is needed for processing • Working on data grid as context for data management • PPDG has SDM ISIC as one of the technology base projects

More Related