1 / 19

Grand Challenge in MDC2

Grand Challenge in MDC2. D. Olson, LBNL 31 Jan 1999 STAR Collaboration Meeting http://www-rnc.lbl.gov/GC/. People. GC system components. Sample User Code. key developers Henrik Nordberg (NERSC) query estimator Alex Sim (NERSC) query monitor Luis Bernardo (NERSC) cache manager

aletha
Download Presentation

Grand Challenge in MDC2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grand Challenge in MDC2 D. Olson, LBNL 31 Jan 1999 STAR Collaboration Meeting http://www-rnc.lbl.gov/GC/

  2. People GC system components Sample User Code • key developers • Henrik Nordberg (NERSC)query estimator • Alex Sim (NERSC)query monitor • Luis Bernardo (NERSC)cache manager • Jeff Porter (BNL-STAR)query object • Dave Malon (ANL)order-optimized iterator &gcaResources API • Dave Zimmerman, (LBL-STAR)tagDB • Stephen Johnson (BNL/SB-PHENIX) • Jie Yang (UCLA,LBL)testing Query Interface Storage Manager Event Data (Objectivity) Expt. code & data GC-MDC2: D. Olson, STAR Mtg

  3. Outline • Review of GC in MDC1 • MDC2 Features • Status of Software • Scenario of Usage • Post-MDC2 (ROOT-GC integration?) • Summary GC-MDC2: D. Olson, STAR Mtg

  4. User Code User Code User Code GCA Interface(QO, ooi) GCA Interface(QO, ooi) GCA Interface(QO, ooi) Multiple simultaneous queries Query3 Analysis codes GC system components Query2 Query1 Storage Manager Event Data (Objectivity) Expt. code & data GC-MDC2: D. Olson, STAR Mtg

  5. Legend File staged to local disk File staged to HPSS disk Event ID’s retrieved by iterator Start of pftp request File released by queries Start of query File purged from disk File ID End of query Symbol color identifies query Time GC-MDC2: D. Olson, STAR Mtg

  6. 3 queries q3 q1 q2 q1,2,3 3 queries with some shared files, time delay between each query, then the same 3 queries are repeated simultaneously. The cache was large enough to hold all files so the second time all queries run at processing speed rather than I/O speed. Detailed description of test results on GC web site under Results. GC-MDC2: D. Olson, STAR Mtg

  7. GC software features for MDC2 • Multiple event components • introduce “file bundles” - a set of events and their corresponding files • Files in multiple directories (use Objy catalog) • Time estimate for query • parallel query execution (multiple iterators, same token) • persistent state storage manager (can restart) (QE at least) • query object can save event collection • Linux analysis codes (Objectivity permitting) (PHENIX?) • More logging • See more details athttp://www-rnc.lbl.gov/GC/email/archive/msg00663.html GC-MDC2: D. Olson, STAR Mtg

  8. SELECT (component_name, component_name, ...)FROM dataset_nameWHERE (predicate_conditions_on_properties) Example: SELECT tracks, hitsFROM Run17WHERE glb_trk_tot>0 & glb_trk_tot<10 & n_vert_total<3 Also for event collections. Multiple named components EvntHdr tracks hits RAW Another component YAC(yet another component) DAQ fmt Objy Objy ROOT XDF GC-MDC2: D. Olson, STAR Mtg

  9. File bundles are delivered to the iterator in the analysis program. {F-1,4,6; E-1,2,3,4} {F-2,4,6; E-5,6,7,8,9} {F-2,5,6; E-10} {F-3,5,6; E-12,14,16,17,18} File Bundles File 6 Bundle Evt# File 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 File 1 File 2 File 3 File 5 EC 3 Evt Component 1 Evt Component 2 (Evts in query, evts not in query) GC-MDC2: D. Olson, STAR Mtg

  10. Real-time Logging/Monitoring Example Netlogger and Viewer from NERSC Data Intensive Computing Group (Tierney et al.) GC-MDC2: D. Olson, STAR Mtg

  11. Status of s/w for MDC2 • Query Estimator - Ready • Query Monitor - Done, Testing • Cache Manager - Ready • Logging - bug fixing • Query Object - few mods required • Order Optimized Iterator - few mods required • ROOT-QO interface - proof of principle done, needs work • ROOT TagDB interface to CG Index - Done GC-MDC2: D. Olson, STAR Mtg

  12. What needs to be done • Interface with STAR event collections • Interface with StAnalysisChain • OID --> build transient event • Define content of STAR Tag Database in ROOT GC-MDC2: D. Olson, STAR Mtg

  13. Analyze Tags, refine collection, (tight cuts) Query Index, make collection (loose cuts) Read Tag, make Index GC Index Usage Scenario GC-MDC2: D. Olson, STAR Mtg

  14. TreeViewer of Tag Database in ROOT Dave Zimmerman, Jan 31. We will look at interfacing this to Query Estimator, GC Index. GC-MDC2: D. Olson, STAR Mtg

  15. Post-MDC2 • Work on scalability issues after MDC2 • MDC2 does not stress scalability • Tighter integration with ROOT • no Objy? • Discussions w/ Rene Brun Jan. 25-27, 1999 at LBL GC-MDC2: D. Olson, STAR Mtg

  16. PROOF TTree TBranch TChain TEventList QO OOI Event ID’s TagDB Bit-mapped index Topics w.r.t ROOT-GC GC-MDC2: D. Olson, STAR Mtg

  17. QO/OOI - TChain • Iterating over the OID list retrieved from the storage manager for a file bundle is analgous to adding a file to a TChain and iterating over a TEventList for the selected events in that ROOT file. • It would be possible to replace OID as event identifier with ROOT file + event sequence number in file in a non-Objectivity implementation. GC-MDC2: D. Olson, STAR Mtg

  18. ROOT mods to Chain mods to EventList R.B. very interested in bit-mapped index GC try OOI w/ Objy OID and get ROOT component w/ filename+evt seq.# Consider non-Objy implementation (replace OID’s) What’s To Do w.r.t. ROOT GC-MDC2: D. Olson, STAR Mtg

  19. Summary • Basic MDC2 features ready. • Need more work on the interface w/ STAR. • Any candidates? (LBL post-doc open) • Need to clarify Objy/ROOT roles soon after MDC2. • Plan on close integration w/ ROOT. GC-MDC2: D. Olson, STAR Mtg

More Related