1 / 22

Conditions Metadata for TAGs

Conditions Metadata for TAGs. Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010. Outline. History of Conditions Metadata / “Current” COMA Schema Evolution in context to overall TAG DB project

azura
Download Presentation

Conditions Metadata for TAGs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Conditions Metadata for TAGs Elizabeth Gallas, (Ryan Buckingham, Jeff Tseng) - Oxford ATLAS Software & Computing Workshop CERN – April 19-23, 2010

  2. Outline • History of Conditions Metadata / “Current” COMA Schema • Evolution in context to overall TAG DB project • Overview of Oracle Schemas for TAGs • TAG DB (Event-wise Metadata) • CATALOG (Dataset Catalogue Metadata) • COMA (COnditions MetadatA) • Upload procedure/Status • Documentation and Links • Fundamental design principles • Uses of Conditions Metadata • ELSSI usage • runBrowser – interface • runBrowserReport – interface • Functionality in our Future • Some unfortunate current operational issues • Conclusions Upon consideration of limited time, see next slide … Elizabeth Gallas - COMA

  3. Current COMA Tables Okay ! Too much detail for 15 minutes talk (don’t send me into the volcano) … See Backup/DocLinks for many details … I’m going to summarize in a few slides, then talk about the fun part: using COMA in ELSSI, runBrowser, runBrowserReport Elizabeth Gallas - COMA

  4. Documentation and Links COMA Documentation • COMA Schema https://gallas.web.cern.ch/gallas/COMA_Visio.pdf • COMA Tables https://gallas.web.cern.ch/gallas/COMA_Tables.html • COMA developer documentation (not user doc): https://gallas.web.cern.ch/gallas/Conditions_to_TAGs.html • COMA user documentation • Will be integrated into the interfaces below COMA User Interface Links (test versions) NOTE: The interfaces require a grid certificate in you browser NOTE: The interfaces are not in production server locations • runBrowser (2) Link: https://lxvm0341.cern.ch/tagservices/dev/rbucking/TAGBrowser/trunk/runBrowser2/runBrowser.php • runBrowserReport Link: https://lxvm0341.cern.ch/tagservices/dev/gallas/TAGS/runBrowser/runBrowserReport.php?q=find+run+142406 for Run 142406 (replace the Run Number in the URL with your Run of interest) Let’s try them after I mention some design principles Elizabeth Gallas - COMA

  5. General Design Principles (1) The fundamental components are • The COMA Relational Database tables • The runBroswerReport – the report interface for the COMA Tables • The runBrowser – the interface for RunLB selection using COMA Tables • COMA Tables: • Must provide information ELSSI needs to decode TAG attributes • Include information for both Online and MC Runs • TAGs for Online/MC have the same attributes (no MC truth) • Catalogue for Online/MC reflects similar processing workflows • Overall system must handle gracefully missing information • Upload select conditions for Runs of ‘analysis interest’ • not all Runs and not all Conditions • Refine/Correct/Derive conditions to form more effective criteria • runBrowserReport = web report interface to COMA Tables • Intended to display what COMA knows about each Run • Provides links to information in other systems • runQuery, AMI, Trigger, Data Quality …or reports using COOLCherryPy • Will be used by ELSSI & runBrowser to provide more information Elizabeth Gallas - COMA

  6. General Design Principles (2) • runBrowser = interface for RunLB selection using COMA Tables • Purpose: Make conditions metadata available as selection criteria in advance of analysis … Envisioned as the Run-level browser for ELSSI … current implementation makes it also available stand-alone. • Intermediate results may be what the user is looking for I.E. show me the Runs taken on this date or during this Fill. • Final output (clicking on “Finish” button): LB level criteria is applied at the final “Finish” stage. Output: A report showing the Run/LBs passing final criteria Output: An xml file (GoodRunList) which can be used by ELSSI etc. • runBrowser IS NOT runQuery (browser to all online Runs in COOL) • Enables not only Run selection by conditions criteria but also displays the possible values of remaining criteria and its relationship to other criteria • Criteria can be imposed in any order … some choices open selection to deeper criteria • Where appropriate: • Allows radio, checkbox, or text (command line) entry of criteria • Allow list and/or ranges of values, wildcards, case insensitivity ... • Incorporate features to customize rows displayed and other tricks to improve performance Elizabeth Gallas - COMA

  7. https://lxvm0341.cern.ch/tagservices/dev/rbucking/TAGBrowser/trunk/runBrowser2/runBrowser.phphttps://lxvm0341.cern.ch/tagservices/dev/rbucking/TAGBrowser/trunk/runBrowser2/runBrowser.php Note: “Under Construction”! Each section expands/collapses Purpose / Instructions Selection Summary  Starts out empty Selection Criteria Data Source Run Type Temporal Selection Project Name Trigger Master Key DAQ Configuration Run Number These are example criteria … more criteria will be added Test version of runBrowser Elizabeth Gallas - COMA

  8. Example demonstrates General Principles: There is no prescribed order of selection or mandatory selections Expand section of interest, make selection: available radio/checkbox or use the textbox to type a list or range of values Click Submit I chose Project “data09_900GeV”, then I see there are 99 Runs left I see their run and date range I see the criteria has appeared in the selection summary I could remove it with button click I see ALL the other sections have changed to reflect this criteria ! Look at the remaining 99 runs .. (next slide) Iterate any number of times  Elizabeth Gallas - COMA

  9. The Run Section includes links to other systems Click on the Run Number  generates the runBrowserReport Other links are to AMI, RunList, and Trigger Reports for that Run number Other related selections to be added to runBrowser2: Run Duration Number of LB Number of Events Recorded Click on the Run Section to open it … the run numbers appear Run selection is NOT mandatory … you can go onto FINISH without any Run explicitly selected runBrowserRun Number Section Next slide: runBrowserReport for Run number 142406 Elizabeth Gallas - COMA

  10. runBrowserReport Overview Each Yellow section expands … This report has 5 Primary sections, the Trigger section has subsections • General Run info • Run Date … • COMA Load Status • COMA loading messages • DQ LBSUMM assessments • COOL tagged/locked • Prescale Evolution • How many times did prescales change during the Run • Trigger section • Has an HLT summary • Counts of active/disabled • Has expanding subsections • HLT Chains (2 subsections) • Physics • Others • Level 1 Items Elizabeth Gallas - COMA

  11. COMA runBrowserReport : Overview Run 142406 Trigger Summary shows: has 23 active physics chains (of 162) Click on the Show/Hide link • to show/hide the grey rows of chain/items tables in respective subsections: • HLT (show/hide disabled chains) • Physics (complete EF-L2-L1 chains) • Others (commissioning chains) • Level 1 (show/hide passive items) The trigger tables show the prescale range and PS,PT,RR flags of the new derived “Run Aggregate prescale” COMA tables This new information allows: ELSSI to show only chains which are “active” during the Run. With report like this to show all chains. https://lxvm0341.cern.ch/tagservices/dev/gallas/TAGS/runBrowser/runBrowserReport.php?q=find+run+142406 Elizabeth Gallas - COMA

  12. Functionality in our Future (1) • COMA: Loading programs continue to be improved/refined • Some areas better resolved than others. We are making a list of the basics needed for production running … listed here are things initiated. • Select Runs with many of the criteria now available in runQuery • Project Name, Magnetic field, Beam conditions, DAQ config, Master Key (Trigger, LVL1/HLT PS Keys) … • Select Runs in particular Streams (with event counts available) • New: Define Run-wise Aggregate prescales, passthrough, rerun • Calculates Run-wise summary flag: PS, PT, Rerun per triggger • Useful to ELSSI to display active/passive/disabled triggers per Run … So that users do not select triggers in a Run which would yield 0 events because that trigger was not active • Useful for runBrowser to find the active chains for each Run • Working with Trigger group for validation • Data Quality upload • Can load tagged/locked LBSUMM DQ assessments • Schema and Loading – groundwork laid, interfaces underway • Can envision ways to include virtual flags with this information • Will work with DQ group for verification/refinement Elizabeth Gallas - COMA

  13. Functionality in our Future (2) • … more Trigger related criteria … • Select Runs with one/more particular trigger chains • Chains configured • Chains active via prescale (have prescale >0) • Chains active via passthrough (have passthrough > 0) • The RAW trigger result per event is a TAG DB attribute • Chains which may be RERUN. Some criteria will use information from the CATALOG Metadata: • Select Runs uploaded to TAG DB • These Runs ARE the Runs of analysis interest • The Runs ATLAS chooses to reconstruct and reprocess • These Runs follow the CREM deletion policy • Select Runs with a particular AMI tag • Processing related information is tracked in the Catalog Elizabeth Gallas - COMA

  14. Some Unfortunate Operational Notes • Recently, the server for runQuery and trigger web interfaces became unrecoverable. • Some services were moved to new locations • Some files on the failed server were lost and are difficult to recover • Some services were in the process of moving to a new server anyway. • This has had 2 effects on COMA/runBrowser/Report: • Some loading programs were using lost data so some runs are now missing that data • Plan: Rewrite those programs to use more stable source. • The web report links point to the old server • Plan: Fix the links to point to the new location. This is not a complaint about these particular web services We need to make COMA loading more robust we have a plan how to do that. Elizabeth Gallas - COMA

  15. Conclusions This talk reflects an evolving system … current information in the system is growing based on information available and use cases • Adding more dimensions to the Conditions data • With suitable relationships to facilitate queries • Making that criteria available in a dynamic useable interface We want to insure the Metadata is • complete enough to satisfy use cases while • reflecting accurately its limitations Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS i.e. runQuery, GoodRunList xml … This facilitates cross checks with other systems Continuous process: talking with various experts to ensure data integrity, completeness, compatibility w/other systems Elizabeth Gallas - COMA

  16. Backup Elizabeth Gallas - COMA

  17. Early history of this Conditions Metadata project • 2007: first Conditions Metadata tables were filled for MC simulation tests/prototype development • Streaming Test, Full Dress Rehearsal (FDR) exercises • Run/LB-wise conditions were collected from MC log files and other sources into relational DB tables (COMA) • And INSERTED into then new folders in Conditions DB (COOL) • Which formed the prototype for the ATLAS Luminosity calculation • Other Trigger/DAQ COOL folders defined at same time (RichardHawkings, Trigger/TDAQ) • 2008: COMA tables used by ELSSI prototype (still just MC) • Conditions, particularly trigger configuration … things not practical to store event-wise • 2009: start extracting Run/LB-wise information from COOL into COMA tables to facilitate efficient access to Conditions Metadata by ELSSI for Online Runs Elizabeth Gallas - COMA

  18. History of this Conditions Metadata project 2009 … (continued … #1) … • runBrowser: new interface for finding Runs sharing common conditions • Initially development tool: check data integrity/relationships of COMA tables • We (TAG developers) realized this runBrowser would be more generally useful … separate Run-browsing (runBrowser) from Event-browsing (ELSSI) … make runBrowser a stand-alone tool • May2009: TAG group developed first DTD for GoodRunList XML • This xml was how runBrowser would communicate to ELSSI the Run/LB selection and selections criteria. • This DTD has since been taken up/over by the ATLAS experiment to communicate good Run/LB ranges between subsystems. • NOTE: runBrowser is NOT a replacement for runQuery (the web based browser to all online Runs in the Conditions DB) • runBrowserReport development started • Which will help ELSSI and runBrowser describe underlying COMA info and provides more links to other ATLAS systems • COMA tables expanded to include more conditions in anticipation of use cases / expand selection criteria Elizabeth Gallas - COMA

  19. History of this Conditions Metadata project 2009 … (continued … #2) … • As data has been added to COMA tables • runBrowser evolved accordingly • At various points … selection criteria included: • Data Source (online,simulation), Data Type, Run Number, Duration, Number of LBs, temporal, DAQ configuration, Filename tag (Project Name), Trigger Super Master Key, Trigger LVL1 and/or HLT Prescale Key, DQ assessments from LBSUMM detectors… • Useful for initial purpose and query development This implementation demonstrated the basic design principles … but the code was not generally well constructed … Elizabeth Gallas - COMA

  20. More History of this Conditions Metadata project 2009 … (continued #3) … into 2010 … • October 2009: Ryan Buckingham (Oxford) joined project • Now up to speed on many aspects of the system. • primarily working on runBrowser • Reorganization/enhancement of runBrowser code • converting components to object oriented structure improved • internal communication • inclusion of criteria to the GRL xml to pass to ELSSI • Facilitates eventual system to save user criteria for future sessions and sharing with others, physics groups • Making addition of selection criteria more modular • Adding features and functionality to facilitate syntax alignment with runQuery • Improving look/feel of interface Elizabeth Gallas - COMA

  21. Evolution of overall TAG DB project (last year) • TAG DB / ELSSI: evolved to a distributed architecture • Realization: • Not possible to upload all TAGs at any one Oracle site • Advantageous to have some TAGs at multiple sites … • So ELSSI: needs to know which TAGs are uploaded at which voluntary sites • Add new relational schema TAGS CATALOG (Elisabeth Vinek) • Contains processing/upload information needed to deploy distributed TAG services on the grid • TAG DB / CATALOG / COMA Schemas work together • Common threads include: Run Number, Stream, Project Name … • COMA tables also use the CATALOG • Upload only Run/LB metadata for Runs in CATALOG • This reduces handling of conditions anomalies  allowing us to focus on Runs of ‘event analysis interest’ • Steps in Database loading – ideally within hours of reconstruction • TAGs uploaded to Oracle • CATALOG tables updated • COMA tables updated Now changing to use Tier-0 DB to advance the COMA loading phase Elizabeth Gallas - COMA

  22. Oracle Database: TAG DB and associated metadata tables RUNS 3. COMA tables 1. TAG DB: Event-wise metadata Run LB Event … • TAG DB Event-wise metadata tables • Stores Event-wise attributes: • electron (Et, eta, phi …) • muon (Et, eta, phi …) … and references to RAW, ESD, AOD files • Official data processing chain: RAW  ESD AOD TAG files • Data Catalogue tables • Stores information on file and dataset processing and location • Project name • AMI tag (what processing occurred)… • Sources: AMI (ATLAS Metadata Catalogue, Tier0 … • ‘COMA’ (COnditions MetadatA) tables • Conditions of data taking • Beam conditions • Trigger and DAQ conditions • Magnetic field … • Various sources: Conditions DB, Log files, xml files, email… ORACLE DB 2. DATA Catalogue Tables Elizabeth Gallas - COMA

More Related