Introduction to COMA (and Metadata)

1. Introduction to COMA(and Metadata) Elizabeth Gallas Oxford ATLAS Software Tutorial June 23-27, 2013

2. Outline • Overview of User Applications (where does COMA fit in?) • Show • COMA Portal (grid certificate required on your browser !) • COMA Period Menu and Period Reports • COMA Report Menu • Many COMA reports are available from the same menu page • How to navigate to the COMA Report you need • Trigger metadata in COMA • And Trigger-related content in COMA Reports • Some new COMA Reports • Conclusions: Caveats, Status, Plans E.Gallas - COMA

3. ATLAS User Application Overview • Subsystem specific: driven by subsystem specific needs • Trigger: wide variety of tools and interfaces • Geometry DB: Detector Description Browser • Conditions DB: • RunQuery (in-depth Run info from Conditions DB) • ATLAS WEB DQ • COOL Tag Browser • Lumi Data Summary Reports (Luminosity, Beam) • GRLs (Good Run List xml) • And the Luminosity calculator • Beam Spot Summary • GANGA and PAthena • Panda / monitor • DQ2 Client • ... (not a complete list !) • Dedicated Metadata Catalogs (3) • AMI – Datasets, processing … other metadata • And the AMI Suite of services • COMA – Run/LB level Conditions and configuration metadata • Plus Conditions DB management metadata • TAGs (and TAG Catalog) – event level metadata • iELSSI and Suite of TAG Services Fundamental areas for every analysis ! See other talks on these topics. E.Gallas - COMA

4. COMA Introduction The COMA Project: • TWiki: ConditionsMetadata • Originally: built to support other systems. • Evolved: into a standalone system with its own interfaces. • Components: • Relational Database (Oracle) • Copied, refined, reduced, derived content from many sources • Unique content (not found elsewhere) •  Data Periods, Derived/Aggregated quantities • Unique Interfaces (Reports and Browsers) • COMA unique content and structure • Ability to aggregate quantities over user-defined temporal periods • Integration with AMI, TAG information and services • General philosophy: minimize typing … navigate w/ links btwn reports Beyond COMA: • COMA is part of general effort to consolidate/relate ATLAS Metadata • Strong ties with AMI and TAG DB • Any COMA information can be made available via “pyAMI” • COMA data/links now found in many ATLAS systems: • AMI, TAGs, DataQuality, RunQuery, Muon alignment, Conditions DB tools • Many links from ATLAS TWiki, physics pages, and personal pages E.Gallas - COMA

5. COMA Interfaces Portal https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html • Most popular • COMA Period • Other Reports • Conditions DB expert tool • (shade: grey) … operational … little current/active development E.Gallas - COMA

6. COMA: ATLAS Data Periods … + aggregating new content Painful to maintain, AFS dependent, Error prone Simple to enter, check integrity, more robust, available • A Data Period is a set of ATLAS Runs grouped for a purpose • Defined by Data Preparation Coordinators • Used in ATLAS data processing, assessment, and selection … • Each Period uniquely defined with a combination of • Project name (i.e. ‘data10_7TeV’) • Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …) • Before 2011, Data Periods were • Described on TWiki page • https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods • Stored in a file based system • Edited by hand by Data Prep Coordination (experts) • Structure evolved over 2010 with experience • This experience  valuable to decide/define long term solution • In 2011: Data Periods moved into COMA • Coordination/Effort: Data Prep, AMI, COMA experts • This made all aspects of Period definitions available programmatically • via pyAMI • Over time, COMA content has grown in many areas • Allows for more details reports and information to other systems • Enables aggregation of LB-wise information by Run, … Period. E.Gallas - COMA

7. https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php • Period Menu Purpose: • Shows an overview of all Data Prep defined Periods • Offer links: Period Description Reports containing verbal description and aggregated info about their Runs. • Choose the Period of interest: • By Year • e.g. all ‘2011’ • Or for ‘all years’ • By Project • e.g. ‘data12_8TeV’ • By Beam Energy or Type • e.g. ‘7TeV’ • By specific Period or Group • Click on the project and then Period of interest Help, Doc Links General feature of COMA Reports “highlighted” link opens expanding sections E.Gallas - COMA

8. Header: Title & Input criteria Links in Table column headers:  Short description of column Yellow links: show / hide members Links to select Period-wise Reports: COMA Run, COMA Period Trigger (new), RunQuery, AMI Container production Members of data12_8TeV.A are A1-A8 Note: some columns removed using the “customize report” feature (not shown) Hover on link: Indicates what will happen E.Gallas - COMA

9. COMA Logic: Input Menu General principle:  Input(s): determine which report is generated • Simple Examples: • Single Run number  COMA Single-Run Report • “Temporal” Criteria: Project, Period, Run, SMKs  COMA multi-Run Report • Level-specific Chain/Item(s) • Single Distinct name (EF_2mu6)  COMA Chain or L1 Item Report • Name with wildcards (L1_Tau*)  COMA Chain/Item Wildcard Report • Single SMK  COMA SMK Report • Combinations of “Temporal” & other inputs, Lists/Ranges, Wildcards, Lone wildcards • Outcome not so obvious ! • Next slide: try to summarize … • You might also read the documentation https://atlas-tagservices.cern.ch/RBR/runBrowserReport.html E.Gallas - COMA

10. What happens when … Temporal and (Chain and/or Stream): > One Run and • Level-specific chain (one or more)  LB-wise Prescale Report • Chain “lone wildcard” (*)  Run Trigger Report • Stream Type/Name “lone wildcard” (*)  Run Stream Report (NEW) [optional Level-specific chain: filter results] > Specific Project [other temporal] & • Level-specific chain (one or more) • Run-wise Prescale Report • Summary Prescale Report • Chain “lone wildcard” (*)  Project, Period Trigger Reports (NEW) [optional Stream Type/Name: filter results] Simple Inputs: • Single-Run Report • Multi-Run Report • Chain/Item(s) • Chain/Item Report • Wildcard Report  SMK Report “Temporal Criteria”: determines Runs considered E.Gallas - COMA

11. COMA Multi-Run Report for Period 8TeV.C2 Purpose: quick overview of any set of Runs COMA Magnet States. LB-wise transitions in COMA Run Report Links to select Run-wise reports: COMA Run, Run Trigger, Run Stream; RunQuery, and Lumi Data Summary COMA SMK Report E.Gallas - COMA

12. COMA Single-Run Report COMA Links • Links: • AMI • LumiDS • RunQuery • TrigConf E.Gallas - COMA

13. Event is recorded for offline physics analysis if it passes at least one trigger (and its prescale) “Metadata”viewof the Trigger “Event”: detector output during a single particle bunch crossing “Lots”: LHC max particle bunch crossing rate is 31.6 MHz “Fewer”: a few hundred events per second “Trigger” is a multi-component selection filter for events: • ATLAS detector hardware/electronics • Many subsystems … TDAQ • ATLAS software: HLT Release • Mostly C++ algorithms collected in a specific ATLAS Software Release • executed by the HLT (2nd,3rd trigger levels) • Trigger Menu: defines ~500 to 1000 Triggers • Every distinct Menu is assigned a unique integer ID • SMK: Super Master Key • Configurable input to the Trigger hardware and software • Specifies what logic or algorithms to execute, including configurable parameters (eg: thresholds) • Assigns each trigger to one/more output Streams • Menu (SMK) is FIXED during each Run (not incl. prescales) • Each trigger: 3 levels of pass OR fail • Each Event either passes or fails each Trigger • Prescales: Blind filter applied by TDAQ when above Trigger logic does not sufficiently reduce event output rate • Prescales can change during a Run (on LB boundary) • Integer identifiers are assigned to sets of prescales • Level 1 and HLT Prescale Keys “Lots” of “Events” “Trigger” Level 1 HLT: L2 HLT PT HLT: EF “Fewer” but more interesting Events E.Gallas - COMA

14. Trigger Metadata: just the tip of the iceberg • Trigger Configuration Metadata in COMA: • SMK Trigger Chains: EF chain, L2 Chain, L1 Item • Names, Versions, Bit Assignments, Streams, ReRun • LVL1, HLT Prescale Keys: • EF, L2, L1 prescales • EF, L2 Passthrough • Details behind Trigger Configuration and what is stored event-wise: need tools from the Trigger Experts • Understanding trigger execution and info storage • Algorithms, cuts, multiplicities, bunch groups • Dead-time veto, BCID / Train / Lumi dependence • Trigger objects related to trigger decisions • HLT algorithm Error codes • Trigger EDM and the Trigger Decision Tool • How to work with Chain Groups (Trigger ‘OR’s) • See the trigger related talks in Software Tutorials: https://indico.cern.ch/conferenceDisplay.py?confId=212225 COMA: Stores this metadata. Combines it w / Period,Run,Lumi data to provide unique reports (overview usage over many Runs) E.Gallas - COMA

15. Every Run has a Trigger Menu: • Every analysis is based on one/more triggers • Critical to understand it … so many tools … different levels of detail • From Trigger experts (see Trigger related tutorials) • http://atlas-runquery.cern.ch (RunQuery portal) • https://atlas-trigconf.cern.ch/ (TrigConf portal) • Trigger Tool (java based) • AtlCoolTrigger.py (command line based) • From COMA (show collected/derived trigger metadata) • Run Report: evolution of states of system over a Run (in LB ranges) • PrescaleKeys, Fills, StableBeam, Ready, Lumi, Magnets(w/links) • RunTrig Report: Run-wise Trigger chain “activity” flags 1. Prescale 2. Passthrough 3. Rerun Clearly shows which configured triggers are NOT active in a Run • Many reports display: • “Aggregate prescale” (L1*L2*EF PS) by Trigger or the Range of AggPS by Trigger, Run, Period, … • Prescale corrected Lumi by trigger • By LB, Run, SMK, Period (or other temporal criteria) • New content: Event count by Trigger (Run/Stream) – in progress E.Gallas - COMA

16. COMA Run Trigger Report Purpose: Show all Triggers in Menu for a Run • Shows which are “active” • Via Prescale, PT, Rerun … and which are disabled • Show range of AggPS, and PS Corrected Lumi • Note PS Corrected Lumi by trigger do NOT include deadtime or DQ !!! Always an upper limit ! • New Content ! Event count by trigger (per stream) E.Gallas - COMA

17. Run Stream ReportEF_*2mu* chainsin Run 208717 New ! Event Counts by Trigger • In selected Streams Assigned stream(s), Range of prescales E.Gallas - COMA

18. Period Trigger Reportdata13_2p76TeV.AllYear New ! Purpose: Summary of all active triggers (by Stream) with Stable Luminosity during selected temporal range • Period, Run, Date range • When each are “active” • Show range of prescales and PS Corrected Luminosity • In development: • aggregating Event Count by trigger E.Gallas - COMA

19. Some Caveats • Metadata is derived from the data • But occasionally does not contain all the details • Generally noted in the report and documentation • Luminosity in COMA is • During Stable Beam and ATLAS Ready flag condition • When that Lumi is tagged by experts in COOL • Some runs are special, intended only for experts • This is why a few runs are missing Luminosity in COMA • Prescale Corrected Lumi by trigger • Does NOT include dead time, Data quality, … !!! • Always an upper limit ! • COMA Lumi is intended to guide analysis • But is not the final luminosity with all corrections • Luminosity for any specific analysis: • Must, in the end, use official LumiCalc tools ! • NEW: Event Counts by Trigger (in each Run and Stream): • Loaded for Period, Stable, Ready Runs • But ~2% of the Run/Streams are under investigation • If the counts are missing (“NA”), I hope to recover it soon E.Gallas - COMA

20. Which is the best Run Report ? • Many Run/LB-level reports … which is best ? • It depends what you are looking for • COMA, RunQuery, Lumi Data Summary, AMI, ... • Contains some of the same information • Perhaps presented in a different way • Contains some unique information • Not found elsewhere • Optimized for its • Data source(s) • User/Experts needs • A balanced diet is recommended • Exploring each application is the only way to understand the benefits (& drawbacks) of each of them E.Gallas - COMA

21. Summary and Status • COMA contains Run/LB level Conditions/Configuration Metadata • Some unique content not found (or not easily found) elsewhere  There-by: unique set of interfaces and tools • Quick overview of • Periods, Runs, active streams, beam conditions, … • Overview of configuration / activation of specific triggers • Does this chain exist !?! • And when was it: configured ? … active ? … see any Luminosity ?… • Give me all chain names matching an input pattern • How they evolved over Run 1 … when was my trigger first prescaled ? • Offer Aggregation of various quantities across Periods, Runs • New content this week: Event Counts by trigger • For most Runs and Streams • Current development (Event Counts): • Content: Investigating problematic datasets (~2% where it is missing) • Reports: Aggregating event counts by trigger over temporal ranges • Comments/Questions on Metadata:  hn-atlas-physicsMetadata@cern.ch E.Gallas - COMA

22. Backup Slides:… show features of some of the other reports … E.Gallas - COMA

23. 2.Configuration Section: Shows where this element is configured: Super Master Key(s) Project (Summary) 3. Period Evolution: Shows chain/itembit, version evolution for EF_g20_loose chains during Period Runs 4. Activation Section: Shows Runs where this chain is ”active” Via prescale Via pass through Via rerun 1. EF_g20_loose 2. 3. 4. Jan 2013 E.Gallas- Metadata E.Gallas - COMA 23

24. COMA Chain Wildcard Reports The NEW version of these reports add integrated prescale corrected Lumi, PS ranges, and Run/Date ranges EF_*ZEE* L1_2EM*_MU* over all periods Jan 2013 E.Gallas- Metadata E.Gallas - COMA 24

25. Metadata definition: Concisely: “data about data” More precisely: “data used to describe the context, content or structure of data” Structural or Descriptive Metadata: used extensively in ATLAS … In fact: No process doesn’t use metadata “Descriptive” examples: Dataset name, Run Number, Channel number in some detector, TWiki Name, Trigger Names, dates/times, DQ Defect, … “Structural” examples: Number of runs or events or files, data volume, structure of compound objects, … Usage examples: Upstream: data taking with the correct calibrations … Downstream: user finding Events of interest … or Luminosity for an event sample Metadata challenges: Size/Scope of ATLAS data … Volume/Diversity of metadata Data/metadata: have grown organically as the experiment evolved Following evolution in Run1 Trying to anticipate changes for Run2 Offer a coherent / integrated view to physicists while Optimizing access / consonance for processing and analysis What is Metadata ? E.Gallas - COMA 25

26. General Comments: tutorials, applications, tools • ATLAS: a complex, multi-purpose experiment • General guidelines for analysis exist … but only take you so far • Every group, analysis, and user is different. • Having a broad understanding of the LATEST tools available will facilitate your analysis and deepen your understanding of data • Software tutorials (~ one week-long, ~ every 3 months): • Overview of ATLAS Computing for users • Geared toward new users • But also useful for others • Advertise a parade of interfaces and tools • Note: They are constantly evolving • So even experienced users look at latest tutorials • Keep up to speed with latest trends, new tools • Consider alternate tools to improve their analysis • One week is a limited time to take it all in • Many details in the slides should be explored independently • Most important !!! • Don’t be afraid to explore, compare, question, comment, rate • Please provide feedback (to the appropriate e-groups) !!! E.Gallas - COMA