Introduction to COMA (and Metadata)

1. Introduction to COMA(and Metadata) Elizabeth Gallas Oxford ATLAS Software Tutorial March 11-15, 2013

2. Outline • What is “Metadata” ? • Challenges in ATLAS • Overview of User Applications • Subsystem specific applications (many) • Dedicated Metadata repositories (3) • AMI • COMA (main topic today) • TAGs • Show • COMA Portal (grid certificate required on your browser !) • COMA Period Menu and Period Reports • Trigger metadata in COMA • And Trigger-related content in COMA Reports • Other COMA Report highlights • Conclusions: Caveats, Status, Plans E.Gallas - COMA

3. Metadata definition: Concisely: “data about data” More precisely: “data used to describe the context, content or structure of data” Structural or Descriptive Metadata: used extensively in ATLAS … In fact: No process doesn’t use metadata “Descriptive” examples: Dataset name, Run Number, Channel number in some detector, TWiki Name, Trigger Names, dates/times, DQ Defect, … “Structural” examples: Number of runs or events or files, data volume, structure of compound objects, … Usage examples: Upstream: data taking with the correct calibrations … Downstream: user finding Events of interest … or Luminosity for an event sample Metadata challenges: Size/Scope of ATLAS data … Volume/Diversity of metadata Data/metadata: have grown organically as the experiment evolved Following evolution in Run1 Trying to anticipate changes for Run2 Offer a coherent / integrated view to physicists while Optimizing access / consonance for processing and analysis What is Metadata ? E.Gallas - COMA 3

4. ATLAS User Application Overview • Subsystem specific: driven by subsystem specific needs • Trigger: wide variety of tools and interfaces • Geometry DB: Detector Description Browser • Conditions DB: • RunQuery (in-depth Run info from Conditions DB) • ATLAS WEB DQ • COOL Tag Browser • Lumi Data Summary Reports (Luminosity, Beam) • GRLs (Good Run List xml) • And the Luminosity calculator • Beam Spot Summary • GANGA and PAthena • Panda / monitor • DQ2 Client • ... (not a complete list !) • Dedicated Metadata Catalogs (3) • AMI – Datasets, processing … other metadata • And the AMI Suite of services • COMA – Run/LB level Conditions and configuration metadata • Plus Conditions DB management metadata • TAGs (and TAG Catalog) – event level metadata • iELSSI and Suite of TAG Services Fundamental areas for every analysis ! See other talks on these topics. E.Gallas - COMA

5. COMA Introduction The COMA Project: • TWiki: ConditionsMetadata • Originally: built to support other systems. • Evolved: into a standalone system with its own interfaces. • Components: • Relational Database (Oracle) • Copied, refined, reduced, derived content from many sources • Unique content (not found elsewhere) •  Data Periods, Derived/Aggregated quantities • Unique Interfaces (Reports and Browsers) • COMA unique content and structure • Ability to aggregate quantities over user-defined temporal periods • Integration with AMI, TAG information and services • General philosophy: minimize typing … navigate w/ links btwn reports Beyond COMA: • COMA is part of general effort to consolidate/relate ATLAS Metadata • Strong ties with AMI and TAG DB • Any COMA information can be made available via “pyAMI” • COMA data/links now found in many ATLAS systems: • AMI, TAGs, DataQuality, RunQuery, Muon alignment, Conditions DB tools • Many links from ATLAS TWiki, physics pages, and personal pages E.Gallas - COMA

6. COMA Interfaces Portal https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html • Most popular • COMA Period • Other Reports • Conditions DB expert tool • (shade: grey) … operational … little current/active development E.Gallas - COMA

7. COMA: ATLAS Data Periods … + aggregating new content Painful to maintain, AFS dependent, Error prone Simple to enter, check integrity, more robust, available • A Data Period is a set of ATLAS Runs grouped for a purpose • Defined by Data Preparation Coordinators • Used in ATLAS data processing, assessment, and selection … • Each Period uniquely defined with a combination of • Project name (i.e. ‘data10_7TeV’) • Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …) • Before 2011, Data Periods were • Described on TWiki page • https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods • Stored in a file based system • Edited by hand by Data Prep Coordination (experts) • Structure evolved over 2010 with experience • This experience  valuable to decide/define long term solution • In 2011: Data Periods moved into COMA • Coordination/Effort: Data Prep, AMI, COMA experts • This made all aspects of Period definitions available programmatically • via pyAMI • Since then, COMA content has grown in many areas • Allows for more details reports and information to other systems • Enables aggregation of LB-wise information by Run, … Period. E.Gallas - COMA

8. https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php • Period Menu Purpose: • Shows an overview of all Data Prep defined Periods • Offer links: Period Description Reports containing verbal description and aggregated info about their Runs. • Choose the Period of interest: • By Year • e.g. all ‘2011’ • Or for ‘all years’ • By Project • e.g. ‘data12_8TeV’ • By Beam Energy or Type • e.g. ‘7TeV’ • By specific Period or Group • Click on the project and then Period of interest Help, Doc Links General feature of COMA Reports “highlighted” link opens expanding sections E.Gallas - COMA

9. Header: Input criteria Links in Table column headers:  Short description of column Links: to COMA,RunQuery, AMI Container production highlight links: show / hide period members Members of data12_8TeV.A are A1-A8 Hover on link: Indicates what will happen Note: some columns removed using the “customize report” feature (not shown) E.Gallas - COMA

10. COMA Multi-Run Report for Period 8TeV.C2 Purpose: quick overview of any set of Runs COMA Magnet States. LB-wise transitions in COMA Run Report COMA SMK and CRTrig Reports: Show all Triggers in the Menu. CRTrig Report adds Lumi per trigger and which trigs are “active” Links: COMA Run, Coma Run Trigger, RunQuery, and Lumi Data Summary E.Gallas - COMA

11. Event is recorded for offline physics analysis if it passes at least one trigger (and its prescale) “Metadata”viewof the Trigger “Event”: detector output during a single particle bunch crossing “Lots”: LHC max particle bunch crossing rate is 31.6 MHz “Fewer”: a few hundred events per second “Trigger” is a multi-component selection filter for events: • ATLAS detector hardware/electronics • Many subsystems … TDAQ • ATLAS software: HLT Release • Mostly C++ algorithms collected in a specific ATLAS Software Release • executed by the HLT (2nd,3rd trigger levels) • Trigger Menu: defines ~500 to 1000 Triggers • Every distinct Menu is assigned a unique integer ID • SMK: Super Master Key • Configurable input to the Trigger hardware and software • Specifies what logic or algorithms to execute, including configurable parameters (eg: thresholds) • Assigns each trigger to one/more output Streams • Menu (SMK) is FIXED during each Run (not incl. prescales) • Each trigger: 3 levels of pass OR fail • Each Event either passes or fails each Trigger • Prescales: Blind filter applied by TDAQ when above Trigger logic does not sufficiently reduce event output rate • Prescales can change during a Run (on LB boundary) • Integer identifiers are assigned to sets of prescales • Level 1 and HLT Prescale Keys “Lots” of “Events” “Trigger” Level 1 HLT: L2 HLT PT HLT: EF “Fewer” but more interesting Events E.Gallas - COMA

12. Trigger Metadata: just the tip of the iceberg • Trigger Configuration Metadata in COMA: • SMK Trigger Chains: EF chain, L2 Chain, L1 Item • Names, Versions, Bit Assignments, Streams, ReRun • LVL1, HLT Prescale Keys: • EF, L2, L1 prescales • EF, L2 Passthrough • Details behind Trigger Configuration and what is stored event-wise: need tools from the Trigger Experts • Understanding trigger execution and info storage • Algorithms, cuts, multiplicities, bunch groups • Dead-time veto, BCID / Train / Lumi dependence • Trigger objects related to trigger decisions • HLT algorithm Error codes • Trigger EDM and the Trigger Decision Tool • How to work with Chain Groups (Trigger ‘OR’s) • See the trigger related talks in Software Tutorials: https://indico.cern.ch/conferenceDisplay.py?confId=212225 COMA: Stores this metadata. Combines it w / Period,Run,Lumi data to provide unique reports (overview usage over many Runs) E.Gallas - COMA

13. Every Run has a Trigger Menu: • Every analysis is based on one/more triggers • Critical to understand it … so many tools … different levels of detail • From Trigger experts (see Trigger related tutorials) • http://atlas-runquery.cern.ch (RunQuery portal) • https://atlas-trigconf.cern.ch/ (TrigConf portal) • Trigger Tool (java based) • AtlCoolTrigger.py (command line based) • From COMA (show collected/derived trigger metadata) • Run Report: evolution of states of system over a Run (in LB ranges) • PrescaleKeys, Fills, StableBeam, Ready, Lumi, Magnets(w/links) • RunTrig Report: Run-wise Trigger chain “activity” flags 1. Prescale 2. Passthrough 3. Rerun Clearly shows which configured triggers are NOT active in a Run • Many reports display: • “Aggregate prescale” (L1*L2*EF PS) by Trigger or the Range of AggPS by Trigger, Run, Period, … • New content: prescale corrected Lumi by trigger • By LB, Run, SMK, Period (or other temporal criteria) E.Gallas - COMA

14. COMA Single-Run Report COMA Links RunQuery Links TrigConf Links E.Gallas - COMA

15. COMA Run Trig Report Show all Triggers in Menu for a Run • Shows which are “active” • Via Prescale, PT, Rerun … and which are disabled • Show range of AggPS, and PS Corrected Lumi Caveats: Lumi in COMA is only during Stable Beam Ready conditions. PS Corrected Lumi by trigger do NOT include deadtime or DQ !!! Always an upper limit ! E.Gallas - COMA

16. 2.Configuration Section: Shows where this element is configured: Super Master Key(s) Project (Summary) 3. Period Evolution: Shows chain/itembit, version evolution for EF_g20_loose chains during Period Runs 4. Activation Section: Shows Runs where this chain is ”active” Via prescale Via pass through Via rerun 1. EF_g20_loose 2. 3. 4. Jan 2013 E.Gallas- Metadata E.Gallas - COMA 16

17. COMA Chain Wildcard Reports The NEW version of these reports add integrated prescale corrected Lumi, PS ranges, and Run/Date ranges EF_*ZEE* L1_2EM*_MU* over all periods Jan 2013 E.Gallas- Metadata E.Gallas - COMA 17

18. Some Caveats Metadata is derived from the data But occasionally does not contain all the details Generally noted in the documentation when known One important caveat for users! Luminosity in COMA is During Stable Beam and ATLAS Ready flag condition When that Lumi is tagged by experts in COOL Some runs are special, intended only for experts This is why a few runs are missing Luminosity in COMA Prescale Corrected Lumi by trigger Does NOT include dead time, Data quality, … !!! Always an upper limit ! COMA Lumi is intended to guide analysis But is not the final luminosity with all corrections Luminosity for any specific analysis: Must, in the end, use official LumiCalc tools ! E.Gallas - COMA

19. Conclusions • Understanding ATLAS Metadata is important: • To use subsystem specific tools and • Understand how that information relates to the big picture • 3 systems dedicated to collecting ATLAS Metadata (AMI, COMA, TAGs) • They contain: • information found in other systems • Plus unique data not found (or not easily found) elsewhere  There-by: unique interfaces and tools • COMA contains Run/LB level Conditions/Configuration Metadata • Quick overview of Periods, Runs, active streams, beam conditions, … • Overview of configuration / activation of specific triggers • How they evolved over Run 1 … when was my trigger first prescaled ? • Offer Aggregation of various quantities across Periods, Runs • COMA Plans: • Adding event counts: Stream, Trigger • Enhance aspects of MC metadata (LS1) • Enhance: Content, Functionality, Usability • Comments/Questions on Metadata:  hn-atlas-physicsMetadata@cern.ch E.Gallas - COMA

20. General Comments: tutorials, applications, tools • ATLAS: a complex, multi-purpose experiment • General guidelines for analysis exist … but only take you so far • Every group, analysis, and user is different. • Having a broad understanding of the LATEST tools available will facilitate your analysis and deepen your understanding of data • Software tutorials (~ one week-long, ~ every 3 months): • Overview of ATLAS Computing for users • Geared toward new users • But also useful for others • Advertise a parade of interfaces and tools • Note: They are constantly evolving • So even experienced users look at latest tutorials • Keep up to speed with latest trends, new tools • Consider alternate tools to improve their analysis • One week is a limited time to take it all in • Many details in the slides should be explored independently • Most important !!! • Don’t be afraid to explore, compare, question, comment, rate • Please provide feedback (to the appropriate e-groups) !!! E.Gallas - COMA