LHC Physics Analysis and Databases

LHC Physics Analysis and Databases Maaike Limper

Introduction • New Oracle sponsored CERN OpenLab fellow: Maaike Limper, started January 2012 • Project outline: Investigate possibility of doing LHC-scale data reconstruction and/or physics analysis within an Oracle database • MINI-CV:MaaikeLimper has a master in Physics from the University of Amsterdam, and a PhD in Particle Physics completed with Nikhef(The Dutch National Institute for Particle Physics). As a part of her PhD and sub-sequent post-doc she worked on the ATLAS experiment,one of the experiments that measures events produced by the Large Hadron Collider at CERN. During her work as an ATLAS physicist, Maaike worked on the construction of the silicon tracker, developed track and vertex reconstruction algorithms and participated in the analysis of the first LHC data recorded with the ATLAS detector. Maaikewas also involved in developing some of the ATLAS data-taking conditions databases for the pixel detector. As Prompt Reconstruction Coordinator for ATLAS, she was responsible for the reconstruction of all data recorded by the ATLAS detector. As of January 2012 she works full-time as an Oracle funded Openlab fellow for the CERN IT department. LHC physics analysis and databases - M. Limper

LHC Large Hadron Collider at CERN • Four main experiments recording events produced by the Large Hadron Collider: ATLAS, CMS, LHCb and ALICE • Examples of LHC-scale data-processing from my experience with the ATLAS experiment Lake Geneva CERN ATLAS LHC physics analysis and databases - M. Limper

LHC-scale data processing data acquisition event data taking reconstruction event reconstruction analysis event analysis raw data analysis objects (extracted per physics topic) simulated raw data event summary data ntuple1 generationsimulationdigitization event simulation ntuple2 ntupleN interactive physics analysis (thousands of users!) LHC physics analysis and databases - M. Limper

data acquisition event data taking Event data taking In 2011 LHC delivered 5.61 fb-1 of p-p collision data raw data • ~20 thousand events that produced a Standard Model Higgs with a mass of 150 GeV • ~300 billion inelastic proton-proton interactions • ATLAS uses a flexible trigger menu to determine which events are interesting enough to record… • ATLAS recorded 1.6 billion events in 2011 LHC physics analysis and databases - M. Limper

ATLAS data reconstruction Raw data (detector hits, energy depositions etc) reconstructed by the ATLAS (C++) software framework • Reconstruction task examples: • Fit particle trajectories from hits measured in the inner detector • Cluster energy deposits measured in the calorimeter to reconstruct “jets” of particles • Fit trajectory from hits in muonspectrometer • Combine track information to determine muon candidate from interaction point LHC physics analysis and databases - M. Limper

ATLAS data reconstruction Raw data (detector hits, energy depositions etc) reconstructed by the ATLAS (C++) software framework • Reconstruction task examples: • Fit particle trajectories from hits measured in the inner detector • Cluster energy deposits measured in the calorimeter to reconstruct “jets” of particles • Fit trajectory from hits in muon spectrometer • Combine track information to determine muon candidate from interaction point LHC physics analysis and databases - M. Limper

ATLAS real physics event example: ATLAS data analysis Reconstruction focuses on creating physics objects from the information measured in the detector Analysis focuses on interpreting information from the reconstructed objects to determine what type of event took place Z->mmcandidate, mmm=93.4 GeV Example: apply quality criteria on muon candidates and calculate the invariant mass from the sum of the muon 4-momentum to find a Z-boson candidate LHC physics analysis and databases - M. Limper

ATLAS data reconstruction • ATLAS uses flexible trigger menus to reduce data-taking rate to ~300 recorded events/second (2011 rate) • Tier-0 computing center at CERN has ~3000 CPUs available to reconstruct ATLAS data while it is recorded • Initial data gets reconstructed twice: • “Express reconstruction” during data-taking • “Bulk reconstruction” 36 hours after end of run, using beamspot and calibration constants derived from express reconstruction number of CPUs increased during busy periods express bulk reco Reprocessing campaignsevery 2/3 months to re-reconstruct all data with latest version of reconstruction software LHC physics analysis and databases - M. Limper

Event Simulation • In addition to the real physics events, physicists require simulated (MC) events to compare/test/understand their results • Each physics group requests sets of signal and background samples ~100 million simulated events requested • GenerationSimulationDigitizationReconstruction • During each ATLAS reprocessing campaign new version of all simulation samples are provided simulated raw data generationsimulationdigitization event simulation LHC physics analysis and databases - M. Limper

Data analysis • Physics analysis at LHC is mainly done with ROOT • C++ • analysis tools (plotting/fitting/statistical analysis) • ROOT-ntuples are centrally produced by physics groups from previously reconstructed event summary data • Each physics group determines specific content of ntuple • Physics objects to include • Level of detail to be stored per physics object • Event filter and/or pre-analysis steps event summary data ntuple1 ntuple2 ntupleN • Variables stored for each event in the form of: • scalar (example: missing energy, number of reconstructed muons) • vectors (example: energy, direction, momentum of reconstructed muons) • vector-of-vectors (example: position of hits on reconstructed muons) LHC physics analysis and databases - M. Limper

Physics Analysis from DB Benchmark Physics Analys in an Oracle DB: • Simplified version of HZbbllanalysis (search for standard model Higgs boson produced in association with a Z-boson) • Select muon-candidates to recontruct Z-peak • Select b-jet-candidates to reconstruct Higgs-peak • Signal sample: 29887 events (3 ntuples) • Background sample (Z->mumu+jets): 289916 events (30 ntuples) • Use ntuple defined by ATLAS Top Physics Group: ”NTUP_TOP” • 4212 physics attributes per event • Initial challenges: • Large number of attributes, many of which are vector-type, difficult to implement in a single table, so I divided data over multiple tables • Need to select data with SQL-queries instead of C++ code containing a loop over all events in the file LHC physics analysis and databases - M. Limper

Physics DB • Initial DB implementation holds 695 out of 4212 variables (16.5%): • “EventData”-table: 184 columns 184 event-related variables (scalar), primaryKey=(RunNumber,EventNumber) • “muon”-table: 271 columns, 268 muon-related variables (muon-vector content), primaryKey=(muonId,RunNumber,EventNumber), foreignKey=(RunNumber,EventNumber) • “jet”-table: 193 columns 190 jet-related variables (jet-vector content), primaryKey=(jetId,RunNumber,EventNumber), foreignKey=(RunNumber,EventNumber) • “MET”-table: 55 columns 53 MET (Missing Transverse Energy)-related variables (scalar), primaryKey=(RunNumber,EventNumber), foreignKey=(RunNumber,EventNumber) ROOT-ntuplesize is 880 MB Current DB-size per stored ntuple(16.5% of contents) is ~ 200 MB Full DB-size would be ~1.2 GB per ntuple~2.6 GB per ntuple LHC physics analysis and databases - M. Limper

Physics Analysis • Simplified version of HZbbll analysis: • muon selection: “IsMuon”-function to return TRUE, include requirement pT>20 GeV and |η|<2.4 plus several requirement on hits and holes on tracks • Require exactly 2 selected muons per event • b-jet selection: tranverse momentum greater than pT>25 GeV, |η|<2.5 and “flavour_weight_Comb”>1.55 (to select b-jets) • Require exactly 2 selected b-jets per event • Require 1 of the 2 b-jets to have pT>45 GeV • Plot “invariant mass” of muons (Z-peak) and of b-jets (Higgs-peak) • Two versions of this analysis: • Standard ntuple-analysis in ROOT (C++) using locally stored ntuples • Analysis from Oracle Physics DB running on same machine as DB and using functions implemented in “PHYSANALYSIS” PL/SQL-package: “IsMuon”, “InvariantMassLeptons, “InvariantMassJets” LHC physics analysis and databases - M. Limper

Physics Analysis in SQL Done using SQL-query making temporary tables for different selections and joining data from different tables via “EventNumber” with selectedmuon as (select "muon_i","EventNumber","RunNumber","E","px","py","pz" from "muon" where MLIMPER.PHYSANALYSIS.IS_MUON("muon_i", "pt", "eta", "phi", "E", "me_qoverp_exPV", "id_qoverp_exPV","me_theta_exPV", "id_theta_exPV", "id_theta","isCombinedMuon", "isLowPtReconstructedMuon","tight","expectBLayerHit", "nBLHits", "nPixHits","nPixelDeadSensors", "nPixHoles", "nSCTHits","nSCTDeadSensors", "nSCTHoles","nTRTHits", "nTRTOutliers",0,20000.,2.4) = 1 ), with selectedmuon as (select "muon_i","EventNumber","RunNumber","E","px","py","pz" from "muon" where MLIMPER.PHYSANALYSIS.IS_MUON("muon_i", "pt", "eta", "phi", "E", "me_qoverp_exPV", "id_qoverp_exPV","me_theta_exPV", "id_theta_exPV", "id_theta","isCombinedMuon", "isLowPtReconstructedMuon","tight","expectBLayerHit", "nBLHits", "nPixHits","nPixelDeadSensors", "nPixHoles", "nSCTHits","nSCTDeadSensors", "nSCTHoles","nTRTHits", "nTRTOutliers",0,20000.,2.4) = 1 ), selectedeventsmuon as (select "EventNumber", COUNT(*) as "mu_sel_n" from selectedmuon group by "EventNumber" HAVING COUNT(*)=2), selectedeventsmuon as (select "EventNumber", COUNT(*) as "mu_sel_n" from selectedmuon group by "EventNumber" HAVING COUNT(*)=2), selectedbjet as (select "jet_i","EventNumber","RunNumber","E","pt","phi","eta" from "jet" INNER JOIN selectedeventsmuon USING("EventNumber") where "pt"/1000>25 and abs("eta")<2.5 and "fl_w_Comb">1.55 ), selectedbjet as (select "jet_i","EventNumber","RunNumber","E","pt","phi","eta" from "jet" INNER JOIN selectedeventsmuon USING("EventNumber") where "pt"/1000>25 and abs("eta")<2.5 and "fl_w_Comb">1.55 ), selectedevents as (select "EventNumber", COUNT(*) as "jet_sel_n" from selectedbjet group by "EventNumber" HAVING COUNT(*)=2) selectedevents as (select "EventNumber", COUNT(*) as "jet_sel_n" from selectedbjet group by "EventNumber" HAVING COUNT(*)=2) select MLIMPER.PHYSANALYSIS.INV_MASS_LEPTONS(mu1."E",mu2."E",mu1."px",mu2."px",mu1."py", mu2."py",mu1."pz",mu2."pz")/1000. as "DiMuonMass", MLIMPER.PHYSANALYSIS.INV_MASS_JETS(jet1."E",jet2."E",jet1."pt",jet2."pt",jet1."phi",jet2."phi",jet1."eta",jet2."eta")/1000. as "DiJetMass" from selectedmuon mu1, selectedmuon mu2, selectedbjet jet1, selectedbjet jet2, selectedeventsevSel where mu1."muon_i"<mu2."muon_i" and mu1."EventNumber"=evSel."EventNumber" and mu2."EventNumber"=evSel."EventNumber" and jet1."jet_i"<jet2."jet_i" and jet1."EventNumber"=evSel."EventNumber" and jet2."EventNumber"=evSel."EventNumber" and jet1."pt"/1000.>45. select MLIMPER.PHYSANALYSIS.INV_MASS_LEPTONS(mu1."E",mu2."E",mu1."px",mu2."px",mu1."py", mu2."py",mu1."pz",mu2."pz")/1000. as "DiMuonMass", MLIMPER.PHYSANALYSIS.INV_MASS_JETS(jet1."E",jet2."E",jet1."pt",jet2."pt",jet1."phi",jet2."phi",jet1."eta",jet2."eta")/1000. as "DiJetMass" from selectedmuon mu1, selectedmuon mu2, selectedbjet jet1, selectedbjet jet2, selectedeventsevSel where mu1."muon_i"<mu2."muon_i" and mu1."EventNumber"=evSel."EventNumber" and mu2."EventNumber"=evSel."EventNumber" and jet1."jet_i"<jet2."jet_i" and jet1."EventNumber"=evSel."EventNumber" and jet2."EventNumber"=evSel."EventNumber" and jet1."pt"/1000.>45. LHC physics analysis and databases - M. Limper

Physics Analysis benchmark Output of SQL-query send to ROOT to produce standard root-histograms: ROOT-macro using original ntuples as input produces identical histograms: LHC physics analysis and databases - M. Limper

Physics Analysis in DB Time to produce plots from physics DB vs from ntuple-files Sample #events#sel.events time from DB time from ntuple HZbbll 9993 421 5 s ?? 7 s HZbbll 29987 1326 95 s 14 s Z+2jets 289916 170 85 s 110s • Both tests done onCERN virtual machine, 2 GB RAM, 2 CPU, SLC5 64-bit • Average time of analysis measured after reboot of virtual machine • Speed from ntuple scales depends on: • number of files • number of used ntuple-branches (=physics-attributes) • DB-speed depends on: • clever implementation of SQL-query: I’m not (yet) an SQL-guru… • table-size:select from jet-table much slower than from muon-table, asmore jet-objects stored per event LHC physics analysis and databases - M. Limper

Physics Analysis in DB • First implementation of Physics Analysis in Oracle DB • Data in multiple tables • SQL-query can reproduce selection in loop over events • Analysis from DB speed similar to original ntuple-analysis but many complexities still not implemented… Possible space-gain for DB version of analysis data: Each physics group optimized their own ntuple-size based on physics and level of detail required for their analysis, but sum of different ntuple contains duplicate info Physics Analysis DB would contains all physics objects, divided over multiple tables, each physics group can choose which tables to use analysis objects (extracted per physics topic) event summary data ntuple1 ntuple2 ntupleN LHC physics analysis and databases - M. Limper

Physics Analysis in DB • First implementation of Physics Analysis in Oracle DB • data in multiple tables • SQL-query can reproduce selection in loop over events • Analysis from DB slower than original ntuple-analysis and many complexities still not implemented… Possible space-gain for DB version of analysis data: Each physics group optimized their own ntuple-size based on physics and level of detail required for their analysis, but sum of different ntuple contains duplicate info Physics Analysis DB would contains all physics objects, divided over multiple tables, each physics group can choose which tables to use analysis objects stored in database event summary data physicsDB LHC physics analysis and databases - M. Limper

Physics Analysis in DB Space requirement for realistic physics DB (order of magnitude): ~1.22.6 GB per ntuple (10k events) ~2 billion events data+simulation in 2011~240 TB520 TB of analysis data ~10 revisions (reconstruction software versions) actively analyzed at a given time (more revisions may need to be archived) ~2400 TB of analysis data ~factor 4 more data from LHC expected in 2012 Analysis DB would need to be accessible by thousands of users! duplicates of DB at different analysis sites needed analysis objects stored in database event summary data physicsDB LHC physics analysis and databases - M. Limper 23

To be continued… Reconstructionversus Analysisof LHC data Analysis data is relatively easily organized in tables and columns Reconstructionof data starts from raw-event data, less easily organized in tables and columns and likely to require data in blobs Analysisuses many select-type arguments and functions than can be implemented in PL/SQL Reconstruction of data in DB will require use of external agent to run the experiment’s C++ reconstruction software at the database Analysis data will need to be accessible by many users doing many different analysis’ at once Reconstruction tasks are centrally organized, data is reconstructed during data-taking and re-reconstructed during re-processing campaigns How to optimize the use of Oracle services for LHC-scale data processing? LHC physics analysis and databases - M. Limper

LHC Physics Analysis and Databases