1 / 45

Square Kilometer Array Visit to CERN May 27 th , 2009

Computing & Software Challenges @ LHC or how to convert 100TB/s into a Nobel prize Vincenzo Innocente (CMS Experiment & CERN/ PH-SFT). Square Kilometer Array Visit to CERN May 27 th , 2009. Outline. Physics signals, signatures and rates Detectors volumes and rates of raw data

hhenson
Download Presentation

Square Kilometer Array Visit to CERN May 27 th , 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computing & Software Challenges @ LHCor how to convert 100TB/s into a Nobel prizeVincenzo Innocente(CMS Experiment & CERN/ PH-SFT) Square Kilometer Array Visit to CERN May 27th, 2009

  2. Outline • Physics • signals, signatures and rates • Detectors • volumes and rates of raw data • Analysis Model (Data Processing) • Data representation, classification, metadata • Computing Model • Software architecture • Interactive Data Analysis Not Covered (backup slides available for discussion) • Organization • Software development process • Grid • Object streaming and persistency (in files and RDBMS) • Specific CERN software: Root, Geant4 • Experience during commissioning and possible future directions Computing Challenge @ LHC – V.Innocente

  3. The Large Hardon Collider at CERN pp, B-Physics, CP Violation (matter-antimatter symmetry) ALICE LHC : 27 km long 100m underground ATLAS General Purpose,proton-proton, heavy ions Discovery new physics: Higgs, SuperSymmetry Heavy ions, pp CMS +TOTEM First Data expected in Autumn Computing Challenge @ LHC – V.Innocente

  4. Collisions at the LHC: summary Computing Challenge @ LHC – V.Innocente

  5. Z at LEP (e+e-) pp collisions at 14 TeV at 1034 cm-2s-1 A very difficult environment … 20 proton-proton collisions overlap & HZZ with Z  2 muons : H 4 muons: the cleanest (“golden”) signature And this (not the H though…) repeats every 25 ns… Computing Challenge @ LHC – V.Innocente

  6. Impact on detector design • LHC detectors must have fast response • Otherwise will integrate over many bunch crossings  large “pile-up” • Typical response time : 20-50 ns  integrate over 1-2 bunch crossings  pile-up of 25-50 min-bias  very challenging readout electronics • LHC detectors must be highly granular • Minimize probability that pile-up particles be in the same detector element as interesting object (e.g.  from H   decays)  large number of electronic channels  high cost • LHC detectors must be radiation resistant: • high flux of particles from pp collisions  high radiation environment e.g. in forward calorimeters: • up to 1017 n/cm2 in 10 years of LHC operation • up to 107 Gy (1 Gy = unit of absorbed energy = 1 Joule/Kg) Computing Challenge @ LHC – V.Innocente

  7. n µ g e n p A Generic Multipurpose LHC Detector Three detector “layers” in a magnetic field Inner tracker: vertex and charged particles Calorimeters: energy of electrons, photons, hadrons External tracker: identify muons Computing Challenge @ LHC – V.Innocente

  8. The Atlas Detector • The ATLAS experiment is • 26m long, • stands 20m high, • weighs 7000 tons • has 200 million read-out channels • orders of magnitude increase in complexity • The ATLAS collaboration is ~ • 2000 physicists from .. • 150 universities and labs in.. • 34 countries • distributed resources • remote development Computing Challenge @ LHC – V.Innocente

  9. DATA ORGANIZATION

  10. Data and Algorithms • HEP main data are organized in Events (particle collisions) • Simulation, Reconstruction and Analysis programs process “one Event at the time” • Events are fairly independent to each other • Trivial parallel processing • Event processing programs are composed of a number of Algorithms selecting and transforming “raw” Event data into “processed” (reconstructed) Event data and statistics • Algorithms are mainly developed by “Physicists” • Algorithms may require additional “detector conditions” data (e.g. calibrations, geometry, environmental parameters, etc. ) • Statistical data (histograms, distributions, etc.) are typically the final data processing results Computing Challenge @ LHC – V.Innocente

  11. High Energy Analysis Model Reconstruction “goes back in time” from digital signals to the original particles produced in the collision MonteCarlo Simulation follows the evolution of physics processes from collision to digital signals Analysis compares (at statistical level) reconstructed events from real data with those from simulation Computing Challenge @ LHC – V.Innocente

  12. Data Hierarchy “RAW, ESD, AOD, TAG” RAW Triggered events recorded by DAQ Detector digitisation ~2 MB/event ESD/RECO Reconstructed information Pseudo-physical information: Clusters, track candidates ~100 kB/event Physical information: Transverse momentum, Association of particles, jets, (best) id of particles, AOD Analysis information ~10 kB/event TAG Classification information Relevant information for fast event selection ~1 kB/event Computing Challenge @ LHC – V.Innocente

  13. Selection/Skimming Locally (Tier 2) Centrally (Tier 0/ Tier1) Distributed (Tier 1/ Tier2) RECO/AOD AOD Datasets pre pre AOD, Cand Cand, User Data AOD pre pre Cand, User Data AOD, Cand Signal dataset Background dataset(s) Laptop ? 500 GB Guessed numbers at start-up 10 times more already possible 50 GB Computing Challenge @ LHC – V.Innocente

  14. Data Bookkeeping (in CMS) • Events are stored in Files • For MC there are O(1000) events/file • Files are grouped in Blocks • We aim to make blocks >= size of a tape • Blocks are grouped into Datasets • We need to track: • Which block a file is in • Which dataset a block is in • File/Block/Dataset metadata • To do all this CMS has developed DBS, the CMS Dataset Book keeping System • Oracle Backend • Web interface Computing Challenge @ LHC – V.Innocente

  15. Version Time VDET alignment HCAL calibration RICH pressure ECAL temperature t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 Data Item Time = T Detector Conditions Data • Reflects changes in state of the detector with time • Event Data cannot be reconstructed/analyzed without it • Versioning • Tagging • Ability to extract slices of data • World-wide access (replica, closest available copy, etc.) • Long life-time Version Tag1 definition Time Computing Challenge @ LHC – V.Innocente

  16. Remote Site Remote Site CalibrationApplication Worker Node CalibrationApplication Rec/Anal Application Algorithm Algorithm Cond DB Slice/ proxy Mgr Tools Mgmt Tools Cond DB Cond DB Replica/ proxy Online DB master copy Conditions Database • Need tools for data management, data browsing, replication, slicing, import/export • Database implementations (more than one) • Use supported features (transactions, replication, …) Computing Challenge @ LHC – V.Innocente

  17. COMPUTING ARCHITECTURE

  18. Physics Selection at LHC Computing Challenge @ LHC – V.Innocente

  19. Online selection HLT: Same Hardware, same Software as Offline Different situation: Cannot repeat, cannot afford even rare crash or small memory leak! Computing Challenge @ LHC – V.Innocente

  20. HEP Data Handling and Computation event filter (selection & reconstruction) detector processed data event summary data Locally Managed raw data batch physics analysis event reprocessing Chaotic analysis objects (extracted by physics topic) event simulation Centrally Managed interactive physics analysis Computing Challenge @ LHC – V.Innocente

  21. CERN 2.5 Gbps 622 Mbps FNAL RAL IN2P3 155 mbps 155 mbps 622 Mbps Lab a Uni b Lab c Uni n    A Multi-Tier Computing Model • Easier to move job to data • Data distributed to processing centers in advance • Analysis-jobs scheduling driven by data location • Data replication possible in case of unbalanced use of resources Tier 0 (Experiment Host lab) Tier 1 (Main Regional Centres) • Hierarchy of computing centers • Matches computing tasks to local interests, expertise and resources Tier2 Tier 3 • Exploit Grid infrastructures • Homogeneous environment • Optimal share of resources Desktop Computing Challenge @ LHC – V.Innocente

  22. CMS Computing Model • Site activities and functionality largely predictable • Activities are driven by data location • Organized mass processing and custodial storage at Tier-1s • ‘chaotic’ computing essentially restricted to data analysis at T2s • Resource evolution 10K “boxes” CPU 15 PB DISK 20 PB TAPE 8core Clowertown 2.3GHz = 10KSI2K Computing Challenge @ LHC – V.Innocente

  23. Tiered Architecture Tier-0: • Accepts data from DAQ • Prompt reconstruction • Data archive and distribution to Tier-1’s CAF: CMS Analysis Facility at CERN • Access to full raw dataset • Focused on latency-critical detector trigger calibration and analysis activities • Provide some CMS central services (e.g. store conditions and calibrations) Tier-1’s: • Real data archiving • Re-processing • Skimming and other data-intensive analysis tasks • Calibration • MC data archiving Tier-2’s: • User data Analysis • MC production • Import skimmed datasets from Tier-1 and export MC data • Calibration/alignment Computing Challenge @ LHC – V.Innocente

  24. CMSData&Work Flow T1/CAF Calibration sample T2 Calibration Jobs TAG/AOD (replica) Replica Conditions DB T2 MASTER Conditions DB DAQ (P5) T0 T1 Replica Conditions DB 25Hz 2MB/evt 50MByte/s 4 Tbyte/day CERN disk pool ~40 TByte (~10 days data) 1st pass Recon- struction Event streams Higgs DST TAG/AOD (10-100 kB/evt) 25Hz 1MB/evt raw 25Hz 0.5MB reco DST T2 TAG/AOD (replica) HLT Filter T2 Event server Higgs background Study (requests New events) Disk cache Archive storage SUSY Background DST CERN Tape archive Prompt & Precise Calibration Analysis Prompt Reconstruction Computing Challenge @ LHC – V.Innocente

  25. APPLICATION SOFTWARE

  26. Software Structure Applications are built on top of frameworks and implementing the required algorithms Applications Event DetDesc. Calib. Every experiment has a framework for basic services and various specialized frameworks: event model, detector description, visualization, persistency, interactivity, simulation, calibrarion, etc. Experiment Framework Simulation DataMngmt. Distrib. Analysis Specialized domains that are common among the experiments Core Libraries Core libraries and services that are widely used and provide basic functionality non-HEP specific software packages General purpose non-HEP libraries Computing Challenge @ LHC – V.Innocente

  27. Software Components • Foundation Libraries • Basic types • Utility libraries • System isolation libraries • Mathematical Libraries • Special functions • Minimization, Random Numbers • Data Organization • Event Data • Event Metadata (Event collections) • Detector Conditions Data • Data Management Tools • Object Persistency • Data Distribution and Replication • Simulation Toolkits • Event generators • Detector simulation • Statistical Analysis Tools • Histograms, N-tuples • Fitting • Interactivity and User Interfaces • GUI • Scripting • Interactive analysis • Data Visualization and Graphics • Event and Geometry displays • Distributed Applications • Parallel processing • Grid computing Computing Challenge @ LHC – V.Innocente

  28. Programming Languages • Object-Oriented (O-O) programming languages have become the norm for developing the software for HEP experiments • C++ is in use by (almost) all Experiments • Pioneered by Babar and Run II (D0 and CDF) • LHC experiments with an initial FORTRAN code base have basically completed the migration to C++ • Large common software projects in C++ have been in production for many years aready • ROOT, Geant4, … • FORTRAN still in use mainly by the MC generators • Large developments efforts are put for the migration to C++ (Pythia8, Herwig++, Sherpa,…) Computing Challenge @ LHC – V.Innocente

  29. Scripting Languages • Scripting has been an essential component in the HEP analysis software for the last decades • PAW macros (kumac) in the FORTRAN era • C++ interpreter (CINT) in the C++ era • Python recently introduced and gaining momentum • Most of the statistical data analysis and final presentation is done with scripts • Interactive analysis • Rapid prototyping to test new ideas • Driving complex procedures • Scripts are also used to “configure” complex C++ programs developed and used by the LHC experiments • “Simulation” and “Reconstruction” programs with hundreds or thousands of options to configure Computing Challenge @ LHC – V.Innocente

  30. Object-Orientation • The object-oriented paradigm has been adopted for HEP software development • Basically all the code for the new generation experiments is O-O • O-O has enabled us to handle reasonably well higher complexity • The migration to O-O was not easy and took longer than expected • The process was quite long and painful (between 4-8 years) • The community had to be re-educated to new languages and tools • C++ is not a simple language • Only specialists master it completely • Mixing interpreted and compiled languages (e.g. C++ and Python) is a workable compromise Computing Challenge @ LHC – V.Innocente

  31. Applications Experiment Framework Simulation DataMngmt. Distrib. Analysis Core Libraries non-HEP specific software packages Software Frameworks • Experiments develop Software Frameworks • General Architecture of the Event processing applications • To achieve coherency and to facilitate software re-use • Hide technical details to the end-user Physicists (providers of the Algorithms) • Applications are developed by customizing the Framework • By the “composition” of elemental Algorithms to form complete applications • Using third-party components wherever possible and configuringthem Example: Gaudi framework (C++) in use by LHCb and ATLAS Computing Challenge @ LHC – V.Innocente

  32. Applications Experiment Framework Simulation DataMngmt. Distrib. Analysis Core Libraries non-HEP specific software packages Non-HEP Packages widely used in HEP • Non-HEP specific functionality required by HEP programs can be implemented using existing packages • Favoring free and open-source software • About 30 packages are currently in use by the LHC experiments • Here are some examples • Boost • Portable and free C++ source libraries intended to be widely useful and usable across a broad spectrum of applications • GSL • GNU Scientific Library • Coin3D • High-level 3D graphics toolkit for developing cross-platform real-time 3D visualization • XercesC • XML parser written in a portable subset of C++ Computing Challenge @ LHC – V.Innocente

  33. Applications Experiment Framework Simulation DataMngmt. Distrib. Analysis Core Libraries non-HEP specific software packages HEP Generic Packages (1) • Core Libraries • Library of basic types (e.g. 3-vector, 4-vector, points, particle, etc.) • Extensions to C++ Standard Library • Mathematical libraries • Statistics libraries • Utility Libraries • Operating system isolation libraries • Component model • Plugin management • C++ Reflexion • Examples: ROOT, CLHEP, etc. Computing Challenge @ LHC – V.Innocente

  34. Applications Experiment Framework Simulation DataMngmt. Distrib. Analysis Core Libraries non-HEP specific software packages HEP Generic Packages (2) • MC Generators • This is the best example of common code used by all the experiments • Well defined functionality and fairly simple interfaces • Detector Simulation • Presented in form of toolkits/frameworks (Geant4, FLUKA) • The user needs to input the geometry description, primary particles, user actions, etc. • Data Persistency and Management • To store and manage the data produced by experiments • Data Visualization • GUI, 2D and 3D graphics • Distributed and Grid Analysis • To support end-users using the distributed computing resources (PROOF, Ganga,…) Computing Challenge @ LHC – V.Innocente

  35. Persistency Framework • FILES - based on ROOT I/O • Targeted for complex data structure: event data, analysis data • Management of object relationships: file catalogues • Interface to Grid file catalogs and Grid file access • Relational Databases – Oracle, MySQL, SQLite • Suitable for conditions, calibration, alignment, detector description data - possibly produced by online systems • Complex use cases and requirements, multiple ‘environments’ – difficult to be satisfied by a single “vendor” solution • Isolating applications from the database implementations with a standardized relational database interface • facilitate the life of the application developers • no change in the application to run in different environments • encode “good practices” once for all Computing Challenge @ LHC – V.Innocente

  36. ROOT I/O • ROOT provides support for object input/output from/to platform independent files • The system is designed to be particularly efficient for objects frequently manipulated by physicists: histograms, ntuples, trees and events • I/O is possible for any user class. Non-intrusive, only the class “dictionary” needs to be defined • Extensive support for “schema evolution”. Class definitions are not immutable over the life-time of the experiment • The ROOT I/O area is still moving after 10 years • Recent additions: Full STL support, data compression, tree I/O from ASCII, tree indices, etc. • All new HEP experiments rely on ROOT I/O to store its data Computing Challenge @ LHC – V.Innocente

  37. Simulation • Event Generators • Programs to generate high-energy physics events following the theory and models for a number of physics aspects • Specialized Particle Decay Packages • Simulation of particle decays using latest experimental data • Detector Simulation • Simulation of the passage of particles through matter and electromagnetic fields • Detailed geometry and material descriptions • Extensive list of physics processes based on theory, data or parameterization • Detector responses • Simulation of the detecting devices and corresponding electronics Computing Challenge @ LHC – V.Innocente

  38. Distributed Analysis • Analysis will be performed with a mix of “official” experiment software and private user code • How can we make sure that the user code can execute and provide a correct result wherever it “lands”? • Input datasets not necessarily known a-priori • Possibly very sparse data access pattern when only a very few events match the query • Large number of people submitting jobs concurrently and in an uncoordinated fashion resulting into a chaotic workload • Wide range of user expertise • Need for interactivity - requirements on system response time rather than throughput • Ability to “suspend” an interactive session and resume it later, in a different location Computing Challenge @ LHC – V.Innocente

  39. Data Analysis: The Spectrum • From Batch Physics Analysis • Run on the complete data set (from TB to PB) • Reconstruction of non-visible particles from decay products • Classification “Events” based on physical properties • Several non-exclusive “data streams” with summary information (event tags) (from GB to TB) • Costly operation • To Interactive Physics Analysis • Final event selection and refinements (few GB) • Histograms, N-tuples, Fitting models • Data Visualization • Scripting and GUI Computing Challenge @ LHC – V.Innocente

  40. Experiment Specific Analysis Frameworks • Development of “Event models” and “high level” analysis tools specific to the experiment physics goals • Example DaVinci (LHCb) Computing Challenge @ LHC – V.Innocente

  41. ROOT • The ROOT system is an Object Oriented framework for large scale data analysis written in C++ • It includes among others: • Efficient object persistency facilities • C++ interpreter • Advanced statistical analysis (multi dimensional histogramming, fitting, minimization, cluster finding algorithms) and visualization tools • The user interacts with ROOT via a graphical user interface the command line or batch scripts • The project started in 1995 and now is a very mature system used by many physicists worldwide Computing Challenge @ LHC – V.Innocente

  42. SUMMARY Computing Challenge @ LHC – V.Innocente

  43. Three Challenges • Detector: • ~2 orders of magnitude more channels than before • Triggers must choose correctly only 1 event in every 500,000 • Level 2&3 triggers are software-based • Geographical spread • Communication and collaboration at a distance • Distributed computing resources • Remote software development and physics analysis • Physics • Precise and specialized algorithms for reconstruction and calibration • Allow remote physicists to access detailed event-information • Migrate effectively reconstruction and selection algorithms to High Level Trigger Main Challenge is in Managing the Complexity Computing Challenge @ LHC – V.Innocente

  44. Analysis, Data and Computing Models • Analysis (data processing) Model with several organized steps of data reduction • Hierarchical Data Model that matches the various steps in the Analysis Model • Multi-Tier Computing Model that • matches the geographical location of physics analysis groups • defines the policies for replicating data to those sites with appropriate resources (and interest!) to run jobs • Grids are important to • providers – centres supplying resources in a multi-science environment in a secure and managed way • consumers – hungry for those resources (potential of resource discovery) Computing Challenge @ LHC – V.Innocente

  45. Software Solutions • A variety of different software domains and expertise • Data Management, Simulation, Interactive Visualization, Distributed Computing, etc. • Application Software Frameworks • Ensure coherency in the Event data processing applications • Make the life of Physicists easier by hiding most of the technicalities • Allows running the same software in different environment and with different configuration • Withstand technology changes • Set of HEP specific core software components • Root, Geant4, CLHEP,Pool • Extensive use of third-party generic software • Open-source products favored Computing Challenge @ LHC – V.Innocente

More Related