LHC Common Persistency Framework

1. LHC CommonPersistency Framework A report on the ongoing work of the RTAG Vincenzo Innocente CERN/EP Persistency RTAG

2. RTAG The LHC Computing Grid (LCG) LHCC CommonComputingRRB Reviews Reports Resource Matters Project Overview Board (POB) OtherComputingGridProjects ProjectExecutionBoard (PEB) Software andComputingCommittee(SC2) Requirements, Monitoring Other HEPGridProjects implementation teams EUDataGridProject Other Labs Persistency RTAG

3. SC2 mandate to the RTAG • Write the product specification for the Persistency Framework for Physics Applications at LHC. • Construct a component breakdown for the management of all types of LHC data • Identify the responsibilities of Experiment Frameworks, existing products (such as ROOT) and as yet to be developed products. • Develop requirements/use cases to specify (at least) the metadata /navigation component(s) • Estimate resources (manpower) needed to prototype missing components Persistency RTAG

4. Guidance from the SC2 • The RTAG may decide to address all types of data, or may decide to postpone some topics for other RTAGS, once the components have been identified. • The RTAG should develop a detailed description at least for the event data management. • Issues of schema evolution, dictionary construction and storage, object and data models should be addressed. Persistency RTAG

5. RTAG Composition • Fons Rademaker (Alice) • David Malon (Atlas, Chair) • VI (cms) • Pere Mato (Lhcb) • Dirk Dullermann (IT/DB) • Rene Brun (Root) Collaborative, friendly atmosphere Real effort to define a common product Persistency RTAG

6.  RTAG timetable • The RTAG met for the first time on 28 January, and three additional times that week. • A comparison of calendars revealed that between the RTAG’s launch week and the LCG launch week (11 March) there were exactly four days on which CERN-based RTAG members would be simultaneously available, even before considering the schedule of the North American RTAG member. • The RTAG met again on 18/19 February. • The RTAG plans to deliver an interim report on 8 March, to use LCG launch week (11-15 March) to solicit additional input and feedback, and to deliver a final report by 29 March. Persistency RTAG

7. Scope • Event Data • Event and DataSet Catalog Data (physics characterization metadata) • Detector Conditions Data (geometry, calibrations, alignments, etc.) • Job Configuration Data ?? • (Virtual data?) Persistency RTAG

8. Component Diagram Persistency RTAG

9. Design Criteria • Abstract Interfaces. Components of the Persistency Framework should implement abstract interfaces and be as technology neutral as possible. Several implementations for single component are not encouraged but they should be possible. • The interaction between components should happen exclusively through the public and agreed interfaces. This is to avoid private communication between components (e.g. static storage) that will make impossible a later replacement of an implementation with another one. • Typically the “end-user” (physicist writing a piece of code) does not interact directly with the framework abstract interfaces. A thin layer to hide the technicalities of such interfaces should be envisaged. Persistency RTAG

10. Design Criteria (cont.) • Re-use existing implementations. If implementations already exists providing the required functionality they should be used to provide the initial implementation. • We target C++ as the main programming language, however we should avoid constructions, which will make impossible the migration to existing or future new languages. • Transient and persistent object representations. The representation of a object in memory may be different that its persistent representation. Persistency RTAG

11. Main Issues • Object Identification • Logical OID, physical OID, context dependent, specialized • Object Placement • Generic (at object level) • Integrated with the client cache • Object Navigation • Specialized • track n of event m in run r • Temperature version x valid at time t • Generic Ref • Catalogs • Client Cache • Specialized (Root Tree) • Generic Persistency RTAG

12. Speculation around the product • Root Dictionary • Transient to Persistent representation mapping • Many to one (one to many??) (not just “type” evolution) • Generic Streamer or Automatic Streamer generation • Extended API • Usable outside the scope of persistency itself • Root file • Documented physical and logical format • Unix-file-tree-like internal organization of buffers • Support for placement of multiple objects in a buffer • Ref and OID • Logical OID • Physical (context dependent) OID? • Automatic opening of file on Ref dereference • Ref usable in pure transient context Persistency RTAG

13. Speculation around the product • Store Manager • Root based • Ensure consistency (global flush and seek) • Client Cache • Native Root trees and folders • Gaudi Tree • Garbage collected cache • Catalogs • Relational technology • OID to logical file • Logical file to physical file • MetaData (Tag) to OID (indeces, collections) • ACID Transaction management (even on behalf of the streamer) Persistency RTAG

14. CacheMgr PlacementSvc FileCatalog StorageMgr StreamerSvc SelectorSvc DictionarySvc StreamerSvc StreamerSvc DictionarySvc MetaData Catalog PersistencyMgr IteratorSvc IMCatalog TGrid TChain TEventList TDSet IFCatalog TBuffer, TKey, TMessage IPers IPers IPlacement TTree Root Mapping IReflection ICnv C++ TClass, etc. TStreamerInfo IPReflection TFile, TDirectory TSocket IReadWrite Persistency RTAG

15. Possible evolutions • Backend different than Root • Relational database • Multiple language bindings • Python, Java • Specialized services • Condition DB • Optimized tuples for analysis • Hierarchical Collections with metadata Persistency RTAG

16. Speculative Time scale • “Better” Root • Yesterday • Catalog • LFN to PFN • OID to LFN • Client Cache other than Root Tree • Abstract interface • Integrated Product We need these Persistency RTAG