The POOL Persistency Framework

1. The POOL Persistency Framework Dirk Duellmann, IT-DB & LCG-POOL Opera Computing Meeting 23th September 2003

2. What is POOL? • Project Goal: Develop a common Persistency Framework for physics applications at the LHC • Pool Of persistent Objects for LHC • Common effort between the LHC experiments and the CERN IT-DB group • for defining its scope and architecture • for the development of its components • Being integrated by CMS, ATLAS and soon LHCb • First test production: now – CMS processed 700k events Dirk Duellmann, IT-DB

3. POOL Timeline and Statistics • POOL project started April 2002 • Ramping up from 1.6 to ~10 FTE • Persistency Workshop in June 2002 • First internal release POOL V0.1 in October 2002 • In one year of active development since then • 10 public releases • POOL V1.3.1 last Friday • Some 50 internal releases • Often picked up by experiments to confirm fixes/new functionality • Very useful to insure releases meet experiment expectations beforehand • Handled some 150 bug reports • Savannah web portal proven helpful • POOL followed from the beginning a rather aggressive schedule to meet the first production needs of the experiments. Dirk Duellmann, IT-DB

4. POOL Persistency • To allow the multi-PB of experiment data and associated meta data to be stored in a distributed and Grid enabled fashion • various types of data of different volumes (event data, physics and detector simulation, detector data and bookkeeping data) • Hybrid technology approach, combining • C++ object streaming technology, such as Root I/O, for the bulk data • transactional safe Relational Database (RDBMS) services, such as MySQL, for catalogs, collections and meta data • In particular, it provides • Persistency for C++ transient objects • Transparent navigation from one object across file and technolog boundaries • Integrated with a external File Catalog to keep track of the file physical location, allowing files to be moved or replicated Dirk Duellmann, IT-DB

5. POOL Architecture • POOL is a component based system • follows the LCG Architecture Blueprint • Provides a technology neutral API • Abstract component C++ interfaces • Insulates the experiment framework user code from several concrete implementation details and technologies used today • POOL user code is not dependent on implementation libraries • No link time dependency on implementation packages (e.g. MySQL, Root, Xerces-C..) • Backend component implementations are loaded at runtime via the SEAL plug-in infrastructure • Three major domains, weakly coupled, interacting via abstract interfaces Dirk Duellmann, IT-DB

6. POOL Work Package breakdown • Storage Manager • Streams transient C++ objects into/from disk storage • Resolves a logical object reference into a physical object • Uses Root I/O. A proof of concept with a RDBMS storage manager prototype underway • File Catalog • Maintains consistent lists of accessible files (physical and logical names) together with their unique identifiers (FileID), which appear in the object representation in the persistent space • Resolves a logical file reference (FileID) into a physical file • Collections • Provides the tools to manage potentially (large) ensembles of objects stored via POOL persistence services • Explicit: server-side selection of object from queryable collections • Implicit: defined by physical containment of the objects Dirk Duellmann, IT-DB

7. POOL Component Breakdown Dirk Duellmann, IT-DB

8. POOL on the Grid Experiment Framework Collections Grid Dataset Registry Grid Resources User Application File Catalog Replica Location Service Replica Manager RootI/O Meta Data Meta Data Catalog LCG POOL Grid Middleware Dirk Duellmann, IT-DB

9. POOL off the Grid Experiment Framework Collections MySQL or RootIO Collection User Application File Catalog XML / MySQL Catalog RootI/O Meta Data MySQL LCG POOL Disconnected Laptop Dirk Duellmann, IT-DB

10. POOL will be mainly used experiment frameworks, as client library loaded by user applications POOL applications are Grid aware via the File Catalog component based on the EDG Replica Location Service (RLS) File resolution and meta data queries are forwarded to Grid middleware requests The POOL storage manager ensures the remote file access via Root I/O (such as RFIO/dCache), possibly later replaced by the Grid File Access Library(GFAL), once it will be available POOL Client <-> Server POOL client on a CPU Node User Application Experiment Framework POOL Grid (File) Services File Meta Data Replica Location Remote File I/O? remote access via ROOT I/O Dirk Duellmann, IT-DB

11. POOL File Catalog Logical Naming Object Lookup LFN2 PFN2, technology PFNn, technology LFNn File metadata (jobid, owner, …) FileID-LFNmapping supported but not used internally FileID-PFNmapping is sufficient for object lookup • Files are referred to inside POOL via a uniqueandimmutable file identifier,(FileID)generated at creation time • POOL added the system generated FileID to the standard Grid m-n mapping • Stable inter-file reference • Global Unique Identifier (GUID) implementation for FileID • allows the production of a consistent sets of files with internal references without requiring a central ID allocation service • catalog fragments created independently can later be merged without modification to corresponding data file Dirk Duellmann, IT-DB

12. Concrete implementations • XML Catalog • typically used as local file by a single user/process at a time • no need for network or centralised server • supports R/O operations via http • tested up to 50K entries • Native MySQL Catalog • handles multiple users and jobs (multi-threaded) • tested up to 1M entries • EDG-RLS Catalog • Grid aware applications • Oracle iAS or Tomcat + Oracle / MySQL backend • pre-production service based on Oracle (from IT/DB) , RLSTEST, already in use for POOL V1.0 Dirk Duellmann, IT-DB

13. Use case: isolated system • The user extracts a set of interesting files and a catalog fragment describing them from a (central) Grid based catalog into a local XML catalog • Selection is performed based on file or collection meta data • After disconnecting from the Grid the user executes some standard jobs navigating through the extracted data • New output files are registered into the local XML catalog • Once the new data is ready for publishing and the user is connected the new catalog fragment is submitted to the Grid based catalog XML lookup input files register output files Import jobs Publish No network EDG Dirk Duellmann, IT-DB

14. Use case: farm production lx1.cern.ch quality check lookup register jobs XML publish MySQL pc3.cern.ch publish publish quality check lookup register jobs publish EDG • A production job runs and creates files and their catalog entries in a local XML file • During the production the catalog can be used to cleanup files • Once the data quality checks have been passed the production manager decides to publishes the production XML catalog fragment to the site database one and eventually to the Grid based catalog Dirk Duellmann, IT-DB

15. POOL Storage Hierarchy • A application may access databases (eg ROOT files) from a set of catalogs • Each database has containers of one specific technology (eg ROOT trees) • Smart Pointers are used • to transparently load objects into a client side cache • define object associations across file or technology boundaries Dirk Duellmann, IT-DB

16. Storage Hierarchy Technology DB name database database Database Objects Cont.name Objects Objects Item ID Objects Objects Objects Container Objects Container Objects Container Data Cache StorageMgr • Object Pointer • Persistent Address DiskStorage Dirk Duellmann, IT-DB

17. Database Technologies • Identify commonalties and differences between technologiesNecessary knowledge when reading/writing • Model adapts to any technology with direct record access • Need to know record identifier in advance • RDBMS: More or less traditional • Primary key must be uniquely determined before writing • Probably two round-trips Dirk Duellmann, IT-DB

18. Dictionary: Population/Conversion .xml .h GCC-XML Code Generator ROOTCINT LCG dictionary code CINT dictionary code Gateway I/O CINT dictionary LCGdictionary Other Clients Data I/O Reflection Technology dependent DictionaryGeneration Dirk Duellmann, IT-DB

19. Client Data Access Manages object cache Ref<T> Data Service Object Cache Client accessdata through References Client Client Client Ref<T> Object Cache Data Service Different context • Event data • Detector data • other Ref<T> Data Service Object Cache Dirk Duellmann, IT-DB

20. POOL Milestones • First “Public” Release - V0.3 December ’02 • Navigation between files supported, catalog components integrated • LCG Dictionary moved to SEAL – and picked up from there • Basic dictionary integration for elementary types • First “Functionally Complete” Release - V1.0 June ’03 • LCG dictionary integration for most requested language features including STL containers • Consistent meta data support for file catalog and event collections (aka tag collections) • Integration with EDG-RLS pre-production service (rlstest.cern.ch) • First “Production Release” - V1.1 July `03 • Added bare C++ pointer support, transient data members, update of streaming layer data, simplified (user) transaction model • Due to the large number of requests from integration activities still rather a functionality release than the planned consolidation release. • EDG-RLS production service (one catalog server per experiment) • Project stayed close to release data estimates • Maximum variance 2 weeks • Usually release within a few days around the predicted target date Dirk Duellmann, IT-DB

21. POOL Experiment Integration • Experiment Integration started after the POOL V1.1 release • CMS • LCG AA s/w environment was easy to pickup scram, SEAL plugin loading • (optional) Object cache provided by POOL is used directly • Many functional request have been formulated and implemented during their first POOL integration • ATLAS • More complicated framework integration eg due to interferences between the SEAL and GAUDI plugin loading facilities • Investigating several integration options: with or w/o POOL object cache – role of LCG object white board • More development work required on the experiment side and more people involved • LHCb • Share most integration issues with ATLAS • Expect no significant problems as key POOL developer and framework integrator is the same person Dirk Duellmann, IT-DB

22. Summary • The LCG Pool project provides a hybrid store integrating object streaming (eg Root I/O) with RDBMS technology (eg MySQL) for consistent meta data handling • Strong emphasis on component decoupling and well defined communication/dependencies • Transparent cross-file and cross-technology object navigation via C++ smart pointers • Integration with Grid wide Data Catalogs (eg EDG-RLS) • but preserving networked and grid-decoupled working modes • Recently also the various ConditionsDB implementations have been moved into the scope of the LCG Persistency Project • Work on software and release integration with POOL and other LCG Application area services will start soon • POOL has been integrated into LHC experiments software frameworks and is use for the pre-production activities in CMS • Selected as persistency mechanism for CMS PCP, ATLAS and LHCb data challenges Dirk Duellmann, IT-DB

23. How to find out more about POOL? • POOL Home Page http://pool.cern.ch/ • POOL Workbook http://lcgapp.cern.ch/project/workbook/pool/current/pool.html • POOL savannah portal (bug reports, cvs) http://savannah.cern.ch/projects/pool • POOL binary distribution (provided by LCG-SPI) http://lcgapp.cern.ch/project/spi/lcgsoft Dirk Duellmann, IT-DB

24. Generic Persistent Model Persistent C++ pointer >> object ID Objects & pointers Objects, object IDs, collections & DBs Transient Dirk Duellmann, IT-DB

25. Cache Access Through References Reference to Cache Manager Ref<T> Object Reference in Cache Manager Dereference Pointer to object • References know about the Data Cache • 2 operation modes: - Clear at checkpoint - Auto-clear with reference count • References are implemented as smart pointers • Use cache manager for “load-on-demand” • Use the object key of the cache manager Dirk Duellmann, IT-DB

26. POOL Cache Access Cache Manager SmartRef<T> Key CacheMgr Pointer Persistency Manager Dirk Duellmann, IT-DB

27. Follow Object Associations Object Token <pointer> <…> <…> <…> (3) (2) (4) Link ID Link Info ... ... DB/Cont.name,... <number> (1) Local lookup table in each file Entry ID Link ID Dirk Duellmann, IT-DB

28. The Link Table • Contains all information to resurrect an object • Storage type • Database name • Container name • Object type (class name) • Cache hints • E.g. other possible transient conversions • Size: O(Associations in class model) • Local to every database • Size is limited Dirk Duellmann, IT-DB

29. File Catalog functionality • Connection and transaction control functions • Catalog insertion and update functions on logical and physical filenames • Catalog lookup functions (by filename, FileID or query) • Clean-up after an unsuccessful job • Catalog entries iterator • File Meta data operations (e.g. define or insert file meta data) • Cross catalog operations (e.g. extract a XML fragment and append it to the MySQL catalog) • Python based graphic user interface for the catalog browsing Dirk Duellmann, IT-DB

30. File Catalog Scaling Tests • update and lookup performances look fine • scalability shown up to a few hundred boxes • need to understand availability and backup in the production context Dirk Duellmann, IT-DB

31. File Catalog Browser Prototype Dirk Duellmann, IT-DB

32. Pros & cons of POOL vs Vanilla Root • More services: • File catalog (disconnected/Grid aware) • Object cache manager • Functionalities: • - Object storage operation more explicit: Write/Read/Update/Delete • Object navigation transparent – Objects linked are resolved automatically inside the store • Simple transaction handling • Root functionalities preserved • Trees and Keys format available (simple to switch) Dirk Duellmann, IT-DB

33. Pros & cons of POOL vs Vanilla Root • Integration in the software: • Dictionary Generation easier with LCG Dict • No changes are needed on class headers • no instrumentation, no inheritance from external classes • Architecture: • Technology independent interfaces • Modular component design • Component can be used separately • Layered structure Dirk Duellmann, IT-DB

34. Pros & cons of POOL vs Vanilla Root • And also: • Easy to replace with Vanilla Root • Allow for alternatives to RootI/O ! • Concerns (POOL disadvantages) • Performance • Multiple software layer on top of Root • - Reliability • POOL is very young Dirk Duellmann, IT-DB