More Interfaces Workplan

1. More Interfaces Workplan Dirk Düllmann, IT-DB dirk.duellmann@cern.ch LCG Persistency Workshop, June 5th 2002

2. Conventions for next slides • All interface • assume usual set of basic fixed length typespool::u32, pool::u64, …. • but drop namespace prefix for readability • On following slides, I will use • open(string &name, …) instead of • open(const std::string &name, …) • proper use of const, &, virtual is left to the reader • Prefix interface by suggested namespace

3. IFileCatalog • Assumptions: • “logical file name” will be replaced by a unique file identifier for the scope of the persistency framework • This logical file IDs needs to be agreed on at least within LCG if not within a larger scope! • I’ll make a proposal after consulting AF people • lfid resolveRef(Ref aRef) • extract lfid from ref (technology dependent) • Proposal: I would move this functionality into the Ref/Token responsibility:eg each Storage Manager token must provide a method to obtain the unique logical file identifier it’s referring to. • string pfn findReplica(lfid) • perform lfid to pfn lookup • assuming EDG RLS will handle lfid rather than lfn • lfid createFile(string pfn) • allocate unique file ID for a new physical file • eg using GUID or from pre-allocated local range or.. • removeFile(lfid)

4. poolImpl::StorageMgr • provides pool::ITransactionHandler • ISmToken write(ISmToken pl, byte *buf, u32 len); • writes an opaque byte sequence into store • byte *read(ISmToken obj, u32 &len) • reads opaque byte sequence from store • ISmToken open(string &psxName, pool::openMode m); • opens or creates a file according to given mode • return token is usable as placement hint for write • close(ISmToken t); • closes a file which has been opened with above method

5. poolImpl::ISmToken class ISmToken { // provides access to file id, eg for catalog lookup/catalog level re-mapping pool::fileUID getFileUID(); // provides access to object id (unique in a file), may be used for object level re-mapping pool::objID getObjectID(); // extenalise/internalise void internalise(const string &exForm); string externalise(); // equality bool operator==(ISmToken rhs); // order / hash int operator>(ISmToken rhs); } • Not sure how abstract we can be…

6. Collections • See David Malon’s proposal

7. Dictionary Interface • see Pere’s, Stefan’s and Victor’s slides • Propose to accept Gaudi interface as starting point • start a quick comparison with • cint (C++) and Java reflection interfaces • possibly extending it • Need agreement about • who provides the master dictionary • I tend to agree with Pere’s arguments • model and prototype for to feeding slaves • by September

8. pool::Ref<T> • Proposal: ODMG like interface • Ref<T> and RefAny from Espresso • CacheMgr interface see next slide • ptr(); // return pointer to open object in cache • close(); // invalidate real memory pointer

9. pool::ICacheMgr and pool::Ref<T> • ICacheMgr • addRef(Ref r) // bi-directional link between ref/cm • removeRef(Ref r) • openRef(Ref r) // state change to data in memory // ref real memory pointer set • closeRef(Ref r) // state change to data on disk // ref contains only PmToken • On cache manager cleanup it calls • closeRef for_each subscribed ref

10. Proposed Work Packages • Refs and Object Lookup • Definition of IStorageManager and ISMToken interface • Implementation of a RootI/O storage manager • File Catalog and Grid Integration • MySQL, XML and EDG based implementations of IFileCatalog • FileUID generation, local catalog management tools

11. Proposed Work Packages • Collections and Iterators • Explicit and implicit collection implementations • for RootI/O and RDBMS • Conversion and Dictionary • Transient and persistent IDictionary implementation • Cross population between cint/RootI/O dictionary and • Dictionary import/export

12. Proposed Work Packages • Meta Data and Query • IAttributeList implementation for RootI/O and RDBMS • Query result iterator • Integration/Common services/Packaging? • RDBMS interface • PersistencyManager • Project specific development infrastructure

13. Main question: How to stage required component decoupling? • My Proposal : • focus first on working Navigation (You all asked for it!) • First Stage: IO and streaming still coupled • Use RootI/O as storage manager which delivers directly C++ objects instead of byte streams • Still using normal Root streaming • Decoupled Dictionaries and StreamerSrv components not required yet • Can already exercise Refs and navigation with real objects • Second Stage: IO and streaming components decoupled • (after September release) • RootI/O StorageManager now delivers opaque bytes • IDictionary and IStreamerSrv moved in place between PersistencyManager and StorageManager

14. Workplan Proposal • Sequence of functional releases which provide an increasing set of interfaces to the user • Start integration with most critical functionality • Proposed priority order • Navigation • Collections • Meta Data • Dictionary Interfaces • “September” release • (decoupled) Streaming • … • Does not mean that component WP could/should not start working on implementation, but not yet integration

15. Workplan • Basic Navigation – July • IStorageManager, ISmToken • provided for RootI/O • Ref<T> • Basic navigation works (physical OID, not into embedded collections) • IFileCatalog • basic lookup by fileUID – no property based retrieval yet • Two back ends (LCG MySQL + LCG XML) • ITransactionHandler - in place for all components • working at least for MySQL FileCatalog • prototype RootI/O checkpointing -> decision about availability in September release • Assume persistency for objects not deriving from TObject as presented by Rene is sufficient • Agreed but not yet integrated • Interface for persistent and transient dictionaries

16. Work Plan II • Collections - August • Collection<T> • basic - no property based retrieval • Explicit & implicit collection implementation for eg RDBMS • Iteration interface & population interface • Ref • Extension: navigation into embedded collections • New IFileCatalog implementation integrated • EDG/Globus • Proof of concept: • Checkpointing of root files • all other components as before

17. Work Plan III • Meta Data & Query – September • Meta Interface eg IAttributeList in place • For file and collection catalogs • Same population and query interface for both • Registry for descriptions and support for attribute based retrieval (query) • IFileProperties + ICollectionProperties • Root I/O Checkpointing works • all other components as before

18. Work Plan IV • Encapsulated Dictionary – December • IDictionary + Meta Class system - Support for dictionary reading via the agreed interface • Proof of concept implementation for population of the dictionary from C++ (cint) and at least one external source through new interface • all other components as before

19. Towards the mid’03 release • Production LCGRef -> single collection element • Production Schema import / export • Full population of LCG dictionary from external (non-C++) sources as ATLAS ADL/ LHCb XML • Sufficient to use ROOT I/O default converters • Proof of concept – Meta Data import/export • How to extract all required meta data for a given event collection/run • Production checkpoint recovery implementation

20. Resources / Schedule • Very aggressive schedule for “September release” • Still large uncertainties about required and available effort • how many contributors can be at CERN at least for some time • Need to get started to see • Expectation • Release will aim to be usable for first production tests… • … but it’s for sure it won’t pass those tests yet! • It will tell us what and how much needs still to be fixed. • No way, that a 3 month old implementation can be production quality by then!