EventStore Managing Event Versioning and Data Partitioning using Legacy Data Formats

1. EventStoreManagingEvent Versioning and Data Partitioning using Legacy Data Formats • Chris Jones • Valentin Kuznetsov • Dan Riley • Greg Sharp • CLEO Collaboration • Cornell University

2. Goals • Fast and scalable • e.g. run through events w/ data in memory: >10,000ev/s • Event data stays in original file format • e.g. CLEO’s object format, Root, etc. • Can manage data and MC • Data will be versioned • can always get back the version of data you used before • Can choose runs based on ‘run conditions’ • e.g. run energy, status of RICH subdetector, etc. • Handles overlapping skims in same job • Easy to add/supercede data to an event • e.g. can add post-reconstruction info: e.g. π0 and reconstructed D’s • No dependence on proprietary software • e.g. drop our use of Objectivity

3. EventStore ‘Sizes’ • Personal • For individual physicists (e.g. laptops) • Holds personal skims • No separate processes (e.g. databases) needed to run • Group • For large offsite collaborators or on-site groups • Holds a large subset of our data • All data on disk • Requires running a 3rd party database • Collaboration • Cornell Site • Holds all of our data with replication for improved performance • Interacts with HSM • Requires running a 3rd party database • Share everything except choice of file meta-data DB

4. Data Organization • Data is organized into ‘grades’ • raw • data directly from detector • physics • Reconstruction output approved for analysis • Skims are defined within a grade • physics: all, qcd, tau, … • skims are just indicies, independent of data clustering • different skims can reference the same event • extremely easy to add additional skims

5. Adding Data • Easy to add new objects to events in an existing grade • e.g. could add π0s to physics grade after the post-reconsruction calibration has been done • Can avoid common run time calculations • e.g. shower energy, π0 finding, … • save CPU time • guarantee consistency when reprocessing

6. File Meta Data • Meta data about all files are stored in a relational DB • System independent of choice of DB • Presently using • SQLite for ‘personal’ • mySQL for ‘group and collaboration’ • Meta data stored • Logical File ID (64-bit number) to file path • What files belong with which grade, skim and run • Versioning information

7. Indexing Data • Three types of files are used when reading data • Index • translates (run, event, MC ID) to location record index • index file has fixed record length • fast random access • Location • knows where in a set of data files can be found each data unit • gives us random access • versioning implemented using different location files • specialized for each data storage format • location file has fixed record length • Data store • any way to store ‘data’ should work • implemented for two file formats with variable record sizes • Why not use a relational database for event indexing? • Indexing is read only • Info is accessed every event • must be very fast • Index is traversed on the client • scales well to many clients reading same data

8. index shower MC ID event track run raw Ks Tau Reading Data Index Location offset offset offset 1 1 1 0 00FF00 0 0 02 08FF12 0 0 02 1 3 1 3 1 7 1 4 22FF34 0 0 02 FF0052 1 1 12 Data Raw QCD

9. Performance: Sequential Access Compare reading sequentially the same data files as a chain of files versus using the EventStore EventStore is a constant 15% slower

10. Performance: Event List Access Compare using an event list to access the same data files using a chain of files versus using the EventStore The EventStore scales better the more events that are skipped

11. Versioning • When starting a new analysis, usually want the most recent reconstruction • When adding new data to an existing analysis, want to go back over the same data • Specify version by giving a date • notation: yyyymmdd • e.g. eventstore in 20040501 • Do not have to specify date of a version change, EventStore will find the closest version just before that date • In analysis, physicists use the date they first processed the data

12. Versioning: Evolution • If data is reprocessed, a new date stamp must be used to distinguish the data • e.g. if CLEO reprocesses data31 must create new date for physics grade • physics 20040101 -has first processing of data31 • physics 20040402 -has more recent processing of data31 • When new datasets are added, CLEO officers can append it to any date stamp • e.g. newly recon data35 can be added to physics 20040101 and 20040402 • When new data types are added, they can be placed in corresponding date stamp • e.g. π0s derived from 20040402 may be stored there • If data type is replaced, need new date stamp

13. 20040101 data31 recon π0 data32 recon π0 Time 20040402 data31 recon data33 recon π0 π0 Versioning: Evolution Version run number

14. Versioning: Information • Each ‘chunk’ of data added will have its own specific versioning information • e.g. Recon-20040312-Feb13_04_P2 for data32 • reconstructed data used software release Feb13_04_P2 with last change to any item that affects recon no later than 2004/03/12 • The version date-stamp is a ‘logical’ version made up of individual specific versions which describe a non-overlapping run range • CLEO officers decide what specific versions should be used together to form a logical version • Want to make tool that given a date stamp and a run range, it will tell you how to generate your MC

15. Run Selection • Support multiple ways of specifying runs • run range • runs 202000 203000 • datasets • datasets 31-34 • energy • energy 1.89 • runs whose energy ‘cluster’ around this beam energy • energy psi(3770) • energy psi(3770)-off • detector state • detector mu unused • mu not used in this analysis so can use runs where mu bad • detector rich used • Data obtained by querying a Web Service • Run meta-data is centralized • Can also be accessed via a web browser

16. Conclusion • We have deployed the EventStore • Features • Adaptable to any legacy data formats and relational databases • Provides random access to events • Allows incremental addition of data • Enables independence of event indexing and data clustering • Reuses data files for different versions • Users reactions have been very favorable