IWLSC – CHEP 2006

1. IWLSC – CHEP 2006 The Evolution of Databases in HEP A Time-Traveller's Tale [ With an intermezzo covering LHC++ ] Jamie Shiers, CERN ~ ~ ~ …, DD-CO, DD-US, CN-AS, CN-ASD, IT-ASD, IT-DB, IT-GD, ¿¿-??, …

2. Abstract • The past decade has been an era of sometimes tumultuous change in the area of Computing for High Energy Physics. • This talk addresses the evolution of databases in HEP, starting from the LEP era and the visions presented during the CHEP 92 panel "Databases for High Energy Physics" (D. Baden, B. Linder, R. Mount, J. Shiers). • It then reviews the rise and fall of Object Databases as a "one size fits all solution" in the mid to late 90's and finally summarises the more pragmatic approaches that are being taken in the final stages of preparation for LHC data taking. • The various successes and failures (depending on one's viewpoint) regarding database deployment during this period are discussed, culminating in the current status of database deployment for the Worldwide LCG.

3. Once open a time… Databases for HEP Panel at CHEP 1992: • Should we buy or build database systems for our calibration and book-keeping needs? • Will database technology advance sufficiently in the next 8 to 10 years to be able to provide byte-level access to petabytes of SSC/LHC data? Drew Baden, University of Maryland (PASS Project); B. Linder, Oracle Corporation; Richard Mount, Caltech; Jamie Shiers, CERN.

4. Buy or build? – Calibration & Bookkeeping • Is it technically possible to use a commercial system? • Would it be manageable administratively and financially? • Questions already investigated during LEP planning phase • Computer Physics Communications 45 (1987) 299-310 • “Possibility” in 1984 (technically but not Q2); • “Probability” in 1992 (technically and conceivably also Q2).

5. Calibration & Bookkeeping in 1992 • Many experiments had independently developed solutions to these problems, with largely overlapping functionality • Two (of the) efforts at producing common solutions: • FATMEN – file catalog and more (used by DELPHI, L3, OPAL etc. ); • HEPDB – calibration database based on DBL3 and OPCAL • Both of these were based on ZEBRA-RZ • ZEBRA-FZ for “updates”, with zftp/zserv for distribution • FATMEN also had an Oracle backend • Later dropped…

6. Arguments - 1992 • A significant amount of HEP-specific code would need to be added – roughly comparable in size (within a factor of 2) of existing home-grown solutions • Commercial systems require well trained support staff at every site. (Home-grown too, but experience shows that this is typically much less than for commercial solutions.) • Licensing and support for large, diverse HEP collaborations clearly a concern We shall revisit these questions later in the show…

7. ¿Use of existing packages for SSC/LHC? • Could / should Zebra be used in 2001? • Hope to exploit language features of Fortran90 (?) and/or C++ • File catalog for multi-PB of data? • Move towards nameserverapproach (LFC = castor ns) • Existing data access and management tools? • Away from sequential access… • Where do we store the data? • Actually in a database; • Assumed for metadata; less likely for data itself • In some ‘super filesystem’ – e.g. based on IEEE MSS reference model

8. Zebra RZ and Scalability Issues • Some fields within RZ were limited to 16 bits • Fine for storage of HBOOK histograms etc but a limitation for larger catalogs • Move from 16-bit fields to 32-bit (Sunanda Banerjee) • A disconcertingly recurrent theme over the next decade… • (Also introduction of ‘eXchange’ format – makes zftp redundant in favour of binary ftp) (Burkhard Holl)

9. CHEP 92 – the Birth of OO in HEP? • Wide-ranging discussions on the future of s/w development in HEP • A number of proposals presented – including ZOO (I preferred “OOZE”) – leading to: • RD41 – MOOSE • The applicability of OO to offline particle physics code • RD44 – GEANT4 • Produce a global object-oriented analysis and design of an improved GEANT simulation toolkit for HEP • RD45 – A Persistent Object Manager for HEP • (and later also LHC++ (subsequently ANAPHE)) • ROOT

10. LHC++ - a “C++ CERNLIB” Jamie Shiers IT/ASD group, CERN http://wwwinfo.cern.ch/asd/lhc++/index.html CERN - Computing Challenges

11. Background - CERNLIB • Widely used, Fortran-based library • Developed at CERN over ~30 years • “Natural” evolution to Fortran 90/95/2K • Significant effort spent on F90 preparation • Circa 1993: strong growth in OO (C++) • Decision to drop plans for “F90 CERNLIB” • Effort redirected towards OO solutions CERN - Computing Challenges

12. Libraries for HEP Computing Why C++? Why LHC++? • “The equivalent of the current CERN program library” (ATLAS TP) • “… only language that meets these requirements is C++” (CMS TP) • Decision of NA45 to move to C++ • Needs of R&D projects (RD41, 44, 45) • Working group setup mid-95 • Provide CERNLIB-like functionality in C++ CERN - Computing Challenges

13. Goals of Working Group • Re-use existing work where possible • e.g. CLHEP • Use standard solutions where appropriate • de-facto or de-jure: STL, OpenGL, NAG C, ODMG, ... • Investigate commercial components • including license / distribution issues / pros & cons • Formal User Requirements gathering step • following ESA PSS-05 guidelines • Be prepared for change! • LHC lifetime is extremely long CERN - Computing Challenges

14. KERNLIB MATHLIB PACKLIB MINUIT HBOOK GRAFLIB (HIGZ etc.) MCLIBs Pythia, Herwig, ... GEANT3 PGMs: PAW, ... C++ , STL, CLHEP NAG C library Applications GEMINI Histograms (HTL) OpenGL, OpenInventor, Event Generators Pythia 7, ... GEANT4 Components & Modules CERNLIB & LHC++ CERN - Computing Challenges

15. Status (Oct. 98) • [ LHC++ ]has achieved the goal of demonstrating that ... standard tools ... could solve typical HEP problems.~2-3K lines of code + commercial components provide a concrete CERNLIB replacement. • Several experiments (CMS, NA45, ...) heavily using LHC++ components • ~100 sites registered for access • ~120predicted in 1998 COCOTIME review • On track for 150-200 by end-1999 CERN - Computing Challenges

16. Modularity • Emphasis on modularity of approach • Can (have) replace(d) individual components Fahrenheit Vendor STL CERN - Computing Challenges

17. Software Migration • Strategy developed assuming migration • Has already taken place! • RogueWave to STL; OpenGL implementations • HEP-specific code, e.g. histOOgrams to HTL • Future migrations: OpenInventor to Fahrenheit? C++ to Java? Java to ??? • Lifetime of future experiments >> individual s/w packages! • Wylbur to now ~now to end-LHC CERN - Computing Challenges

18. Referee Report on LHC++ LCB 10 March 1999 J. Knobloch http://nicewww.cern.ch/atlas/wwwslides/lhcpp9903/index.htm LHC++ Report - LCB 10 March 1999

19. Alternatives to HEP Explorer should be investigated - growing interest in Java-based tools. There is interest in histogram storage without an OO DB Availability issues (frequency of releases, platforms and some licensing issues) remain to be finalised. Improvement in end-user training - slow outside CERN. CLHEP issues: releases coupled with rest of LHC++ or not? Need for a policy on further development. All experiments participating in LHC++ should define their milestones. Requirements and solutions for scripting should be evaluated and implemented. Information should be provided on LHC++ usage. Release strategy and priority setting should be looked at again, perhaps best through setting up an executive board. December 98 - Outstanding issues LHC++ Report - LCB 10 March 1999

20. Current usage of LHC++ Position on the various components Priorities Concerns Activities Received answers from all 4 experiments ALICE repeats general rejection of LHC++ expressed at the last LCB: We have briefly evaluated LHC++ and we were not very satisfied. At the moment we are using ROOT and we are content with it. Currently we have no plans regarding LHC++ We will consider very seriously any solution that is adopted by the other three experiments as a common solution We believe that ROOT can be such a common solution and we encourage the other experiments to consider it as such Asked the 4 Experiments LHC++ Report - LCB 10 March 1999

21. Major progress has been made Meetings, new personnel Plan to establish an alternative to Explorer by something more adapted to the user requirements Improve the HEP specific documentation of commercial components Document the overall architecture and dependencies Clarify the boundaries of LHC++ Items from previous LCBs: Scripting Usage statistics LHC++ is now moving from the development phase to the production phase. It is therefore appropriate to revisit the management of the project by establishing a “LHC++ Executive Board” composed of the LHC++ project leader and representatives of the LHC experiments Discussing development and release plans Setting priorities Proposal of conclusions LHC++ Report - LCB 10 March 1999

22. Now back to our main thread… • We left off back at CHEP ’92… • (The birth of OO in HEP?) CERN - Computing Challenges

23. RD45 – Initial Milestones • [The project ] should be approved for an initial period of one year. The following milestones should be reached by the end of the 1st year. • A requirements specification for the management of persistent objects typical of HEP data together with criteria for evaluating potential implementations. [ Later dropped – experiments far from ready ] • An evaluation of the suitability of ODMG's Object Definition Language for specifying an object model describing HEP event data. • Starting from such a model, the development of a prototype using commercial ODBMSes that conform to the ODMG standard. The functionality and performance of the ODBMSes should be evaluated. • It should be noted that the milestones concentrate on event data. Studies or prototypes based on other HEP data should not be excluded, especially if they are valuable to gain experience in the initial months.

24. RD45 – Initial Steps • Contacts with the main ODBMS vendors of that time • O2, ObjectStore, Objectivity, Versant, Poet, … • Many presentations scheduled at CERN • Training on O2, Objectivity/DB, a few licenses acquired… • Prototyping focussed on Objectivity/DB with Versant (later) as primary fall-back • Scalability of architecture

25. Objectivity/DB Scalability 16 bits again! 216 databases (files) of 100GB = 6.5PB CERN has requested extended OID

26. LCRB review, March 1996 • The RD45 project has made excellent progress in identifying and applying solutions for object persistence for HEP based on standards and commercial products • RD45 should be approved for a further year • The LCRB agrees with the program of future work outlined in the RD45 status report and regards the following activities (below) and milestones (next) as particularly important: • Provide the object persistence services needed for the first release of GEANT4 in early 1997 • Collaborate with ATLAS and CMS in the development of those aspects of the Computing Technical Proposals which may be affected by the nature of object persistence services

27. RD45 Milestones - 96 • Identify and analyse the impact of using an ODBMS for event data on the Object Model, the physical organisation of the data, coding guidelines and the use of third party class libraries • Investigate and report on ways that Objectivity/DB features for replication, schema evolution and object versions can be used to solve data management problems typical of the HEP environment • Make an evaluation of the effectiveness of an ODBMS and MSS as the query and access method for physics analysis. The evaluation should include performance comparisons with PAW and Ntuples

28. RD45 Milestones - 97 • Demonstrate, by the end of 1997, the proof of principle that an ODBMS can satisfy the key requirements of typical production scenarios (e.g. event simulation and reconstruction), for data volumes up to 1TB. The key requirements will be defined, in conjunction with the LHC experiments, as part of this work, • Demonstrate the feasibility of using an ODBMS + MSS for Central Data Recording, at data rates sufficient to support ATLAS and CMS test-beam activities during 1997 and NA45 during their 1998 run, • Investigate and report on the impact of using an ODBMS for event data on end-users, including issues related to private and semi-private schema and collections, in typical scenarios including simulation, (re-)reconstruction and analysis.

29. RD45 Milestones - 98 • Provide, together with the IT/PDP group, production data management services based on Objectivity/DB and HPSS with sufficient capacity to solve the requirements of ATLAS and CMS test beam and simulation needs, COMPASS and NA45 tests for their '99 data taking runs. • Develop and provide appropriate database administration tools, (meta-)data browsers and data import/export facilities, as required for (1). • Develop and provide production versions of the HepOODBMS class libraries, including reference and end-user guides. • Continue R&D, based on input and use cases from the LHC collaborations to produce results in time for the next versions of the collaborations' Computing Technical Proposals (end 1999).

30. Toward the 2001 Milestone“Choice of ODBMS vendor” “If the ODBMS industry flourishes it is very likely that by 2005 CMS will be able to obtain products, embodying thousands of man-years of work, that are well matched to its worldwide data management and access needs. The cost of such products to CMS will be equivalent to at most a few man-years. We believe that the ODBMS industry and the corresponding market are likely to flourish. However, if this is not the case, a decision will have to be made in approximately the year 2000 to devote some tens of man-years of effort to the development of a less satisfactory data management system for the LHC experiments.” (CMS Computing Technical Proposal, section 3.2, page 22) And the rest, as they say, is History…

31. Risk Analysis: Issues • Choice of Technology • ODBMS, ORDBMS, RDBMS, “light-weight” Persistency, files + meta-data, ... • Choice of Vendor (historically) • #1 Objectivity, #2 Versant • Size of market • Did not take off as anticipated; unlikely to grow significantly in short-medium term CERN - Computing Challenges

32. Risk Analysis and Actions • ODBMS market has not grown as predicted • Need to understand alternatives • Possibilities include: • “Open Source” (?) ODBMS solution • ORDBMS-based solution (also for event data) • “Hybrid solutions”, incl. Meta-data + files • RD45 investigating  & directly • Based on experience at FNAL / BNL ... • Essential to consider all requirements • And not just file I/O… CERN - Computing Challenges

33. Espresso • Espresso is a proof-of-concept prototype built to answer questions from Risk Analysis • Could we build an alternative to Objectivity/DB? • How much manpower would be required? • Can we overcome limitations of Objy’s current architecture? • Support for VLDBs, multi-FD work-arounds etc. • Test / validate import architectural choices CERN - Computing Challenges

34. Espresso – Lessons Learnt • Initial prototype suggests that building a full ODBMS is technically feasible • Discussions with other sites suggest that interest goes well beyond HEP • Manpower estimates / possible resources indicate “project” would have to start “soon” • 2002: 3-year project with full system end-2004 CERN - Computing Challenges

35. ODBMS – In Retrospect • Used – in production – by several experiments at CERN, SLAC and other labs for a total of a few PB of data for just under a decade • It was – for some extended period – the baseline of ATLAS and CMS • Enhancements to the product obtained (with some effort) • MSS interface (actually much more general  xrootd); • Linux port • VLDB support • (ODMG compliance) • Much experience with ODBMS-like solutions obtained • “Risk analysis” clearly identified need for fallback, proof-of-concept prototype (Espresso) and eventually full solution (POOL) • Migration to Oracle+DATE (COMPASS) / POOL (LHC expts) successfully reported on a CHEP 2004 (scale of several hundred TB)

36. The Story So Far… • 1992: CHEP – DB panel, CLHEP K/O, CVS … • 1994: start of OO projects • 1997: proposal of ODBMS+MSS; BaBar • 2001: CMS change of baseline Objy • Now: LCG Persistency Framework RTAG • Resulted in POOL project… CERN - Computing Challenges

37. 15. Observations – IT “Eloise” Retreat 2000 • Large volume event data storage and retrieval is a complex problem that the particle physics community has had to face for decades. • The LHC data presents a particularly acute problem in the cataloguing and sparse retrieval domains, as the number of recorded events is very large and the signal to background ratios are very small. All currently proposed solutions involve the use of a database in one way or another. • A satisfactory solution has been developed over the last years based on a modular interface complying with the ODMG standard, including C++ binding, and the Objectivity/DB object database product. • The pure object database market has not had strong growth and the user and provider communities have expressed concerns. The “Espresso” software design and partial implementation, performed by the RD-45 collaboration, has provided an estimate of 15 person-years of qualified software engineers for development of an adequate solution using the same modular interface. This activity has completed, resulting in the recent snapshot release of the Espresso proof-of-concept prototype. No further development or support of this prototype is foreseen by DB group. • Major relational database vendors have announced support for Object-Relational databases, including C++ bindings. • Potentially this could fulfil the requirements for physics data persistency using a mainstream product from an established company. • CERN already runs a large Oracle relational database service CERN - Computing Challenges

38. Recommendation • The conclusion of the Espresso project, that a HEP-developed object database solution for the storage of event data would require more resources than available, should be announced to the user community. • The possibility of a joint project between Oracle and CERN should be explored to allow participation in the Oracle 9i beta test with the goals of evaluating this product as a potential fallback solution and providing timely feedback on physics-style requirements. Non-staff human resources should be identified such that there is no impact on current production services for Oracle and Objectivity. CERN - Computing Challenges Fellow, later also openlab resources

39. Oracle for Physics Data • Work on LHC Computing started ~1992 (some would say earlier…) • Numerous projects kicked off 1994/5 to look at handling multi-PB of data; move from Fortran to OO (C++) etc. • Led to production solutions from ~1997 • Always said that ‘disruptive technology’, like Web, would have to be taken into account • In 2002, major project started to move 350TB of data out of ODBMS solution; >100MB/s for 24 hour periods • Now ~2TB of physics data stored in Oracle on Linux servers • A few % of total data volume; expected to double in 2004 • [ I guess its 10 x this by now? ] Oracle 10g launch, ZH

40. LCG and Oracle • Current thinking is that bulk data will be streamed to [ ROOT ] files • RDBMS backend also being studied for ‘analysis data’ • File catalog (109 files) and file-level metadata will be stored in Oracle in a Grid-aware catalog • [ This was the “RLS” family ] • In longer term, event level metadata may also be stored in the database, leading to much larger data volumes • A few PB, assuming total data volume of 100-200PB • [ This was probably an over estimate – TAG : RAW ratio? ] • Current storage management system – CASTOR at CERN – also uses a [Oracle] database to manage the naming / location of files • Bulk data stored in tape silos and faulted in to huge disk caches Oracle 10g launch, ZH

41. Физические изыскания - перспективы • Реинжениринг всех сервисов СУБДдля физической науки на базе Oracle 10g RAC • Цели: • Изолирование – ‘сервисы’ 10g и / илифизическое разделение • Масштабируемость - как для вычислительной мощности для обработки БД, так и для устройств хранения • Надежность – автоматический обход сбоя в случае проблем • Управляемость – упрощение процессов администрирования • Вернемся к этому вопросу позже, в разделе ‘Enterprise Grids’ …

42. Physics Activities - Futures • Re-engineering all DB services for Physics on Oracle 10g RAC • Goals are: • Isolation – 10g ‘services’ and / or physical separation • Scalability - in both database processing power and storage • Reliability – automatic failover in case of problems • Manageability – significantly easier to administer than now • Will revisit this under ‘Enterprise Grids’ later… Oracle Grid Tech. day, Moscow

43. CERN & Oracle • Share a common vision regarding the future of high performance computing • Wide spread use of commodity dual processor PCs running Linux; • Focus on Grid computing • CERN has managed to influence Oracle product • Oracle 10g features: • Support for native IEEE floats & doubles; • Support for “Ultra large” Databases (ULDB);  16 bit fields again! • Cross-platform transportable tablespaces; • Instant-client developer etc. Oracle Grid Tech. day, Moscow

44. LHC DB Applications • Clear that many “LHC construction / operations applications will use Oracle • This is true also for detector construction / monitoring / calibration applications • But the main change is closer to “physics applications” • There was no “general purpose” DB service for the physics community at the time of LEP • Some applications certainly (OPAL online tape DB, …) • But these are legion at the time of LHC… • See hidden slides for some more info… • Moreover…

45. 2D Workshop - Background • A number of discussions, initially with LHC online community, revealed significant number of new database applications, and developers, in the pipeline • Meeting in October 2004 with offline and online representatives from all 4 LHC experiments led to proposal of Database Developers’ Workshop • This is it! • Significant amount of preparation by many people • Please profit from it – and not the wireless network • Silencing your mobile phone would be much appreciated… [ A free Tom Kyte book was given to all attendees]

46. (Workshop) Goals & Motivation • We are (finally) very close to LHC startup • Many new projects coming along • There is simply no time to waste! • Non-optimal (DB) applications are very expensive to ‘fix’ a posteriori • Try to establish some basic ‘best practice’ guidelines • And also well defined dev integration  production path

47. WLCG and Database Services • Many ‘middleware’ components require a database: • dCache – PostgreSQL (CNAF porting to Oracle?) • CASTOR / DPM / FTS* / LFC / VOMS – Oracle or MySQL • Some MySQL only: RB, R-GMA#, SFT# • Most of these fall into the ‘Critical’ or ‘High’ category at Tier0 • See definitions below; T0 = C/H, T1 = H/M, T2 = M/L • Implicit requirement for ‘high-ish service level’ • (to avoid using a phrase such as H/A…) • At this level, no current need beyond site-local+ services • Which may include RAC and / or DataGuard • [ TBD together with service provider ] • Expected at AA & VO levels *gLite 1.4 end October #Oracle version foreseen +R/O copies of LHCb FC?

48. Those Questions again… • Should we buy or build database systems for our calibration and book-keeping needs? • It now seems to be accepted that we buildour calibration & book-keeping systems on top of a database system. • Both commercial and open-source databases are supported. • Will database technology advance sufficiently in the next 8 to 10 years to be able to provide byte-level access to petabytes of SSC/LHC data? • We (HEP) have run production database services up to the PB level. The issues related to licensing, and – perhaps more importantly – support, to cover the full range of institutes participating in an LHC experiment, remain. • Risk analysis suggests a more cautious – and conservative – approach, such as that currently adopted. (Who are today the concrete alternatives to the market leader?)

49. If you want to know more… • Visit http://hepdb.blogspot.com/ • And also: • http://wwwasd.web.cern.ch/wwwasd/cernlib/rd45/ • http://wwwasd.web.cern.ch/wwwasd/lhc++/indexold.html • http://hep-proj-database.web.cern.ch/hep-proj-database/

50. General Conclusions • “The past decade has been an era of sometimes tumultuous change in the area of Computing for High Energy Physics.” • This is certainly true – virtually no stone has been left unturned • Could we have got where we are now more directly? • With hindsight this might seem clear, but… “Predicting is always very difficult. Especially predicting the future.” • Niels Bohr