Database Deployment and Operations

1. Database Deployment and Operations US ATLAS DDM Operations and MC Production Workshop BNL, Brookhaven, NY September 28-29, 2006 Alexandre Vaniachine (Argonne)

2. Outline • WLCG 3D Project and ATLAS Database Deployment and Operations • Synergistic activities: • Project GSIMF • Project SHIELDS • Secure database deployment in production • Database release deployment • Conclusions Alexandre Vaniachine

3. WLCG Distributed Deployment of Databases (3D) Project • To set-up database services and facilities for relational data transfers as part of the WLCG infrastructure the WLCG Distributed Deployment of Databases (3D) project is coordinating database deployment and operations activity between LHC experiments and WLCG tier sites [http://lcg3d.cern.ch] • The 3D project is deploying distributed database services and developing advanced technologies allowing LCG applications to access the required data • 3D full production service: • Oct ‘06- onwards Alexandre Vaniachine

4. Database Deployment & Operations: Activity • As ATLAS Computing moves from a development-centric organization to an operations-centric one a new “Database Deployment & Operations” activity is now defined: • The activity consists in the development and deployment (in collaboration with the WLCG 3D project) of the tools that allow the worldwide distribution and installation of databases and related datasets, as well as the actual operation of this system on ATLAS multi-grid infrastructure http://twiki.cern.ch/twiki/bin/view/Atlas/DatabaseOperations Alexandre Vaniachine

5. Database Deployment & Operations: Domains • Distributed Deployment - Grigori Rybkine (U.K.) • LCG Deployment: Mireia Dosil (Spain) and Suijian Zhou (Taiwan) • OSG Deployment: Yuri Smirnov (U.S.) • NorduGrid Deployment: Frederik Orellana (Switzerland) • Legacy Sites (batch): John Idarraga (Canada) • Distributed Operations (ATLAS subset of WLCG 3D operations) • Tier1 operations: Gancho Dimitrov, Florbela Viegas (CERN) • Tier2 operations: Stefan Stonjek (U.K.) • Distributed calibration centers: Manuela Cirilli (U.S.) • Development - Jerome Fulachier (France) • Monitoring, SHIELDS, FroNTier/Squid, Dynamic Deployment,… • Documentation & User Support - vacant • Data access, File Transfer (for users), Best Practices, ... • Further information is at http://twiki.cern.ch/twiki/bin/view/Atlas/DatabaseOperations Alexandre Vaniachine

6. ATLAS Software & Computing Workshop, Sept. 11-15 • The first Database Deployment and Operations session was held during the ATLAS Software Workshop this week http://indico.cern.ch/conferenceDisplay.py?confId=a057208#2006-09-12 • The very first session was quite full • Which is a good sign for the newly defined activity Alexandre Vaniachine

7. US ATLAS Efforts • Tier1 activities – see next two presentations: • Iris Wu and Yuri Smirnov • ATLAS Tier2 Database Center Services • Marco Mambelli / Rob Gardner • http://twiki.mwt2.org/bin/view/DataServices/WebHome • ATLAS Database Grid Authentication • Wensheng Deng • http://www.hep.anl.gov/atlas/ACG/SHIELDS • ATLAS Database User Support • Jerry Gieraltowski • http://twiki.cern.ch/twiki/bin/view/Atlas/ReleaseValidation • ATLAS FroNTier evaluation (now led by David Front - LCG support) • Richard Hansen (Argonne) • Carlos Gamboa (Brookhaven) • http://twiki.cern.ch/twiki/bin/view/Atlas/AtlasFroNTierEvaluation • Focus on COOL Performance Studies (CMS is not using COOL) Alexandre Vaniachine

8. A Side Note: ATLAS FroNTier Issues with COOL • Based on general FroNTier performance from the CMS experience prospects are good to achieve the performance required for ATLAS • ATLAS does not store large amounts of data in COOL: • The bulky payload data are stored outside of COOL - in files • It is proposed that these ATLAS COOL POOL ROOT files will be managed by the ATLAS DDM • Currently they are included in ATLAS Database Release • Which is not a scalable solution • Also to be addressed in FroNTier evaluation • Squid cache consistency with COOL • Current issues with COOL tagging interface • May result in stale cache condition • Potentially a showstopper • if there is no COOL manpower to address this issue Alexandre Vaniachine

9. Leveraging Synergistic US Activities • Database deployment support from Open Science Grid • FroNTier will become a part of OSG release • Grid-enabled MySQL client is already a part of OSG release • Building technical collaboration with two small–scale projects • Tech-X Corp. project GSIMF: • Grid Software Installation Management Framework http://collaborate.txcorp.com/collaborate/distributed-technologies/grid-software-installation-management-framework-gsimf • Argonne is providing specs to Tech-X so that their software product could be used by individual OSG users for ATLAS Database Release deployment • Argonne project SHIELDS: • securing databases on the Grid Alexandre Vaniachine

10. Project SHIELDS Alexandre Vaniachine

11. Client Globus-secure communications Client End-to-end transport-level Globus security Middleware Vulnerable back-end communications Database Database What SHIELDS is doing:End-to-End Globus Security Solution for Database Access • A drawback of a middleware approach is vulnerability of the actual database access at the back-end • To eliminate this vulnerability SHIELDS delivers end-to-end Globus security Alexandre Vaniachine

12. Building Upon the Foundation of DASH • A collaborative DOE HEP SBIR project was completed by April 2006 • A partnership of Argonne HEP Division and PIOCON Technologies: • A database consulting business with a track record of Database and Grid integration projects • Project DASH studied the feasibility of the grid-enabled MySQL database development • Building upon the foundation of previous studies project SHIELDS speeded-up the development cycle and provided the first release • http://www.hep.anl.gov/atlas/ACG/SHIELDS/mysql-shields-1.0-alpha-pc-linux-gnu-i686.tar.gz Alexandre Vaniachine

13. Elaborate Globus Security Infrastructure Embedded SHIELDS Release 1.0-alpha ldd bin/mysqld linux-gate.so.1 => (0x00626000) librt.so.1 => /lib/librt.so.1 (0x001ff000) libz.so.1 => /usr/local/osg/MonaLisa/Service/VDTFarm/pgsql/lib/libz.so.1 (0x00e47000) libpthread.so.0 => /lib/libpthread.so.0 (0x00fd3000) libglobus_gsi_proxy_core_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_proxy_core_gcc32dbgpthr.so.0 (0x00ecb000) libglobus_gsi_credential_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_credential_gcc32dbgpthr.so.0 (0x00111000) libglobus_gsi_callback_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_callback_gcc32dbgpthr.so.0 (0x00c37000) libglobus_oldgaa_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_oldgaa_gcc32dbgpthr.so.0 (0x00120000) libglobus_gsi_sysconfig_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_sysconfig_gcc32dbgpthr.so.0 (0x00d3f000) libglobus_gsi_cert_utils_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_cert_utils_gcc32dbgpthr.so.0 (0x00a9f000) libglobus_openssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_openssl_gcc32dbgpthr.so.0 (0x00823000) libglobus_openssl_error_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_openssl_error_gcc32dbgpthr.so.0 (0x009cb000) libglobus_proxy_ssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_proxy_ssl_gcc32dbgpthr.so.0 (0x0012a000) libglobus_common_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_common_gcc32dbgpthr.so.0 (0x0012f000) libssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libssl_gcc32dbgpthr.so.0 (0x0016a000) libcrypto_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libcrypto_gcc32dbgpthr.so.0 (0x00212000) libltdl_gcc32dbgpthr.so.3 => /usr/local/osg/globus/lib/libltdl_gcc32dbgpthr.so.3 (0x00c47000) libdl.so.2 => /lib/libdl.so.2 (0x008f7000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x006a3000) libnsl.so.1 => /lib/libnsl.so.1 (0x00c82000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00336000) libm.so.6 => /lib/libm.so.6 (0x0041f000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x001a0000) libc.so.6 => /lib/libc.so.6 (0x00445000) /lib/ld-linux.so.2 (0x00627000) Alexandre Vaniachine

14. Secure Database Deployment in Production • The deployment of SHIELDS software product and its integration into the existing distributed data management system in production requires additional efforts, e.g., the • connection protocol string proposed by Wensheng Deng: mysqlx509://hbf7e4e5a:x509up_u2914@adbpro.usatlas.bnl.gov:3306/test_lrc • A dedicated meeting on that and other technical issues of grid-enabled databases took place at Brookhaven during this workshop • In particular we discussed plans for deployment in production at • US ATLAS Tier1 Computing Center at Brookhaven Lab • As well as deployment of latest versions for testing at • US ATLAS Tier2 Computing Center at the University of Chicago • Can we accomplish the deployment soon? Alexandre Vaniachine

15. Database Release Deployment Technologies • Common LHC software used by ATLAS for data is technology neutral – that is why various technologies can be used for actual data storage: • POOL ROOT files • Relational Databases • Server-based: • Oracle • MySQL • File-based: SQLite • Database-resident data replication technologies: • For server-resident data • Oracle streams • FroNTier • For file-based data • ATLAS Distributed Data Management technology (dataset subscription) • ATLAS Software Installation technology Alexandre Vaniachine

16. Validation and Decoupling VALIDATION • Software installation is not a simple data transfer (also compilation) • Thus, the Software Installation technology includes integrated release validation procedures • In contrast, the validation of the data transfer is much simpler • e.g. the checksum • Database Release deployment does not need complex validation after the data transfer DECOUPLING • The event data and the software release are decoupled • Any data can be processed by any software release • A similar decoupling for the Database Release data is required • A detailed discussion thread in the Computing Operations mailing list showed that ATLAS software installation technology will not support decoupling of the Database Release from the Software Release • Note, that on the grid the software and the event data are installed at each site in different SE areas Alexandre Vaniachine

17. Technical Choices • For now, the two candidate technologies are the • ATLAS Software Release Installation technology • Push Model • ATLAS DQ2 dataset subscription technology • Pull Model • During the Recent ATLAS Software Workshop we tried to reach conclusion on the database release deployment strategy • I recommended to implement proven DQ2 dataset subscription technology Alexandre Vaniachine

18. Database Replicas Deployment: Implementation • Two proven ATLAS technologies – SW Release installation and the DDM DQ2 dataset subscription - are available for implementation: • Integration with the Software Release installation technology • Possible if DB Release and SW Release are tightly coupled • i.e. each DB Release results in new SW Release installation • not sustainable in the long term • Attempts to decouple DB Release and SW Release installations revealed technical ‘showstoppers’ in the current production system • Integration with the DDM DQ2 dataset subscription mechanism • Implemented now thanks to Martin, Vakho, Pavel and Grigori • To collect experience before making long term deployment decisions the deployment technology will be tested in Release 12.0.3 production Alexandre Vaniachine

19. Conclusions • New ATLAS Database Deployment and Operation Activity has been defined • US ATLAS contributes efforts to the Activity • To secure production operations the grid-enabled database has to be deployed • Before making long term deployment decisions • FroNTier cache consistency must be validated for COOL • The new database deployment technology will be tested in Release 12.0.3 production Alexandre Vaniachine