Marcin Nowak RD45 Project IT / Object Databases

1. Marcin NowakRD45 ProjectIT / Object Databases Objectivity/DB - HPSS Interface Marcin Nowak IT/DB/RD45

2. Interface Features • Direct (POSIX) interface to HPSS • abandoned due to low performance • Staging interface • currently used version • requires staging pool management • AMS-like performance for databases in the staging pool • HPSS-like performance for databases on tape • RPC timeouts need to be addressed • Extended (extensible) AMS 5.2 • AMS protocol extended with GAP, OIP, RRP and DRP • multi-threaded • only one AMS version - no special linkable AMS • uses shared libraries (liboofs.so, libooss.so, librfio.so) Marcin Nowak IT/DB/RD45

3. Network Layer Network Layer Filesystem Access Layer INTERFACE DBfiles AMS ObjectivityClientApplication HPSSServer HPSS client Direct HPSS Interface • Standard Objectivity/DB configuration in distributed environment Objy Page ServerAMS ObjectivityClientApplication Network transferin DB pages DB pages • Objectivity/DB with HPSS FS level interface DB pages DB pages Marcin Nowak IT/DB/RD45

4. Staging Interface Overview Migration daemon Migration (write) request DB files transfer Stage Pool UNIX FS HPSSServer DB pages transfer AMS with HPSS interface Staging (read) request Marcin Nowak IT/DB/RD45

5. Software Components • OOAMS 5.2 executable • liboofs.so • liboss.so • librfio.so (multithreaded, HPSS aware version) • other shared libraries e.g. security • ooss_Stage.pl script • moving files from HPSS to the local disk (stage pool) • Migration/Purge daemon • ooss_MigrPurg.pl • migrating idle database files from the local disk to HPSS • maintaining stage pool free disk space • RFIO commands • rfcp, rfmv Marcin Nowak IT/DB/RD45

6. AMS Structure AMS 4.0.10 AMS 5.0 build 56 AMS 5.0 build 58 AMS 5.2 OOFS v2 OOFS v3 OOFS v4 OOFS v5 O O S S HPSS RFIO interface Other RFIO UNIX FS Marcin Nowak IT/DB/RD45

7. AMS/HPSS Interface at CERN • CERN version of AMS/HPSS interface is using RFIO • HPSS servers are running a special RFIO daemon used by SHIFT and Castor • RFIO eliminates the need to link with HPSS libraries • no need for DCE on the host running the interface • porting to different platforms is easier • no performance degradation compared to PFTP • full use of the HPSS parallel, point-to-point transfer protocol • 2GB file size limit • current RFIO library does not support HPSS SetCOS call Marcin Nowak IT/DB/RD45

8. Staging Interface with RFIO AMS RFIO calls File & catalog management RFIO daemon Stage-in requests Network HPSSServer UNIX FS I/ODB pages Disk Server (Solaris) Migrationdaemon HPSS Mover Tape Robot Disk Pool RFCP (RFIO copy) Marcin Nowak IT/DB/RD45

9. Space Management in the Staging Pool • Space in the disk pool is controlled by the migration and purge daemon (Perl) • Parameters of migration and purge are set via configuration file • The process is inactive between space checks, but can be waked up by AMS if free space is low • File migration is done in parallel • File locking is used to synchronize with AMS • additional lock file for every database Marcin Nowak IT/DB/RD45

10. Control Scripts • startMaster - start master script • starts AMS and migration/purge daemon • stopAll - stops master script • stops migration/purge daemon and tires to stop AMS • rereadConf - reread configuration file • suspendMigr - stop migrating files • suspendPurg - stop deleting files • restartMigr - start migrating files • restartPurg - start deleting files Marcin Nowak IT/DB/RD45

11. Configuration • Single configuration file for AMS and migration daemon specified by OOAMS_CONFIG environmental variable • Dynamic configuration refresh on the SIGCONT signal • Configurable parameters: • local and remote root paths • list of directories to scan for files to migate • file purge thresholds (in percent of disk space) • migration and purge frequency and delays • selection of file transfer method (PFTP/RFCP) • debug levels, log file location and other • Timeout problems • set OO_RPC_TIMEOUT environment variable Marcin Nowak IT/DB/RD45

12. Database File Names • Elements from the interface configuration file ($OOAMS_CONFIG) • hpss_server • local_root • remote_root • Elements from FDB catalog: • host_name • db_pathname • Database name in: FDB catalog: host_name::db_pathname ‘host_name’: db_pathname HPSS: hpss_server:remote_root/ db_pathname local_rootis removed from db_pathname Marcin Nowak IT/DB/RD45

13. Example of Path Handling • Configuration values Hpss server: hpsssrv1 local_root: /datapool remote root: /hpss/cern.ch/cms • Command oonewdb -host dbsrv1 -filepath /test/mydb.DB -db mydb • Database names FDB catalog: dbsrv1::/test/mydb.DB on dbsrv1: /datapool/test/mydb.DB in HPSS: hpsssrv1:/hpss/cern.ch/cms/test/mydb.DB Marcin Nowak IT/DB/RD45

14. Protocol Enhancements in 5.2 • Generic Authentication Protocol (GAP) • allows security information to be transferred from client application to the AMS • supports private and public key mechanisms • information is used by most AMS operations • Opaque Information Protocol (OIP) • allows arbitrary information to be transferred from client application to the AMS • useful for passing hints to HPSS (like Class Of Service) • used when a database is opened Marcin Nowak IT/DB/RD45

15. Protocol Enhancements in 5.2 • Defer Request Protocol (DRP) • allows an AMS to control the timeout of a client’s request • designed to deal with filesystems with variable latencies • handled internally by Objectivity • should make the use of AMS/HPSS interface completely transparent to the client application • Request Redirection Protocol (RRP) • allows a group of AMS’ s to perform load balancing using a redirection scheme • assumes that such a group has a single “master” AMS that is always contacted first and redirects the request Marcin Nowak IT/DB/RD45

16. Current Activities • Solve rfio_stat() problem • Test migration and purge • Test interface with CMS software • Implement request deferring • Test other AMS protocol enhancements • opaque information • security Marcin Nowak IT/DB/RD45