1 / 8

CMS Online-To-Offline Data Replication

CMS Online-To-Offline Data Replication. Saima Iqbal and Lee Lueking (Fermilab) LCG 3D Database Workshop Oct 18, 2005. Outline. Infrastructure for CMS Online-To-Offline (o2o) Data transform/transfer Elicited requirements for o2o o2o application implementation

honorato-soto
Download Presentation

CMS Online-To-Offline Data Replication

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CMS Online-To-Offline Data Replication Saima Iqbal and Lee Lueking (Fermilab) LCG 3D Database Workshop Oct 18, 2005 Saima Iqbal: CMS on to off-line replication

  2. Outline • Infrastructure for CMS Online-To-Offline (o2o) Data transform/transfer • Elicited requirements for o2o • o2o application implementation • Plan to use IT services at CERN for o2o data transfer Saima Iqbal: CMS on to off-line replication

  3. CMS Databases and Data Flows Saima Iqbal: CMS on to off-line replication

  4. Elicited Requirements for o2o • Req1- Data from Online database needed offline will be selected and transferred to the Offline database. • Req2- The transfers will be triggered periodically either by a Cron job or other driving mechanism. This will be configurable. • Req3-A failsafe mechanism must assure that only data not previously transferred will be moved, and all data which is attempted to transfer is completely moved with no failures. • Req4- This data transfer and transformation process must be transparent to the offline Clients. • Req5- The tool developed for this transform and transformation of data should be independent of online or offline software. • Req6- If the IOV (Interval-Of-Validity) information is maintained in the online database and is required to generate the offline objects, then it too will be transferred in the o2o process. • Desirable- The procedures for transferring the o2o data must be easily managed and maintained. Therefore, using a common, configurable, tool for all types of data, and for all sub-detectors is desirable. Saima Iqbal: CMS on to off-line replication

  5. o2o Application (current status/implementation) Offline Database (Oracle database at CERN IT division) POOL-ORA API POOL-ORA API use the streamed payload data (materialized views) to generate POOL-ORA objects Stage 3: steam (using Oracle Stream) data from Materialized views to the offline database o2o Application CMS-Online Database (P5 Oracle Server) Stage 2: materialized views created on the transformed data Stage 1: data selected to transfer to the offline database and transformed according to POOL-ORA payload format Saima Iqbal: CMS on to off-line replication

  6. Test Plan to use Oracle Stream for o2o Data Transfer Following is a test plan test Oracle Stream for o2o data transfer: - Functionality Test: Test data transfer by using Oracle Stream in between CMS online Oracle database server and an Oracle database server at IT - Scalability/Performance Test: Test scalability/Performance of developed application and Oracle Stream for the data of the range of GB - Data Transfer Monitoring: Test the reliability of data transfer in between P5 and servers at IT - Failure Mode Tests: Test Oracle Stream with different failure modes - Reverse Streaming: Test data stream from Offline-to-Online database Saima Iqbal: CMS on to off-line replication

  7. Summary • Initial implementation/functionality of o2o application is in place • A test plan is proposed to use Oracle Stream resources at CERN IT • Looking forward to implement Oracle Stream test plan to test o2o data transfer • Use this o2o setup for stress test in November Saima Iqbal: CMS on to off-line replication

  8. Questions Saima Iqbal: CMS on to off-line replication

More Related