130 likes | 274 Views
STREAMS: Downstream Capture. PSS-DES meeting 11 April @ CERN. Eva Dafonte P érez. Agenda. STREAMS overview STREAMS on the TESTBED Configuration PROBLEMS STREAMS log mining configuration survey Downstream capture Split & Merge solution. STREAMS Overview.
E N D
STREAMS:Downstream Capture PSS-DES meeting 11 April @ CERN Eva Dafonte Pérez
Agenda • STREAMS overview • STREAMS on the TESTBED • Configuration PROBLEMS • STREAMS log mining configuration survey • Downstream capture • Split & Merge solution Streams: Downstream Capture
STREAMS Overview • Flexible feature for sharing information • Basic elements: • Capture • Staging • Consumption • Replicate data from one database to one or more databases • Databases can be non identical copies Streams: Downstream Capture
capture apply propagation CAPTURE PROCESS APPLY PROCESS STREAMS Architecture LCRs SOURCE QUEUE capture changes propagate events LCRs log changes apply changes DESTINATION QUEUE REDO LOG TARGET DATABASE (replica) SOURCE DATABASE user changes Streams: Downstream Capture
capture apply apply capture propagation propagation CERN CERN LCR LCR LCR LCR LCR LCR LCR CNAF Sinica IN2P3 BNL RAL STREAMS on the TESTBED insert into employees values ( 03, “Joan”, ….); Streams: Downstream Capture
apply capture capture apply propagation propagation spill over performance CERN CERN LCR LCR LCR LCR LCR LCR LCR LCR LCR LCR CNAF Sinica IN2P3 RAL BNL Configuration PROBLEMS insert into employees values ( 03, “Joan”, ….); Streams: Downstream Capture
STREAMS log mining survey • Objectives • Source database (CERN) isolation against network or database problems at the replicas • Downstream Capture • Replica sites isolation against each other • split & merge solution Streams: Downstream Capture
CERN RAC RAL CERN CNAF downstream capture propagation jobs COPY redo log files SOURCE DATABASE DOWNSTREAM DATABASE DESTINATION SITES DOWNSTREAM Capture • Capture process runs on a different database • Redo log files are copied from source to downstream • Use of fewer resources and data loss protection • Definite latency in the replication process https://twiki.cern.ch/twiki/bin/view/PSSGroup/DownstreamDatabaseConfiguration Streams: Downstream Capture
DOWNSTREAM Capture • Real-Time Downstream capture • Redo transport services use the LGWR • records data in the online redo log at the source db • The redo data is stored in the standby redo log at the downstream db • Capture process captures changes from • standby redo log -> whenever possible • archived redo log files -> whenever necessary • Reduces the amount of time required to capture changes made at the source database Streams: Downstream Capture
Split & Merge solution • Split the capture process • (original) Real-Time capture for sites “in good shape” • (new) normal capture for site/s unstable/s • new capture queue and propagation job • original propagation job is dropped • spilled LCRs are dropped from the original capture queue • Merge the capture processes • Real-Time capture is used for all the sites • Resynchronization Streams: Downstream Capture
propagation jobs propagation jobs propagation jobs CERN RAL RAL RAL RAL Real-Time Downstream Capture CERN CERN … LCR LCR RAL CNAF … … CNAF CNAF start scn minimum start scn normal Capture x x propagation job propagation job RAL RAL Split & Merge solution x start scn: valid scn from which the capture process should start capturing changes with the help of Volker Kuhr (Oracle consulting) Streams: Downstream Capture
Questions & Answers Streams: Downstream Capture