1 / 70

Oracle GoldenGate New Features and Product Strategy (CON8867)

Oracle GoldenGate New Features and Product Strategy (CON8867). Director of Product Management.

bjorn
Download Presentation

Oracle GoldenGate New Features and Product Strategy (CON8867)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Oracle GoldenGate New Features and Product Strategy(CON8867) Director of Product Management

  2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

  3. Agenda • Market Overview • Oracle GoldenGate New Functionality • Support for New Database Features • Ease of Use Enhancements • Security Enhancements • Q & A

  4. Data Integration Market Overview

  5. Market OverviewExponential Growth in Data Volumes 45 PB+ Maintained by Large Orgs 45%+ Annual Growth 15 out of 17 Industry Sectors in the U.S. Have More Data Than the U.S. Library of Congress

  6. Market OverviewExponential Growth in Data Volumes More Data leads to More Data Challenges

  7. Market OverviewData Integration and Data Quality Markets estimates the DI market will reach $2.8 billion estimates the DQ market will reach $1.75 billion by 2016 with an average growth rate of 18.2%

  8. Oracle Data Integration Solutions • Thousands of customers including Fortune 500 companies, representing Financial Services, Communications, Healthcare, Public Sector, Retail, Utilities industries • 4 of top 5 largest commercial banks • 3 of top 3 busiest ATM networks • 5 of top 5 financial data services companies • 5 of top 5 telecommunications providers • 3 of top 5 largest food & drug stores

  9. Fun Facts About Oracle GoldenGate That’s the Magic of Macy’s Shopping at the worlds largest retails and your credit card transactions are moved in real-time to data centers around the world, before being approved, using Oracle GoldenGate (Retail Decision) Good TV. Better TV. DirecTV. If you call DIrecTV customer service, your call data will move to their data warehouse via Oracle GoldenGate, so the rep can provide you with personalized recommendations and promotions. Say it with Flowers Ordering from 1800-Flowers, Oracle GoldenGate will move your order to their reporting systems to help process and track your order seamlessly No Charge ATM Fees 3 of the 3 busiest ATM Networks use Oracle GoldenGate for Continuous Availability. Get Sick in Pittsburgh If you get a health check at UPMC, the clinical applications that doctors use are supported by Oracle GoldenGate to enable high-availability

  10. Oracle Data Integration (DIS) Breadth Enables Leverage and Agility Data Quality Data Integration Match/Merge Profiling Standardization Heterogeneous Virtualization Big Data ETL/E-LT Replication

  11. Real-time replication; optimized for Database 12c and Oracle Exadata End-to-end integrated with simplified deployment Unified tooling for both structured data sources and Hadoop / NoSQL Flexible deployment on-premise or in the Cloud for heterogeneous systems Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions Oracle Data Integration 12c Delivering Real-Time Integration for Big Data and Cloud Cloud Oracle Data Integrator Apps Oracle GoldenGate Oracle Enterprise Data Quality Database Oracle Data Service Integrator Big Data

  12. Oracle Data IntegratorBulk Data Processing and Data Transformation Oracle Data Integrator • Certified for leading technologies to deliver fast time to value • High-performance, low cost of ownership E-LT architecture • Lightweight deployment • Flexible, easy to enrich functionality Cloud High Performance E-LT Apps Declarative Design Database Extensible Knowledge Modules CEP Data Services Big Data Capable of Integrating Structured & Unstructured Data

  13. Oracle GoldenGate Real-time Data Integration Oracle GoldenGate • High-performance, low-impact real-time data integration and replication • Timely data for improved business insight • Continuous availability for 24/7 operation Cloud Log-based Change Data Capture and Delivery Apps Active-Active Replication Heterogeneous Source and Targets Database Reliability and Transaction Integrity Big Data Java Integration, Real-time Events

  14. Oracle Enterprise Data Quality Data Quality for Customer and Product Data Oracle Enterprise Data Quality • Improves data accuracy, usability and ‘fitness for purpose • Unified interface for ease-of-use, lower TCO • Depth of capability in multiple data domains reduces project risk Cloud Profile, Explore, Audit Apps Parse, Standardize, Cleanse Match, Merge, Enrich Database Governance Big Data Case Management

  15. Oracle Data Services Integrator Data Federation and Virtualization Oracle Data Services Integrator • Easily develop data services and ad-hoc queries without infrastructure changes • Open and standards based—SOA, Web Services, Java, Service Data Object Cloud Federated Data Services Apps Data Virtualization Integrates with Oracle SOA Database Bidirectional Reads & Writes Big Data Integrates with Oracle Business Analytics

  16. Oracle GoldenGate 12.1.2 New FeaturesFeatures Overview

  17. Oracle GoldenGate 12c Simplified Deployment with Intelligent Integration Optimized for Oracle Database 12c Multitenant and Cloud-based Real-Time Replication Integrated Delivery for the Oracle Database Leveraging lightweight Streaming API built Exclusively for Oracle GoldenGate Coordinated Delivery for Non-Oracle Databases Orchestrates the High-Speed Apply Processes & Simplifies Setup and Management Zero Data Loss Downstream Capturing that will run even when the source database is down Expanded Heterogeneity Investing in the Latest Version of Major Database

  18. New Feature Overview Oracle GoldenGate 12.1.2 New Functionality • Extract Enhancements • Integrated Capture • Performance Improvement • Eliminate DDL Trigger • Replicat Enhancements • Integrated Delivery • Coordinated Delivery • Conflict Detection and Resolution Enhancements

  19. New Feature Overview Support New Database/Features • Oracle Database • Oracle 11.2.0.4 specific functionality • Oracle Database 12c • Multitenant Container Database (CDB) • More Datatypes: Large VARCHAR2 • Other Databases • MySQL 5.6 and MySQL Cluster 7.1/7.2 • Sybase 15.7 • Teradata 14.10 • DB2 LUW 10.1

  20. Oracle GoldenGate 12.1.2 New FeaturesFeature Deep Dive

  21. Oracle GoldenGate 12.1.2 New FeaturesIntegrated Delivery

  22. Overview Integrated Delivery • Integrated Delivery for Oracle target databases only • 12.1.0.1+ and 11.2.0.4+ • Leverages database parallel apply servers for automatic dependency aware parallel apply • Minimal changes to replicat configuration • Single replicat parameter file for all tables • Single replicat, no need to use @RANGE or THREAD or other splitting

  23. Integrated Delivery Architecture Diagram Oracle Database Inbound Server Coordinator Preparer Receiver Applier N Applier 2 Applier 1 • Replicat • Reads the trail file • Constructs logical change records (LCRs) • Transmits LCRs to Oracle Database via the Lightweight Streaming API • Inbound Server (Database Apply Process) • Receiver: Reads LCRs • Preparer: Computes the dependencies between the transactions (primary key, unique indexes, foreign key) , grouping transactions and sorting in dependency order. • Coordinator: Coordinates transactions, maintains the order between applier processes. • Applier: Performs changes for assigned transactions, including conflict detection and error handling. Lightweight Streaming API Trail Files Replicat

  24. Integrated Delivery Processing • ADD REPLICAT… INTEGRATED command will automatically register replicat with database • There are options that are controlled through the Replicat parameter file and through the database itself • All configuration done based on Replicat parameter file • Database memory for inbound server is configured with STREAMS_POOL_SIZE, • recommend 1 GB per apply thread. • Parallelism is controlled by INTEGRATEDPARAMS (MAX_PARALLELISM) and is defaulted to 30. • Replicat attaches to database Inbound Server • Unresolved errors are redirected to replicat for handling, which means that if there are a lot of errors, it could affect performance. • Performance Heathcheck Scripts are recommended.

  25. Integrated Delivery Configuration • Create integrated replicat using GGSCI • GGSCI>add replicat repdb1, integrated, exttrail… • Configure using Oracle GoldenGate parameter file. • Specify optional Inbound Server parameters in Replicat parameter file as: DBOPTIONS INTEGRATEDPARAMS(max_parallelism 6) • Modify integrated parameters during runtime: SEND REPLICAT repdb1, INTEGRATEDPARAMS(max_parallelism 10)

  26. Converting to Integrated Delivery • Convert to integrated mode from existing nonintegrated replicat GGSCI> STOP REPLICAT repdb GGSCI> ALTER REPLICAT repdb, INTEGRATED • Updates checkpoint file to indicate INTEGRATED • DB checkpoint table is not maintained when using integrated replicat • Consolidate the parameter files into a single file • Start replicat with proper parameter file • GGSCI> START REPLICAT repdb • Registers replicat repdb database • Starts and attaches to inbound server • INFO REPLICAT repdbwill show INTEGRATED in output

  27. Bidirectional Replication Filtering • Replicat can be configured to tag its changes in the redo • A tag is used to avoid cycling in bidirectional replication .SETTAG 00 • Integrated Extract can use this tag to avoid capturing these changes.TRANLOGOPTIONS EXCLUDETAG 00 • Existing Oracle GoldenGate functionality for eliminating cycles is still available • EXCLUDEUSER, EXCLUDEUSERID, EXCLUDETRANS • GETAPPLOPS, GETREPLICATES

  28. Statistics and Performance Monitoring • Replicat handles some changes directly: • DDL • SQLEXEC, EMI actions, User Exits • Datatypes not currently supported by inbound server (ADTs) Integrated Delivery Statistics: Total transactions:        51           Redirected:          1           DDL operations:        12 Stored Procedures: 0   Datatype functionality:         1           Event actions:         0Direct transactions ratio:        27%

  29. Oracle GoldenGate 12.1.2 New FeaturesCoordinated Delivery

  30. Coordinated Delivery Purpose • Full barrier coordination across events requiring ordering • DDL • Primary Key Update • EMI • SQLEXEC • Ease of Use • Simplify parameter file manageability (fewer param files) • Simplify error recovery • Applicable to all databases

  31. Coordinated Delivery Architecture Checkpoint File Replicat Thread 1 Thread Creation/Shutdown Dependency Coordination Statistics Aggregation Two Way Communication Apply in Parallel Unified Parameter File Trail File … Coordinator Checkpoint Table Database Checkpoint File Two Way Communication Replicat Thread N Checkpoint File

  32. Unified Parameter File (THREAD) REPLICAT (Before) COORDINATED REPLICAT (After) • REP1.PRM • MAP sales.acct1, TARGET sales.acct1; • REP2.PRM • MAP sales.acct2, TARGET sales.acct2; • REP3.PRM • MAP sales.acct3, TARGET sales.acct3; REP.PRM MAP sales.acct1, TARGET sales.acct1, THREAD(1); MAP sales.acct2, TARGET sales.acct2, THREAD(2); MAP sales.acct,3 TARGET sales.acct3, THREAD(3);

  33. Unified Parameter File (THREADRANGE) REPLICAT (Before) COORDINATED REPLICAT (After) • REP1.PRM • MAP sales.acct, TARGET sales.acct, FILTER (@RANGE (1, 3, ID)); • REP2.PRM • MAP sales.acct, TARGET sales.acct, FILTER (@RANGE (2, 3, ID)); • REP3.PRM • MAP sales.acct, TARGET sales.acct, FILTER (@RANGE (3, 3, ID)); REP.PRM MAP sales.acct, TARGET sales.acct, THREADRANGE(1-3, ID));

  34. Apply Coordination • Without THREAD/THREADRANGE specification, transactions are applied by the lowest specified thread • That means that if a THREAD is not specified in a MAP it will default to THREAD(1), this will allow customers that are upgrading to not have to modify their parameter files. • If the user specified COORDINATED in a map, it is applied with full barrier coordination. • MAP scott.deptTARGETnwagner.dept, COORDINATED; • Table DEPT is executed by the lowest (as no thread is specified) with full barrier synchronization • On the other hand: • MAP scott.emp, TARGET nwagner.emp, THREAD(2); • Table EMP is executed by Thread 2 assuming no need for coordination. • If a transaction touches both EMP and DEPT, it will get executed by the Barrier thread with Coordination.

  35. Creating Coordinated Delivery Groups • GGSCI> add replicat C_REP, COORDINATED, exttrail …, MAXTHREADS10 • Creates a coordinated replicat with 10 threads. • Creates 11 processes in total • C_REP1 (Coordinator) • C_REP1001 (Thread 1 that is created by default) • C_REP1002, C_REP1003, C_REP1004…. C_REP1010 (Threads 2, 3 … etc.) • Each process has its own checkpoint file, report file and is a fully functioning replicat. • If MAXTHREADS is not specified, 25 threads are created by default. • Replicat group name (excluding generated characters ) must be 5 characters maximum.

  36. Key Concepts Coordinated Delivery vs. Integrated Delivery

  37. Oracle GoldenGate 12.1.2 New FeaturesSupport for Oracle Database New Features

  38. Large VARCHAR Support • Extends VARCHAR2 limit from 4000 to 32764 • Internally stored as LOBs • Cannot be logged supplementally • Conflict detection and resolution not supported • Also applies to RAW, NVARCHAR2 • Support limitations • Table must have a unique/primary key not containing the 32K column • Trail Version has to be 12.1 • Classic Capture will not support this

  39. Large VARCHAR Support Extends VARCHAR2 limit from 4000 to 32764 Internally stored as LOBs Cannot be logged supplementally Conflict detection and resolution not supported Also applies to RAW, NVARCHAR2 Support limitations Table must have a unique/primary key not containing the 32K column Trail Version has to be 12.1 Classic Capture will not support this

  40. Multitenant Container Database Support Idea is to use common pool of resources to handle multiple databases SGA Background Processes One stream of redo One UNDO Containers with user data are called Pluggable Database (PDB) PDBs can be unplugged from one CDB and plugged into another CDB ALL PDBs share a common redo stream CDB$ROOT is a special container CDB$ROOT cannot be unplugged or plugged Extract must be configured at CDB$ROOT

  41. Specifying PDBs Object Name Two part name: SCHEMA.TABLE Three part name: CATALOG.SCHEMA.TABLE No default schema is allowed in three part names User must specify SCHEMA part when specifying CATALOG part We continue to allow default schema if CATALOG part is not used Three part and two part names cannot coexist in the same Trail file or table definition file (defgen)

  42. CDB Support Overview Extract (Capture) Only supported in Integrated Extract mode Can mine many PDBs with a single Extract (REGISTER EXTRACT) Can write changes from multiple PDBs in a one or more trail files Replicat (Apply) Supported only at a PDB level Can read trail file with changes from multiple PDBs

  43. Oracle GoldenGate 12.1.2 EnhancementsIntegrated Capture

  44. Integrated Capture DDL Trigger Removal • Requirements: • Database must be running Oracle Database 12c (12.1.0.1), 11.2.0.4 and later versions. • DDL Trigger Removal is required for CDB. • Behavior • Integrated Capture running against Oracle Database 12c or 11.2.0.4 will automatically run in this mode • No need to run DDL setup scripts • DDL trigger can remain enabled • Integrated Capture running against Oracle Database 11g Release 11.2.0.3 will get metadata gathered from DDL trigger • Classic Capture will continue to require existence of DDL trigger

  45. Oracle GoldenGate 12.1.2 New FeaturesEase-of-Use Enhancements

  46. Agenda- Ease of Use Enhancements Automatic Discard Improved SHOWSYNTAX Support Improved Initial Load Support Schema Wildcard Support

  47. Wildcard Schema and Catalog Applications support wildcard schema and catalog ggsci defgen extract replicat Wildcard schema support is available for all databases Wildcard catalog support is available for Oracle CDB and SQL/MX Oracle CDB with PDB login does not support wildcard catalog

  48. Wildcard Schema and Catalog (cont.) SOURCECATALOG parameter SOURCECATALOG pdb1 TABLE schema.table; -- captures pdb1.schema.table TABLEEXCLUDE, SCHEMAEXCLUDE and CATALOGEXCLUDE TABLEEXCLUDE behavior is same by default (EXCLUDEWILDCARDOBJECTSONLY changes behavior) SCHEMAEXCLUDE excludes by schema level CATALOGEXCLUDE excludes by catalog level

  49. Wildcard Schema and Catalog (cont.) Duplicate target table checking is enabled for extract -- only captures first TABLE parameter for SCOTT.EMP DML TABLE SCOTT.EMP, COLMAP(…); TABLE *.EMP; -- example of duplicate target w/wo TARGET parameter TABLE NWAGNER.EMP; TABLE *.EMP, TARGET NWAGNER.*;

More Related