How to Achieve 25TB/hour Backups and 15TB/hour Restores for Your Oracle Database

2. How to Achieve 25TB/hour Backups and 15TB/hour Restores for Your Oracle Database Doug Chamberlain, Principal Product Manager Greg Drobish, Senior Software Engineer Gagan Singh, Intel Corporation

3. Program Agenda • Engineered Systems Backup Challenges • Oracle ZFS Storage • Exadata Backup Test Results • Customer Case Study: Intel Corporation

4. Oracle Engineered Systems Requires Extreme Protection

5. Top Backup and Recovery Concerns • Risk • Data loss – 1 in 7 backups & 1 in 6 restores fail* • Slow data recovery leads to business outages • Missed SLAs lead to lost opportunities • Data-at-rest security and regulatory concerns • Complexity • More systems and storage to integrate • More data and shorter backup windows • Separate database and storage processes • Managing Budgets • Reducing CAPEX & OPEX • Improving staff efficiency • Freeing resources for investment * Backup and Restore Failure Rate from Enterprise Strategy Group, 2008

6. Program Agenda • Engineered Systems Backup Challenges • Oracle ZFS Storage • Exadata Backup Test Results • Customer Case Study: Intel Corporation

7. Oracle’s New ZS3 Storage Systems World-Record Performance Trusted Storage Architecture • Fastest database queries (GB/s) • Fastest database responsiveness (latency) • Over 100M hours in production environments • Relied on by the majority of the Global F100 • Foundation of Oracle *IT and Oracle cloud Best-in-Class Efficiency • 3rd Gen. Hybrid Storage Pool (1/5th $/IOPS) • 2-4x more I/O efficient than everyone • Advancements in award-winning analytics Best for Oracle Software • Only with OISP (cut DB tuning time in ½) • Best-in-class HCC (50x compress, block/file) • Integrated ADG, EM, OVM, and Eng. Systems

8. ZFS Storage Appliance for Oracle Engineered Systems Backup & Recovery Application Engineered Storage for Oracle Engineered Systems Backup Less Risk with Extreme Performance 25TB/hour backup, 15TB/hour restore Less Complexity with Fewer Storage Systems and Streamlined Management Lower TCO with Extreme Efficiency

9. Oracle ZFS Storage Appliance Advantages Co-Engineered with Oracle Database and Engineered Systems Less Business Risk • Short backup windows with Infiniband and optimized Direct NFS • Data reduction with encrypted data • Fast, predictable restore times • Integrated with StorageTek tape tier for long-term archiving Less Complexity • HCC compression and higher performance mean fewer storage systems • Secondary processing with no additional storage systems • Configuration Utility for fast, foolproof setup • No changes to Oracle best practices Lower TCO • Fewer systems for lower capital, integration, and operating costs • Unsurpassed backup visibility with DTrace Storage Analytics • Snap Management Utility for DBA management

10. Oracle ZFS Storage is Fast • Native InfiniBand Connectivity • Highest bandwidth for general purpose storage • Scales horizontally, allowing for upgrades • Leverages engineered systems’ internally managed InfiniBand network, no network upgrade required • Backup and restore operations can be automatically parallelized across all database nodes and Sun ZFS Storage Appliance channels and controllers • Restore throughput • 17TB/hr for Exadata • 7 TBhr for SuperCluster • Backup throughput • 26TB/hr for Exadata • 14TB/hr for SuperCluster

11. ZFS Storage Is Cost Effective and Complete • Simple architecture reduces cost • If not using tape, no media server required • No backup and recovery software license fees • Augment Oracle DR utilizing replicated copies of RMAN backups • Full complement of data services available • Thin provisioning • Compression • Snapshot • Performance Analytics • Layers of data integrity • RMAN protects against logical corruption, ZFS Storage protects against physical corruption • Self-healing Fault Management Architecture

12. Compression Unit Hybrid Columnar Compression 10x to 15x Storage Space Savings, Unique to Oracle Column 1 Column 4 Column 2 Column 5 Column 3 • Ingests RMAN image backups of HCC compressed tables • Snaps and clones RMAN image backup • Access cloned databases for test and development • Preserve space savings – no need to uncompress cloned tables before accessing 10x to 15x Reduction

13. Configuration Utility Simplifies Deployment Exadata Backup • The Oracle Exadata Backup Configuration Utility configures ZFS Storage and Exadata nodes through command-line-based Wizard • Required manual steps automated through utility; best practice configuration can be enforced • Available via Oracle Technology Network

14. Stay With Current Best Practices • Implementation • Tested, validated, and supported • Integrated InfiniBand simplifies implementation • Consistent with standard RMAN methodology • Use a fast recovery area • Perform an initial RMAN image copy backup • Perform daily RMAN incremental level 1 backups • Effectively an “incrementals forever” strategy • RMAN incremental only back up of changed blocks • Full backups unnecessarily burden the network

15. Simplifying Disk-to-Disk-to-Tape • Disk-to-disk-to-tape (D2D2T) allows backup administrators to have the best of both worlds • Keeping off-site backups elsewhere • While retaining fresh backups on disk for ultra-fast restores • Storage hierarchy that balances cost and performance • Automated data movement enabled by Oracle Secure Backup or third-party backup and recovery applications • Backup Backupset: RMAN-aware backup of disk backup on tape • RMAN Archive to tape: managed and tracked by RMAN

16. Disk-To-Disk-To-Tape Tape Backup Media Management Serverwith Oracle Secure Backup, or third-party backup & recovery software Backup Backupset RMAN StorageTek SL3000Tape Library ZFSStorageAppliance Oracle ExadataDatabase Machine

17. Program Agenda • Engineered Systems Backup Challenges • Oracle ZFS Storage • Exadata Backup Test Results • Customer Case Study: Intel Corporation

18. Exadata Backup Test Results Controllers • Clustered ZS3-4 • 1 TB DRAM (512GB per controller) • InfiniBand Connectivity • ZFS Storage OS8 Software • No Read Cache SSD • Peak Sustained Backup: • 23 TB/hr (backupset) • 26 TB/hr (imagecopy) • Peak Sustained Restore: • 15 TB/hr (backupset) • 17 TB/hr (imagecopy) Enclosures • 8 High Capacity Disk Shelves • 7200 RPM drives • No Write Log SSD

19. Superior RAID-Z Performance • High performance RAID-Z gives customers the choice of mirrored or striped parity based on their needs • Variable stripe width means every write is a full stripe write • RAID-Z reduces the IOPS overhead RAID-Z Parity Ap A1 A2 A3 B3 B1 B2 Bp C3 C1 Cp C2 Disk4 Disk1 Disk2 Disk3

20. RAID-Z1 vs Mirror: Calculating IOPS Overhead • Single Parity • 158.2 IOPS per every 100 MBPS • Average disk IO size: 700.8 KB • Mirrored • 250.6 IOPS per every 100 MBPS • Average disk IO size: 841.1 KB Mirroring Single Parity Conclusion • Mirroring generates 58% more IOPS • 5x higher DB wait time • IOPS per disk correlate to AWR average wait time • IOPS per disk correlate to AWR average wait time

21. Optimization For Image Copy or Backupset Image Copy: • Infinite Incremental, synergies with cloning, superior performance, generally accessible Backupset: • Tape support, bypasses unused datafile blocks, RMAN default, multi-section backup Backupset Tuning Image Copy Tuning • Fewer channels • More buffers per channel • 1MB buffer sizes • More channels • Fewer buffers per channel • 4MB buffer sizes

22. RMAN Tuning Recommendations Multi-Section Backups • Use filesperset=1 for all backupset and incrementals to improve buffer performance and streamline restores • Set _file_size_increase_increment=2044M to optimize how RMAN allocates new space for backups • Use section size 48g to optimize multi-section backups • Configure Oracle Net Services for optimal performance and availability • Reset or remove _backup_ksfq_bufsz and _backup_ksfq_bufcnt deprecated parameters RMAN Backup Services

23. Save Money on SSD, Spend it on HDD • SSD is critical for achieving optimal transactional and mixed-IO performance • Dedicated RMAN workloads may perform better without SSD Incoming Writes TXG Synchronous Writes • Transaction groups transfer data from the ARC to HDD • All writes whether synchronous or asynchronous are first written to the ARC • Synchronous writes are committed to the ZIL before an acknowledgement is returned to the client

24. SSD Is Not Always Faster • RMAN backupsets require synchronous semantics but are not very latency sensitive • IO queuing can create a bottleneck and reduce throughput • Set logbias=throughput will bypass SSD write cache (if present) and write to HDD IO Queuing Sharing the Load • Bandwidth intensive workloads benefit from wide IO distribution across the storage pool • Analytics shows that IO queuing on the 4 SSDs is creating a bottleneck

25. More HDD Improves Performance • Peak sustained throughput increases as more disk are added to the ZS3 configuration • For optimal performance always use a single storage pool per controller for RMAN backup/restore • High performance drives scale RMAN performance at a steeper rate

26. Infiniband Enables Superior Performance • Infiniband fabric provides an extremely high performance network for transferring backup data • Low-latency, high-bandwidth (2.1 GB/s), low CPU utilization • Seamless integration with Exadata • Use an active/active IPMP configuration Active/Active CPU • Peak throughput per controller with low CPU overhead • Active/active IPMP group provides performance and redundancy

27. Direct NFS • dNFS provides integrated performance tuning when backing up an Oracle database to a ZS3 • Bypasses the operating system • Data is cached just once in user space with no 2nd copy in kernel space • Load balance across multiple network interfaces and stripe network buffers over up to 4 addresses OISP • Oracle Intelligent Storage Protocol (OISP) integrated with dNFS in 12c dynamically tunes record size and logbias variables on the ZFS Storage Appliance to optimize performance

28. Program Agenda • Engineered Systems Backup Challenges • Oracle ZFS Storage • Exadata Backup Test Results • Customer Case Study: Intel Corporation

29. Oracle Exadata Database MachineBackup and Recovery with the Sun ZFS Backup Appliance • Gagan Singh • Sr. Database Architect • Intel Corporation

30. Agenda

31. Intel CorporationCompany Overview • U.S.-based, publicly traded multinational semiconductor chip maker corporation headquartered in California • Designs and manufactures advanced integrated digital technology platforms • Microprocessor and chipset • Develop, sell software and services primarily focused on security and technology integration • $53.3B in revenue for 2012* * http://www.intc.com/intel-annual-report/2012/

32. Legacy Data Warehouse Environment • Manufacturing Data Warehouses -Highly automated environment with complex integrated systems -Geographically independent systems, decentralized environment • Reliability, Availability and Performance • Robust Backup and Recovery, but lacked performance - Affects scheduling • No formal VLDB Disaster Recovery -Data not protected from loss of a site

33. Previous Backup Architecture • All Oracle Databases backed up to Flash Recovery Area • RMAN Disk-to-Disk-to-Tape backup strategy • Backup Strategy • Level 0 to disk; Block Change Tracking for daily Level 1 backups • Incrementally updated: L1 merged into L0 • Weekly L0 and Daily L1 also stored on tape • Challenges • Image copies needs same space as database size • Several operational issues managing backups • Tape bandwidth is limited • Backup load on Primary Server/Database FRA Oracle Database Server NetBackupMedia Server Virtual Tape Library

34. Impetus for Exadata Purchase • Increase data retention without impacting SLA by leveraging compression; hardware reliability; compute power • Compelling Value Proposition • Exadata X3-8 for production and one half-rack for pre-production • Compression • Redundancy • Platinum Support • Scalability • Faster backup & recovery times • I/O Resource Manager

35. Why ZFS ?

36. New Backup Architecture • ZFS Storage Appliance • Connected via InfiniBand to Exadata • RMAN Backupsets from Active Data Guard Host • Centralized Backup from EM 12c • Backup Monitoring through EM 12c

37. Lessons Learned • DnFS enabled, channel distribution across Exadata Nodes • Split loads evenly on both controllers • Ability to define logical space within ZFS appliance. • Leverage the ZFS GUI for real time Monitoring and Operations • _backup_disk_bufcnt=64  • _backup_file_bufcnt=64

38. Reference Notes • Sun Storage 7000 Unified Storage System: FAQ: RMAN backup from Exadata to Sun ZFS Backup Appliance (Doc ID 1354980.1)

39. Thank You

40. Additional Resources • Backup and Recovery Performance and Best Practices using Sun ZFS Storage Appliance with Oracle Exadata Database Machine http://www.oracle.com/technetwork/database/features/availability/maa-wp-dbm-zfs-backup-1593252.pdf • Backup and Recovery Performance and Best Practices for Exadata Cell and the Sun Oracle Database Machine http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-11202-183503.pdf • Exadata Backup & Recovery My Oracle Support Note:https://support.oracle.com, ID 1354980.1 • SuperCluster Backup & Recovery My Oracle Support Note:https://support.oracle.com, ID 1517107.1 • Oracle Exadata Database Machine: http://www.oracle.com/exadata • Oracle SuperCluster: http://www.oracle.com/supercluster • Oracle Recovery Manager: http://www.oracle.com/goto/rman • Sun ZFS Storage Appliance: www.oracle.com/us/products/servers-storage/storage/unified-storage