1 / 21

MicroTerabyte

MicroTerabyte. Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA. Background. Exploding data volumes are presenting new challenges to small and medium sized organizations.

rendor
Download Presentation

MicroTerabyte

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MicroTerabyte Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA

  2. Background • Exploding data volumes are presenting new challenges to small and medium sized organizations. • These organizations need a new generation of technology that delivers powerful analytical capability with reduced cost and complexity. • CMA, in partnership with Dell, QLogic, Oracle, and EMC, has developed a sophisticated solution to address this growing need.

  3. MicroTerabyte PlatformWhat is It ? • A pre-configured, integrated, fault tolerant, high performance, commodity hardware and software based runtime environment • Leverages the power of the Oracle database running on commodity Linux servers, and a unified InfiniBand fabric over QLogic multi-protocol fabric directors • Provides a scalable, reliable, lower cost platform for Business Intelligence, Custom Software, or Commercial Software

  4. MicroTerabyte PlatformFeatures • High Performance • Scalable • Fault Tolerant • Commodity Hardware • Lower Cost • Smaller Footprint • Reduced Power Consumption

  5. MicroTerabyte PlatformAttributes • Red Hat Linux Operating System • Commodity Hardware (Dell / EMC CX / QLogic) • 42U Rack Footprint with Fault Tolerant 42U rack • Clustered Oracle Database • Clustered Storage Provisioning Layer • Unified Storage and Interconnect Fabric

  6. MicroTerabyte Single Rack Configuration • Sample Hardware Components (per rack) • (2) Dell 1950 Database Servers • 32GB RAM Each • 16 Processor Cores • (1) Dell 1950 ETL Server • 16GB RAM • 8 Processor Cores • (1) Dell 1950 Business Intelligence Server • 16GB RAM • 8 Processor Cores • (1) QLogic 9020 InfiniBand Fabric Director • (2) FVIC Modules • (2) EMC CX3-40 Storage Arrays • 2TB – 5TB Storage • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console Physical Data Guard

  7. MicroTerabyte Mid-Range Configuration • Hardware components (two racks) • (4) Dell 1950 Database Servers • 32GB RAM Each • 32 Processor Cores • (1) Dell 1950 ETL Server • 32GB RAM • 8 Processor Cores • (1) Dell 1950 Business Intelligence Server • 32GB RAM • 8 Processor Cores • (1) QLogic 9040 InfiniBand Fabric Director • (2) FVIC Modules • (2) EMC CX3-40 Storage Arrays • 8TB – 15TB Storage • (2) EMC SATA 1TB Drives (for backup) • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console

  8. MicroTerabyte Large Scale Configuration • Hardware components (three racks) • (8) Dell 1950 Database Servers • 32GB RAM Each • 64 Processor Cores • (2) Dell 1950 ETL Server • 32GB RAM Each • 8 Processor Cores • (2) Dell 1950 Business Intelligence Server • 32GB RAM Each • 8 Processor Cores • (1) QLogic 9040 InfiniBand Fabric Director • (4) FVIC Modules • (4) EMC CX3-40 Storage Arrays • 24TB – 32TB Storage • (2) EMC SATA 1TB Drives (for backup) • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console

  9. Compute Nodes Unified Fabric Layer Storage Nodes MicroTerabyte ArchitectureMid-Level Diagram Dell 1950 PowerEdge Servers / RHEL v5 QLogic Multi-Protocol Fabric Director EMC CX3 / CX4 Flare OS / Navisphere

  10. ERP Application Package ISV Third Party Software CMA BI Suite Business Intelligence MicroTerabyte ArchitectureDetailed Diagram Applications Oracle Grid Control 11g MT Clustered Database / Storage Provisioning Oracle EE 11g w/ RAC and Partitioning Oracle ASM 11g MicroTerabyte Platform QLogic OFED InfiniBand Drivers Oracle Clusterware 11g MT O/S Binding Red Hat Linux O/S Oracle ASMlib Public Networking Dell PowerConnect 6248 Server Compute Dell 1950/2950 + QLogic 7104 HCA MT Core QLogic 9020/9040 Multi-Protocol Director Unified Fabric Storage Infrastructure EMC CX3/CX4

  11. MicroTerabyte PlatformOracle Software

  12. MicroTerabyte Compute NodesDell 1950/2950 • MicroTerabyte solution consists of 2/4/8 RAC Database nodes • 2 ETL and BI Nodes • Each node is a Dell 1950/2950 consisting of: • Processor: One/Two quad-core Intel Xeon X5355 @ 2.66GHz • Memory: 16-64 GB • Hard drives: 218GB-3TB Internal Storage • RAID Controller: PERC 5/i • 1 DDR InfiniBand HCA • Network interface cards: Dual gigabit NICs (100baseTx-FD) • Power supply: 670W, optional hot-plug redundant power (1+1) • Operating system: RedHat Enterprise Linux v5

  13. Front-End Host Connectivity • Two storage processors per CX4 • Each processor has: • Four 4Gb Fibre Channel optical ports • FCP SCSI-3 protocol • Back-End Disk Connectivity • Each processor has 4Gb Fibre Channel arbitrated loops. • Multiple RAID groups may be distributed across redundant loops • Supports a maximum of 480 disk drives • System Memory • Each processor has 8GB of Memory • Power Consumption (Processor Chassis) • 355 VA (290W max) MicroTerabyte Storage NodesEMC CX4 Model 480 • Power Consumption (Disk Expansion Chassis) • 440 VA (425W max)

  14. MicroTerabyte Unified Fabric LayerWhat is it? • QLogic 9020 & 9040 Multi-Protocol Fabric Directors • 9020 with two (2) FVIC IB-FC Virtual I/O Controllers • 9040 with up to four (4) FVIC IB-FC Virtual I/O Controllers • each FVIC provides 10 DDR (20Gb) IB ports & 8 4Gb FC ports • supports up to 128 Virtual HBA ports per module • automatic sensing 1/2/4 Gb/s • load balancing • automatic port and module fail-over • LUN mapping and masking features • QLogic 7104 Host Channel Adapters • Dual Port, DDR • IPoIB, RDS, SRP

  15. MicroTerabyte Unified Fabric LayerGeneral Benefits • Managing one fabric • Reduced footprint • Compact implementation • Fewer host components needed to support I/O and interconnect • Increased bandwidth and reduced latency • Reduced host resources (1 HCA vs. several HBAs) • Path failover through SRP protocol • Well positioned to take advantage of advances in InfiniBand technology

  16. MicroTerabyte Unified Fabric LayerOracle RAC Benefits • Scalable platform to support Oracle RAC • More predictable response times • Capability to drive more Oracle I/O through fewer compute nodes • Ability to exploit storage capability at a lower cost • Reduced Oracle messaging latency via RDS/IB

  17. MicroTerabyte ORION Benchmarks CONFIGURATION A mid-size MicroTerabyte configuration, including: Servers: Four (4) Dell 1950 Intel quad core servers. Each 1950 includes 8 cores, 16GB memory and 1 dual channel HCA. The server is running Red Hat Linux 5 update 1 (2.6.18-53 kernel). Storage: Two (2) EMC CX3-40 storage arrays. Each storage array includes 8 4Gbps front-end fiber channel connections for a total of 16 4Gbps FE adapters -- approx 7.5TB usable storage configured in 4+1 RAID sets. Unified Interconnect and Storage IB fabric: One (1) QLogic 9040 Multi-Protocol Director with 2 FVIC modules HARDWARE COST Approximate total cost (market): $500,000 TEST METHOD AND RESULTS Testing simulator: Oracle ORION -- Oracle I/O Numbers Calibration Tool for Linux >> Random I/O Test Results: Test Type: random I/O I/O size: 8K Rate observation 1: 9,600 sustained IOPS @ 1.87 ms per I/O avg node latency Rate observation 2: 30,566 sustained IOPS @ 2.96 ms per I/O avg node latency Rate observation 3: 42,615 sustained IOPS @ 2.96 ms per I/O avg node latency >> Sequential I/O Test Results: Test Type: sequential I/O I/O size: 1MB Rate observation 1: 677MB/sec sustained I/O seq. throughput @ 26.48 ms per 1MB I/O avg node latency Rate observation 2: 2.098GB/sec sustained I/O seq. throughput @ 91.48 ms per 1MB I/O avg node latency

  18. MicroTerabyte Oracle Benchmarks Hardware Servers: (4) Dell 1950 Dual Socket Quad Core 16 GB RAM per server Single dual port IB HCA Unified I/O Fabric: QLogic 9040 (2) FVIC Modules Storage: (2) EMC CX-3 Model 40 (16) Fibre Channel ports Software O/S: Red Hat Linux v 5.1 Kernel 2.6.18-51 UDEV Unified I/O Fabric: QLogic IB drivers v 4.2.0.0.39 IB/RDS IB/SRP IPoIB Database: Oracle RAC 10gR2 Oracle Clusterware 10gR2 Oracle ASM Oracle RMAN Oracle DataGuard Storage: Flare O/S EMC Navisphere Backup: EMC Networker

  19. MicroTerabyte Benchmark Comparisons

  20. Summary As demonstrated in CMA’s MicroTerabyte platform, Infiniband can provide an extremely capable transport mechanism for unifying interconnect and i/o traffic Increases bandwidth and reduces latency Reduces Oracle messaging latency via RDS/IB Provides more predictable response times Reduces host resource requirements (i/o processing workload off-loaded to HCA card) Consistent with Oracle’s strategic technology direction

  21. For More Information Brian Dougherty Chief Architect, CMA bdougherty@cma.com

More Related