1 / 31

Large Data Joint Capability Technology Demonstration

Explore how the Large Data JCTD enables information sharing across the DoD/IC enterprise with resilient failover, enhanced transport, and dense shared storage. Utilizing open standards and commodity hardware for optimal performance and scalability.

rustyc
Download Presentation

Large Data Joint Capability Technology Demonstration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Joint Capability Technology Demonstration OSD (RFD) – USSTRATCOM – NRL – NGA – INSCOM – DISA

  2. Warfighter Problem Large Data Concept of Operations Operational Utility Assessment Why LD JCTD Works Transition Summary Agenda

  3. LD Enables Information Sharing across DoD/IC Enterprise Global enterprise data available “as if on local desktop” Warfighter Data Combatant Command Ops & Intel Enterprise • High-performancedataaccess • Resilientfailover/Continuity of Operations • Enhancedtransport • Extremely dense, sharedstorage Intelligence Community (IC) Enterprise Other Enterprise Data CALA Medical, Weather, Logistics, etc

  4. Application Servers Web Servers Work Stations Net-File Servers Database Servers File System Client File System Client File System Client File System Client File System Client File System Client Large Data Reference Model Graphic Stations Wide Area Networks • Open Standards • OFED • Key Interfaces • InfiniBand DDR/SDR • Ethernet • Software Standards • SuSE Linux • OMAR (OGS WMS) • Lustre (POSIX) • Open Source Software • Lustre File System • Commodity Hardware • Intel Servers • Multiple suppliers for • modular components • Fully MOSA Compliant Architecture InfiniBand Switches WAN Bridge Workgroup Ethernet Type 1 Encryptor Ethernet Switches IP, SONET, or Lambda Links File Servers Storage Appliance Storage Appliance Client Application Domain Shared File System • Links to Disks: • Fibre Channel • SCSI • SATA II • SAS Disk Arrays Key Interfaces: InfiniBand Ethernet IP or SONET • Examples: • Standard Workstations • Thin Clients • File System Clients Core Large-Data Domain Scalable to 1000’s of Nodes — Petabyte File System today…Exabytes tomorrow

  5. Large Data DISN Interface • LD architecture efficiently exploits • GIG-BE/DISN • Cost effective system interfaces can • take full advantage of 10 Gbps • Flexibility to utilize all encryption • solutions • Options to leverage lambda, IP/MPLS, • Ethernet services • Options to connect to current DISN Interfaces • 10 Gbps lambda • 1 Gbps IP for smaller clients • 2.5 Gbps PT-to-PT • Options to connect to future DISN Interfaces • 40** Gbps lambda • 10/40** Gbps IP/MPLS service* • 10/40** Gbps Ethernet service* • * Requires no new development; not yet validated in LD • ** 40 Gbps hardware development underway Large Data JCTD Data - Centric Info Services, Collaboration, Visualization, Applications DISN Servers IP - Centric Global File System Voice, Video, and Data All security domains (e.g. Unclassified etc.) Storage Virt. RDMA IP Lambda Access IB & Ethernet IP DWDM FIBER

  6. Standard 2-Rack Node • Scalable & Flexible • Extremely Dense • High Performance • Low Latency • Open Standards • Open Systems • Open Source • Commodity Parts

  7. LD Storage-Processing Solutions 1.2 PB • Availability: Access to global data – “as if on desktop” • Availability: Drive failure tolerance, typically 8+2 parity, Raid6 technology • Availability: Journaling, providing the Reduction / Elimination of LUN rebuilds through I/O bit mapping • Reliability: Detection and protection against silent data corruption • Parity calculation on all reads • On-the-fly-data-correction • Space: Extreme disk density (150TB/ft2) • Performance: Delivers sustainable real-time raw large block I/O at near line rates – 8 GBps • Economics: Sleep mode for power/cooling savings 75 TB 22,400 cores – 1540u up to 11.8 PB 5-16u 90TB Man-portable • High Performance = Low Latency Access to Critical Data

  8. LD Web Services • LD provides “point and click” access to globally distributed data • Thin-client access with thumbnail previews • Provides near real-time updates to planning and intelligence applications

  9. Use of Open Technology in JCTD • Lustre Key benefits • Open source, multi-platform and multi-vendor under GPL • Aggregates petabytes of storage and tens of thousands of nodes into one file system • Production-quality stability and high availability • POSIX-compliant • Object-based architecture • Windows support with pCIFS • Heterogeneous networking • RDMA performance enhancements with • InfiniBand networks • Manual and automated failover • All designs and the internals course are on lustre.org web site • CVS tree (repository of all code as it develops) is open to all • Architecture discussions open to all on lustre development mail list • Managed by a Fortune 500 major technology company (Oracle Sun) and in wide use among high performance systems • Lustre • Providing scalable, global distributed file system • Leverages DoE investment and Sun Microsystems backing • OSSIM Mapping Archive (OMAR) • Provides dynamic web services • Leverages IC investment with Radiant Blue • Open Federation Enterprise Development (OFED) • Industry consortium for an open standard high performance transport, storage and processing protocol stack

  10. LD JCTD Operational Utility Assessment Enhance situational awareness by enabling global data access Large Data LD LD LD LD LD “While the first of about 10 files was still being transferred to the legacy work station, Large Data had all ten files.” LD JCTD IOUA Report, Nov 09, JHU/APL Globally Synchronized, Shared Data and High Resolution Collaboration

  11. LD Site Backend Processing Comparison Sites 1 & 2 orthorectify and display Imagery to thin clients (on-demand); Site 3 converts MPG-2 to FLV (on-demand) and streams video to clients Site 1 AVG = 3.77 sec Site 2 AVG = 4.44 sec For Imagery Processing Site 3 AVG = 9.35 sec For Video Processing LD Web Services (Backend) No discernible difference to user for remote (~1200 miles*) vs. local imagery processing *~2000 fiber miles

  12. Operational Demonstration Results • Operational Impact • GEOINT access & web services for warfighters • UNCLAS US Gov’t and NGO support • Remote access to large, distributed ISR files • Data virtualization & near real-time failover • “Simply put, the system NRL has in place for delivering large, AOI detailed imagery is outstanding and truly a model for the DoD/IC.” • – Sean Wohltman, Google Inc. Rampant Lion on www http://mapserver.cmf.nrl.navy.mil/ • Suitability • Demonstrated TRL-7/8/9 • Cost effective, open source • Commodity components • Stability/availability on par with • operational systems • “A quick overview of the system was all that was required for ease of use” • -- LD User Effectiveness: LD met or exceeded Transition Thresholds

  13. LD System Performance • Test 2: Remote Access Results:User “accessing” vs. “copying” file from remote file system to local workstation • RDMA-enabled • Access “as if local” to large ISR files • over MAN/WAN/Long Haul WAN • Matches local performance • Saves 99.9% of time over “copy to • local storage & open”! Test 1: File System-to-File System Data Results: 91%-99% of Max Theoretical Data Transfer Rate • Test 3: Data Stream Results: • File System-to-User Workstations • OC-192 Link: • -- MAN: 808 MBps (85.9% Max) • -- WAN: 759 MBps (80.8% Max) • OC-48 Partial Link: • -- LH Wan: 160 MBps (75.4% Max) OFED RDMA / Lustre file system enables remote access and/or very rapid file delivery to user applications

  14. LD Network Performance Typical RDMA/IB Performance Typical RDMA/IB Performance TESTS ON 8 Gbps CIRCUIT (~1200 miles) [~2000 fiber miles] Typical TCP/IP/ETHERNET Performance • RDMA/Infiniband (IB) provides highly efficient use of available bandwidth • IB scales well with multiple, concurrent data flows • RDMA/IB performance ≥ 80% • TCP/IP performance ≤ 40% • RDMA CPU usage estimated 4x less • IB is lossless and provides “fair share” of bandwidth TESTS ON 1 Gbps CIRCUIT (~8000 miles) [ ~13,000 fiber miles]

  15. Large Data DISN Interface • LD architecture efficiently exploits • GIG-BE/DISN • Cost effective system interfaces can • take full advantage of 10 Gbps • Flexibility to utilize all encryption • solutions • Options to leverage lambda, IP/MPLS, • Ethernet services • Options to connect to current DISN Interfaces • 10 Gbps lambda • 1 Gbps IP for smaller clients • 2.5 Gbps PT-to-PT • Options to connect to future DISN Interfaces • 40** Gbps lambda • 10/40** Gbps IP/MPLS service* • 10/40** Gbps Ethernet service* • * Requires no new development; not yet validated in LD • ** 40 Gbps hardware development underway Large Data JCTD Data - Centric Info Services, Collaboration, Visualization, Applications DISN Servers IP - Centric Global File System Voice, Video, and Data All security domains (e.g. Unclassified etc.) Storage Virt. RDMA IP Lambda Access IB & Ethernet IP DWDM FIBER

  16. Shared IP/MPLS Test Configuration encryp WAN GW • IP/MPLS Configuration Testing goals: • Validate carrying LD JCTD • traffic over a HAIPE/ • IP/MPLS network • Validate feasibility of • operating at scalable • provisioning levels • (2, 3, 4, 5, … Gbps) • Test LD JCTD in the • presence of competing • IP traffic Ciena Site 1 encryp WAN GW Ciena Ciena Site 2 Site 2 DISA Lab NRL Lab

  17. RDMA/IB Performance on DISN Core RDMA/IB performance with competing traffic on 10 Gbps IP/MPLS network • Validated: • RDMA/IB traffic over a shared IP/MPLS core network in a DISA lab • RDMA/IB network transactions performed well at a range of operating levels Excellent RDMA results over IP core up to assigned BW Expected RDMA results over IP core beyond assigned BW This is the approx. volume of datathat the node attempted to send. RDMA performs well within DISN assigned bandwidth

  18. Final Configuration Initial Configuration Database Database VM0 VM0 Final Database Host Initial Database Host VM3 VM4 VM4 VM3 Linux VM Linux VM VM1 VM1 VM2 VM2 Site 1 Server Site 1 Server Site 2 Server Site 2 Server Mirrored VM Storage Mirrored VM Storage VM5 VM5 Site 1 Site 1 Site 2 Site 2 VM6 VM6 Applications Operator Initiated Migration Linux VM Ethernet Switch Ethernet Switch VM Hypervisor VM Hypervisor WAN Bridge WAN Bridge Storage (IB) Switch Storage (IB) Switch Ethernet InfiniBand Site 1 Data Site 2 Data Virtual Storage Controller Virtual Storage Controller VM Storage VM Storage “Distributed RAID 1” Mirrored Storage Site 2 Site 1 LD Data Virtualization/Failover Various combinations of data migration from locations ~1200 miles apart Shared Resource Cluster Shared Resource Cluster High performance computing/storage for virtualized enterprise Remote mirroring allows VM clusters to include servers from different remote data centers

  19. Why the LD JCTD works client client client The LD JCTD demonstrated the use of RDMA and a clustered Global File System over long distances to create a globally accessible storage and compute cloud client Data Center Data Center Data Center • Data available to clients anywhere in the world • Remote clients may disconnect at will • Centralized apps available to clients Global File System + • LD JCTD used RDMA and parallel file systems to build multiple scalable, cost effective data centers • LD JCTD extended RDMA over a high bandwidth WAN to virtualize the data centers • Clients connect to virtual data center via cost effective, low bandwidth (1 Gbps) IP networks RDMA interconnect LD JCTD used standards-based COTS technology and components to demonstrate a secure, cloud computing infrastructure operating over the DISN

  20. Global File System Structured Data Access RDMA interconnect LD JCTD – Instant Data Access • Global ingest, processing, distribution of data • Data is “known” from the moment of ingest • Direct file access (file search not required) • Scalable cost effective storage - Inherent COOP • Globally distributed clustered file system • Local caching (for performance) • Direct file access (file transfer not required) • Fast data access times (seconds, not hours) • Efficient WAN usage, fast data transfers • Scalable, cost effective, low power data centers Instantaneous data access model – “as though the data were on your desktop”

  21. LD Technical Readiness Level Demonstrated TRL-7, with major elements of TRL-8 and TRL-9 Successful LD operations in mission environment

  22. DoD and IC programs of record are adopting LD benefits and capabilities in FY10/11 for: Rapid, global data access and federated exploitation for very large files such as imagery and wide area persistent surveillance Operationally responsive data dissemination/transfer Data federation & synchronization for planning Support to global intelligence operations Enhancing net-centric data delivery to warfighters LD Transition

  23. SC09 Demonstration for Laboratory for Advanced Computing @ UIC / NRL NRL Data Center 1 PB Disk Array LAC/UIC Chicago, IL Data Center LAC/SC09 Portland, OR IP Router (IPoIB-to-IPoE) IPv4oE InfiniBand 4x DDR Open Cloud Testbed Layer 3 (IPv4) IPv4oE • IBoIPV6oE, switched • and routed including • RDMA • Displayed high-res • (2560x1600x70fps video • stream in Portland from • Washington • Rendered images in • Portland, processed in • DC, then retransmitted • back to Portland • Adjusted BW in presence • of congestion and loss to • achieve max rates InfiniBand Switch Fabric Data Center IPv4oE InfiniBand 4x DDR Ethernet Switch Ethernet Switch TeraFlow Layer 2 InfiniBand 4x SDR UDToE InfiniBand 4x SDR UDToE Workstation 10-GigE FPGAs FPGAs IBoIPv6oE 10-GigE IBoIPv6oE WAN Gateway InfiniBand 4x SDR WAN Gateway InfiniBand 4x SDR WAN Gateway WAN Gateway NLR Layer 3 (IPv6) 10-GigE 10-GigE IBoIPv6oE IBoIPv6oE UDToIPv6oE UDToIPv6oE Workstation LD HPC Testbed NRL Washington, DC 10-GigE Monitor NRL/SC09 Portland, OR 18 November 2009 Super Computing 2009 Bandwidth Challenge Winner (“Overall” and “Manifold-Process Implementation” Categories)

  24. Dr. Hank Dardy, 1943-2010

  25. Questions?

  26. LD JCTD Concept of Operations Moving Data to Users Moving Data to Users

  27. Large Data Reference Model Client Domain(RDMA/IB) Client Domain (IP/Eth) Workstations Web Servers Workstations Application Servers DISN InfiniBand Switch InfiniBand Switch InfiniBand LAN GW Lustre File System Remote Systems Encryptor WAN GW Remote Systems Encryptor WAN GW Storage Controller Open Standards Open Systems Open Source Commodity Parts Disk Drives WAN Domain Core Domain Scalable to 1000’s of Nodes — Petabyte File System today…Exabytes tomorrow

  28. Warfighter Problem 1018 Exabytes Warfighter Problem Future Sensor Z Future Sensor Y 1015 Sensor Data Volume Petabytes Future Sensor X Notional Gap Large Data JCTD GLOBAL HAWK DATA PREDATOR UAV VIDEO UUVs FIRESCOUT VTUAV DATA 1012 Terabytes GIG Data Capacity (Services, Transport & Storage) 2000 Today 2010 2015 & Beyond

  29. Summary • LD underpins net-centric warfighting by providing a data-centric DoD information enterprise • LD seeded in key programs • Next generation performance (scalable to exabytes) in smaller footprint at lower cost • Working with Transition Partners to ensure integrated enterprise implementation

  30. still too slow… LD JCTD – What Does It Mean? Standard DVD = 4.7 GB 1 Hr of WAOS* raw data = 1 TB • 5 Mbps (Home Connection) • DVD • WAOS • OC-3 (155 Mbps – decent business connection) • DVD • WAOS • 1 Gbps (Standard GIG/DISN LAN) • DVD • WAOS • 10 Gbps (OC-192 – 8 Gbps effective) • DVD • WAOS 2.1 Hrs 18.5 Days 4 Min 14 Hrs 37.6 Seconds 2.2 Hrs 4.7 Seconds 16.7 Min… • Shared File Systems and Distributed SAN structure – no “FTP” latency • DVD • WAOS Nearly instant ≤ 5 seconds effective remote access time *Wide Area Optical Surveillance

  31. Large Data JCTDGlobal Access, Global Visualization OE Fritz Schultz 703.697.3443Fritz.Schultz@osd.mil OM Randy Heth 402.232.2122hethr@stratcom.mil TM Jim Hofmann 202.404.3132jhofmann@cmf.nrl.navy.mil XM Mike O’Brien 703.735.2721Michael.A.Obrien@nga.mil XM Mike Laurine 703.882.1358 Michael.Laurine@disa.mil

More Related