1 / 17

Limitless Storage, Boundless Opportunities

Limitless Storage, Boundless Opportunities. Technology Overview – January 2009. Cleversafe Mission. Enable the world to confidently store and distribute limitless data. Commercial Products Providing Products and Service to companies who are building and operating Dispersed Storage Networks

nancyd
Download Presentation

Limitless Storage, Boundless Opportunities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Limitless Storage, Boundless Opportunities Technology Overview – January 2009

  2. Cleversafe Mission • Enable the world to confidently store and distribute limitless data • Commercial Products • Providing Products and Service to companies who are building and operating Dispersed Storage Networks • Dispersed Storage - Open Source Project • Creating a genuine open standard and a broad technical community 2

  3. Data Storage Growth Traditional Data Additional, New Data + Documents Character & numerical databases Images – 500KB per picture Audio – 5,000 KB per song Video – 5,000,000 KB per movie Digital Content - 85% of all data by 2012 - Growing 10x every 4 years Source: IDC

  4. Current High Availability Scenario Server1 @ Location 1 The quick brown fox jumps over the lazy brown dog. 11010010 00110010 A2 A1 A3 Parity Internet Connection RAID3 Controller Server3 @ Location 3 Internet Connection The quick brown fox jumps over the lazy brown dog. 11010010 00110010 A2 A1 A3 Parity Server2 @ Location 2 The quick brown fox jumps over the lazy brown dog. 11010010 00110010 RAID3 Controller A2 A1 A3 Parity Internet Connection RAID3 Controller • Higher Cost • More Power • More Management • More Space • More Equipment • More Security Risks • 300% Disk Storage Overhead + Tape Backup • Total bytes stored =4x usable capacity • 200% Bandwidth Overhead • Each node supports full operational requirement • Total bandwidth required =3x operational requirement

  5. Digital Data Storage - An Antiquated Approach • Currently Data Storage =Data Copies • Not Secure • 200 major announced security breaches since 2004 • Not Private • Data copies are… data copies • Not Long Term • Tied to hardware which doesn’t last over 5 years • More Reliable = More Cost • Additional copies, synchronization traffic, high cost hardware • Not Scalable • Performance and management degrades as scale increases

  6. Information Dispersal • With the emergence of Broadband and modern microprocessors, Information Dispersal Algorithms (IDA’s) can be used to store the world’s data… • Inherently secure • Inherently private • Inherently reliable • Inherently long term • Similar mathematical methods are the basis of digital mobile telephony and the Internet • Packet Switching, Reed-Solomon, Erasure Coding, Forward Error Correction, etc.

  7. How Information Dispersal Works 36 example characters = 36 total Bytes Information Dispersal Algorithms - Quick Mathematical Transformation This Slicing example has a 60% Storage Overhead - Total bytes stored = 1.6X usable capacity 16 example slices = 58 total Bytes “Slices” are to data storage …what “packets” are to data communications. - Provide inherently reliable, private, secure and long-term storage

  8. 5 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Copy Copy Copy Copy Parity Parity Copy Copy Copy Copy Copy Parity Parity Parity Copy Dispersal versus Replication Dispersal Nines of Reliability Storage Overhead Bandwidth Overhead Access Choices Slice Storage Width Threshold Source data size Storage Overhead size 8 8 16 16 32 64 5 6 10 12 24 56 6 7 >16 11 >16 >16 60% 33% 60% 33% 33% 14% 60% 33% 60% 33% 33% 14% 70 28 8008 1820 11 million 214million 1 2 3 4 6 7 Typical Configurations Replication Copies + Parity Storage Nines of Reliability Storage Overhead Bandwidth Overhead Access Choices Copies Parity Source data size Storage Overhead size 2 2 3 3 No Yes No Yes 5 10 7 >16 100% 167% 200% 300% 100% 100% 200% 200% 2 2 3 3

  9. Data Storage with Information Dispersal Object dsNet Client iSCSI File Accesser

  10. Data Retrieval with Information Dispersal Object dsNet Client iSCSI File Accesser X X X X

  11. Product – Appliance Components • 4 TB “Raw” capacity per 1U server • Store, return and delete slices • Unlimited vaults (similar to LUNs) per dsNet • Deployable in a single rack or geographically distributed around the world Slicestor Dispersed Storage server Accesser Dispersed Storage router • Slices-disperses-retrieves data to/from Slicestors • Provides standard storage interfaces • Ideal for digital content loading • Deployable in redundant configurations dsNet Client Dispersed Storage client software • Disperses and retrieves data to/from slice servers • Approximately 3 MB of Java code • Ideal for content distribution

  12. Exabytes of Data Storage Require a Paradigm Shift Scenario 1: Exbibyte (~1,000,000 Terabytes) Usable Storage, 10 nines of Reliability 1 EiB Usable 25% Parity 1 EiB Usable 33% Dispersal Overhead 1 EiB Replicated 25% Parity 1 EiB Replicated 25% Parity Replication: Total Raw Storage ~ 3.75 EiB Dispersal: Total Raw Storage ~ 1.33 EiB Traditional Storage Dispersed Storage Replication: ~300% Storage overhead Additional costs for replication SW Dispersal: ~33% storage overhead built in multi-site availability

  13. dsNet – Standard Interfaces iSCSI: Acts like a hard drive • Works with any OS or file system WebDAV: Acts like a URL • Works with any browser • Java client enables any device to access a dsNet: • Media players, phones, set top box, security cameras, sensors, etc. 13

  14. Complete Storage Architecture Client Layer Network REST API Web Service iSCSI WebDAV dsFTP NFS/CIFS Named Object Java SDK Protocol Layer SCSI Simple Object File System Java SDK Interface Layer Object Block Vault Structure

  15. Project Overview • An Open Source Project with Commercial Backing • Dispersed Storage – an Open Source Project • Hosted at www.cleversafe.org • Includes the complete protocol and algorithms • Incorporates and/or enhances additional open source software • Bouncy Castle – Cryptography • JSAP – Java Simple Argument Parser • Bzip2 – Data Compressor • Apache Commons – Logging, Statistics, basic Internet protocols • JUnit – Testing Framework • Log4j – Logging Utility • MINA -- Network Application Framework • SLF4J – Simple Logging Façade for Java • SVNKit – Java Subversion library • Wrapper – Java Service Wrapper • ws-commons – Webservices Common Utilities • jSCSI – iSCSI Initiator

  16. Services • Training • Certification • Support • Additional Capabilities • Management • Reporting • Products • - Integrated hw/sw Appliances • Customized OS • Additional hardware features • Performance • Interoperability Protocols • - Standards • Open Source software Open Source Complements Commercial Commercial Internet Equipment Providers Cleversafe Dispersed Storage Commercial Capabilities Commercial Capabilities Contribute to standards efforts Contribute to standards efforts Open Source

  17. Thank You

More Related