1 / 47

A Storage Engine for Amazon S3

A Storage Engine for Amazon S3. How to store a hundred billion BLOBs without buying a single disk. Who is this guy?. http://fallenpegasus.com/code. Storage Engine. Protocol Translator. Traditional Storage Engines use the local disk. Network Storage Engines. Federated ODBC HTTP MemCacheD

nico
Download Presentation

A Storage Engine for Amazon S3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.

  2. Who is this guy?

  3. http://fallenpegasus.com/code

  4. Storage Engine

  5. Protocol Translator

  6. TraditionalStorage Enginesuse the local disk

  7. Network Storage Engines

  8. Federated ODBC HTTP MemCacheD and...

  9. S3

  10. What is S3?

  11. Petabytes?!

  12. This can't be free

  13. Why do I want to use it?

  14. “An empty disk costs the same as a full one.”

  15. Can I move myexisting database over?

  16. Translating S3 to MySQL AWS Account → CREATE SERVER S3 Bucket → Table S3 Item → Row S3 Item Key → VARCHAR PRIMARY KEY S3 Item Contents → BLOB or VARCHAR

  17. CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');

  18. CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';

  19. SELECT defn FROM bierce WHERE word='WIT';

  20. SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.

  21. INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');

  22. DELETE FROM bierce WHERE word='AUTHOR';

  23. So, what's it good for?

  24. “If you build it,they will come.”

  25. Saving EC2 work SQL CMS for S3 virtual hosting Huge list of persistent primary keys Big slow BLOBs, to join against fast local tables. “The Image Server Problem” ... and more

  26. Transactions?

  27. What do you mean,“No temporal guarantees”?!

  28. No WHERE clausemeans“MAKE MONEY FAST”for Amazon

  29. The Future(watch me wave my hands)

  30. Code Improvement S3 & HTTP Metadata Multiple Data Columns Information Schema Security & Authentication Transfer & Storage Compression Streaming BLOB More Storage Engines

  31. Code Improvement

  32. S3 & HTTPMetadata

  33. Multiple Data Columns

  34. Sharing a solution withHTTP and MemCacheDEngines

  35. Information Schema

  36. Security &Authentication

  37. Transfer & StorageCompression

  38. (new slide)Streaming BLOBslike PBXT

  39. More storage engines

  40. More storage engines “A Storage Engine for AWS SQS” There are more AWS services rolling out Replication distribution via S3 Cluster persistent storage via S3 ... and the EC2 holy grail:

  41. “A Generic SchemaStorage Engine for S3”

  42. Back to the present...

  43. “Where do I get it?”

  44. “Where do I get it?” MySQL 5.1 GPL http://fallenpegasus.com/code

  45. “You never write,you never call...”

  46. Suggestions Use Cases Bug Reports Patches Feature Requests Support Requests Gigs & Proposals mark@fallenpegasus.com

  47. Questions?

More Related