1 / 18

Hashing

Hashing. Then and now Mike Smorul – ADAPT Project. Commodity Storage Performance. 2003 JetStor III IDE-FC 62MB/s large block 2013 218MB/s workstation SSD Perc 6/MD1000, 400MB/s+. Chip Speed. 2003: Pentium 4 3.2Ghz 2013: Core i7 Extreme 3.5Ghz. Hashing Performance.

annis
Download Presentation

Hashing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hashing Then and now Mike Smorul – ADAPT Project

  2. Commodity Storage Performance • 2003 JetStor III IDE-FC • 62MB/s large block • 2013 • 218MB/s workstation SSD • Perc 6/MD1000, 400MB/s+

  3. Chip Speed • 2003: Pentium 4 • 3.2Ghz • 2013: Core i7 Extreme • 3.5Ghz

  4. Hashing Performance • SHA-256 Hashing • Java: 85MB/s • Crypto++: 111-134MB/s • Real World Penalty • Java: 20-40% penalty on slow seek disk

  5. Implications • Flipped bottlenecks

  6. How to overcome • Faster/weaker digests • Simultaneous transfers • Data locality, tape? • Improve single stream performance

  7. Parallelize Single Stream • Independent IO and digest threads • Always have work for the digest algorithm. • Large files saw over 95% of algorithm potential. • Small files unchanged.

  8. Securing Data in Motion ?

  9. Where to apply fixity • Internal integrity services • At Transfer via manifests • End to End?

  10. Operational Integrity • Internal Auditing • Prove your hardware • Error, not malice detection • Peer-Auditing • Prove your friends

  11. Transporting Integrity • Manifest Lists • Transfer validation • Digital Signatures • Prove identity • Token Based • Prove time

  12. Chronopolis Integrity • Current: • Producer supplied authoritative manifest • Peers locally monitor integrity • Manually trace back to point of ingest

  13. Chronopolis Integrity • In-progress • Single integrity token back to ingest • Ideal • Tokens issued prior to arrival • ‘Prove’ the state of data to point before Chronopolis

  14. Manifests 2.0 • Beyond simple transfer list • Token manifests • Portable, embeddable • Python, etc

  15. Cloud Integrity • Digests in a cloud validate transfer only • Http headers can pass extended integrity information • End-user verification

  16. Integrity as provenance • Integrity checking forward in time • Consumer level verification of data • Integrity from object creation • Start integrity checking before archiving

  17. Closing • Why are you hashing? • What do you want to prove? • Hashing Cost/performance

  18. Contact Mike Smorul msmorul@sesync.org http://adapt.umiacs.umd.edu/ace

More Related