1 / 18

Storage Systems in HPC

Storage Systems in HPC. John A. Chandy Department of Electrical and Computer Engineering University of Connecticut. Research Summary. Storage Systems Active Storage Parallel File Systems Reliable Data Storage Active Storage Networks. Storage Systems. Parallel Computing

chin
Download Presentation

Storage Systems in HPC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage Systems in HPC John A. Chandy Department of Electrical and Computer Engineering University of Connecticut

  2. Research Summary • Storage Systems • Active Storage • Parallel File Systems • Reliable Data Storage • Active Storage Networks

  3. Storage Systems • Parallel Computing • Building parallel file systems to support HPC • Computation at the storage node • Data organization methods to improve performance • Reliable Data Storage • Customizable and extensible storage for reliability • Backup strategies using personal storage devices • Data security, trust, and reliability in the cloud

  4. Parallel File Systems • Network Attached Storage • Put the storage on the network with a computer (server) acting as the go-between Network

  5. Parallel File Systems • Separate the metadata from the storage Metadata Network

  6. Parallel File Systems • How do you improve metadata performance? • Distribute metadata services on data nodes • Use active storage and object services

  7. Active Storage • Allows us to run applications on storage nodes • Can dramatically reduce data traffic • Eliminate large network latencies • Take advantage of fast RAID arrays and SSDs • Drives bottle-necked by slow networks • Run applications in parallel across multiple nodes • Make use of unused processor time

  8. Programming Model • Based on object storage • RPC based • Executable objects • RPC calls have full access to all object functions – read, write, create, set attribute, etc. • Functions can be synchronous or async • Supports multiple languages (C, Java, Python)

  9. Programming Model • Based on work by Acharya, Riedel - Stream based • Our model is Remote Procedure Call (RPC) based • Use executable objects • Added command to begin execution • Allow full access to all OSD functions • Functions can be run sync or async • Due to iSCSI 30sec timeout • Working to allow queries for async • Allow parallel execution using async • Support multiple languages (c, java, python)

  10. Security • Multiprocess implementation • Limits AS functions from directly accessing objects • Limits access to the object services library • Enforces use of object security mechanisms • chroot sandboxing • C/Java engines run in a chroot directory • Allows limited system libraries – e.g. libc

  11. Security • Multiprocess Implementation • Limits AS functions from directly accessing objects • Limits access to the OSD services library • Forces the use of RPC • Enforces the use of OSD security mechanisms • Chroot Sandboxing • Applied to engines • Limits engines inside a single directory • Allows limiting of libraries • AS versions of libraries possible

  12. Active Storage Code Example

  13. Results: AES Local vs. Active Storage

  14. Results: Scaling with Multiple OSDs

  15. Results: C vs. Java

  16. High Performance Computing • Active storage network • Computing in the network • SIMD-like processing of data in motion • Adaptive computing network elements • Application optimizations for database queries, scientific applications, data mining, sort, etc.

  17. Active Storage Networks Data Sort

  18. BECAT Collaboration • Large Data Problems • Parallel File Systems Implementation

More Related