1 / 10

The Lightweight File System

The Lightweight File System. Presentation at SciDAC face-to-face January 2005 Ron A. Oldfield Sandia National Laboratories. The Lightweight File System Project. An experimental object-based storage system (funded by Sandia’s CS research foundation). Participants:

elroy
Download Presentation

The Lightweight File System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Lightweight File System Presentation at SciDAC face-to-face January 2005 Ron A. Oldfield Sandia National Laboratories 1

  2. The Lightweight File System Project An experimental object-based storage system (funded by Sandia’s CS research foundation) • Participants: • SNL: Ward, Oldfield, Riesen, Lawry • UNM: Maccabe, Arunagiri • Motivation • Risk-mitigation for Red Storm PFS (Lustre) • Vehicle for parallel I/O research • Design Goals • Lightweight design • only critical I/O functionality (direct access to storage, security) • special functionality implemented in I/O libraries (above LWFS) • Scalable • expect clients with tens-to-hundreds of thousands of processes • I/O system should not hinder scalability of the application • Secure • authentication and authorization • fine-grained access controls with revocation 2

  3. Key features of LWFS • Initial focus is on “secure storage architecture” (not a file system) • Policy vs mechanism • Separate policy decisions from policy enforcement • Metadata servers and storage servers cooperate for policy consistency • Access-control • capability-based • immediate revocation • automatic refresh • fine-grained operation control • Containers for related objects • the unit for access control (no control for byte, object, or block access) • Every object is in a container • LWFS knows nothing about the structure of container • We have backwards pointers (not forwards) 3

  4. What LWFS does not do • naming (e.g, metadata) • data synchronization • data consistency • caching • prefetching • quota enforcement • data distribution 4

  5. Basic LWFS Architecture • Security model (Authentication and Authorization) • Separation of policy from mechanism • Scalable • No connection-based mechanisms (e.g., Kerberos) • Capabilities for access control (well... not quite) • not independently verifiable • Similar to the NASD access credential • Coarse-grained access control (containers) • Storage service • object-based • enforcement of access control policies 5

  6. Security Assumptions and Requirements • We assume a trusted transport mechanism • prevents replay attacks, man-in-the-middle attacks, and eavesdropping • provides network integrity • We do not need to provide encryption • Authentication • Use existing mechanisms (e.g., Kerberos, GSS-API) • Scalable (no connection-based schemes) • Authorization • Coarse grained access controls (at the container level) • Capabilities for scalability • “Immediate” revocation of capabilities • Integrity • Make sure users cannot modify capabilities 6

  7. Compute (n) File I/O (m) Service Users /home Net I/O Understanding the Scalability Requirements The I/O system should not hinder scalability of application • Prohibit ops with O(n) communications • Prohibit data structures of size O(n) • Limit ops with O(m) comms to rare events 7

  8. traditional PFS low-level I/O lib Motivating the “Open-Architecture” Model Applications should choose nearly all features of the I/O system • Application-specific APIs • MPI-IO • HDF5, PnetCDF • ChemIO, SalvoIO • Application control of policies • Caching and prefetching • Data distribution • Synchronization application access to datasets, app-specific APIs, caching, prefetching high-level I/O lib acesss to (parallel) files, reliability, data distribution, consistency, synchronization access to bytes (in objects), metadata management, security LWFS 8

  9. Status • APIs defined • Storage service • Authentication and Authorization • Naming service • Transactions (Locks and Journals) • Prototype implementations • Storage service • Authorization service • Naming service (in progress) 9

  10. Current and Future Work • Vehicle for parallel I/O research • Scalable metadata • Application-specific I/O libs and file systems • data distribution, data consistency, caching, prefetching to match access patterns • mobile app/lib code (e.g., active disk) • Lock-free synchronization schemes • Lightweight database (using LWFS) • Framework for developing production-level parallel I/O systems 10

More Related