Distributed File Systems

1. Distributed File Systems

2. Purposes of a Distributed File System • Sharing of storage and information across a network • Convenience (and efficiency) of a conventional file system • Persistent storage that most other services (e.g., Web servers) need Information Management NTU

3. Properties of Storage Systems Other properties include availability, timing guarantees, etc. Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

4. Files • Files are an abstraction of permanent storage. • A file is typically defined as a sequence of similar-sized data items along with a set of attributes. • A directory is a file that provides a mapping from text names to internal file identifiers. Information Management NTU

5. File Attributes Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

6. File Systems • Responsible for the (a) organization, (b) storage, (c) retrieval, (d) naming, (e) sharing, and (f) protection of files. • Provide a set of programming operations that characterize the file abstraction, particularly operations to read and write subsequences of data items beginning at any point of a file. Information Management NTU

7. File System Modules A basic distributed file system implements all of the above plus modules for client-server communication and distributed naming and location of files. Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

8. UNIX File Operations Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

9. Distributed File System Requirements • Transparency: access, location, mobility, performance, and scaling transparency. • Concurrency (and Consistency) • Replication/Caching (and Consistency) • Hardware/operating system heterogeneity • Fault-Tolerance • Security (Access Control, Authentication) • Efficiency Information Management NTU

10. A File Service Architecture Note: The modules communicate with one another by remote procedure calls. Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

11. File Service Components • Flat file service: implementing operations on the contents of files, which are referred to by unique file identifiers (UFIDs) • Directory service: mapping text names of files (including directories) to their UFIDs • Client module: integrating and extending the previous two services under a single application programming interface * Why is this structure more open and configurable? Information Management NTU

12. Flat File Service Operations Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

13. Difference from UNIX • Immediate access to files using UFIDs (without open or close) • Read or write starts at the position indicated by a parameter • All operations, except create, are repeatable • Allows a stateless implementation Information Management NTU

14. Access Control • Conventional access rights checks (at open calls) not feasible • Two ‘stateless’ approaches: * Capability (by manipulating the UFID) * User identity sent with every request (adopted in NFS and AFS) • Main problem: forged requests; some authentication mechanism is needed Information Management NTU

15. Capabilities and UFIDs A capability is a binary value that acts as an access key; it can be encoded in the UFID. • Basic construction of a UFID: file group id + file number + random number • Additional field: permissions • Additional field: encryption of the permission field Information Management NTU

16. Directory Service Operations Note: Each directory is stored as an ordinary file with a UFID. Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

17. The Network File System (NFS) • Introduced by Sun Microsystems in 1985, now an Internet standard • Runs on top of RPC (RFC 1831) • Implemented on most operating systems • Version described here: UNIX implementation of NFS Version 3 (RFC 1813, June 1995) • Most recent version: NFS Version 4 (RFC 3010, December 2000) Information Management NTU

18. NFS Architecture Note: Each computer can act as both a client and a server. Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

19. The Virtual File System Module • Access transparency • File handles (file identifiers): • ‘filesystem indentifier’ + ‘i-node number’ + ‘i-node generation number’ • One VFS structure for each mounted filesystem • relates a remote filesystem (identified by its file handle obtained at mount time) to a local directory on which it is mounted • One v-node per open file • indicates whether a file is local or remote, etc. Information Management NTU

20. The NFS Client Module in UNIX • Integrated with the kernel • Emulates the UNIX file system primitives • A single client module serves all user-level processes • The encryption key for authentication stored in the kernel • Caches file blocks Information Management NTU

21. NFS Server Operations Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

22. NFS Server Operations (cont’d) Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

23. Remote File Acceses Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

24. File System Information in UNIX saturn:~ 35 % df -k Filesystem kbytes capacity Mounted on /dev/dsk/c0t3d0s0 143903 91% / /dev/dsk/c0t3d0s6 267943 99% /usr /dev/dsk/c0t3d0s3 15383 3% /tmp galaxy:/usr/local.real 4030440 53% /usr/local lucky:/var/mail.real 564648 86% /var/mail cosmos:/home.real/student/xxx 3941760 60% /home/xxx galaxy:/home.real/faculty/yyy 2964512 51% /home/yyy * Note: The output of ‘df -k’ has been edited. Information Management NTU

25. Caching • Server caching • read-ahead • write-through • delayed-write with the commit operation • Client caching • cache validation (freshness interval and validation timestamp, modification timestamp and getattr, …) • bio-daemon (for read-ahead and delayed-write caching at the client side) Information Management NTU

26. Achievements of NFS • Access and location transparency • Mobility transparency (partially) • Read-only file replication: the automounter • Fault-tolerance: stateless servers, the automounter • Efficiency: caching of disk blocks (main problem: frequent use of getattr) Nonachievements: scalability, concurrency and consistency, security (Kerberos), ... Information Management NTU

27. The Andrew File System (AFS) • Developed at CMU • Current versions: AFS-2, AFS-3 • Compatible with NFS • Main achievement over (older) NFS: better scalability by minimizing client-server communication • Key characteristics: whole-file serving and caching (partial file caching allowed in AFS-3) Information Management NTU

28. Observations onUNIX File Usage • Files are mostly small • Read operations are more common • Sequential accesses are more common • Most files are written by one user • Files are referenced in burst Information Management NTU

29. AFS Architecture Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

30. AFS File Name Space Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

31. System Call Interception in AFS Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

32. AFS System Calls Implementation Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

33. Cache Consistency • A callback promise is provided when Vice supplies a copy of file to a Venus process • The callback promise stored with the cached copy is in either valid or cancelled state • When Venus handles an open, it checks the cache. Information Management NTU

34. The Vice Service Interface Information Management NTU Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.

35. Enhancements to NFS and AFS • Spritely NFS • add open and close, use callbacks • NQNFS (Not Quite NFS) • use callbacks and leases • WebNFS • allow browsers and other applications to interact with an NFS server directly • NFS Version 4 (RFC 3010, December 2000) • incorporating all of the above and more • DCE/DFS (based on AFS) • use callbacks and write tokens (with a lifetime) Information Management NTU

36. New Features of NFS Version 4 • Adoption of the RPCSEC_GSS (RFC 2203) security protocol • Multiple operations in one request • Better migration and replication abilities • A client may query the location(s) of a file system. • Introduction of open and close operations • Lease-based file locking • Callback-based delegation of files Information Management NTU

37. New Design Approaches • Background • high-performance storage technology (e.g., RAID) • log-structure file systems (e.g., Sprite, BSD LFS) • high-performance switched networks (e.g., ATM, high-speed Ethernet) • Goals: high scalability and fault-tolerance • Main ideas: distribute file data among many nodes, separate responsibilities, … • Constraints: high level of trust Information Management NTU

38. More Recent File System Designs • xFS • Serverless: all data, metadata, and control can be located anywhere in the system; any machine can take over the responsibilities of a failed one • Frangipani • Two-layer structure • the Petal distributed virtual disk system • the Frangipani server module Both designs utilize RAID-style striping, log-structured file storage, etc. Information Management NTU

39. Log-based Striping in xFS Information Management NTU Source: T.E. Anderson et al., Serverless Network File Systems, ACM TOCS 1996

40. An xFS Configuration Information Management NTU Source: T.E. Anderson et al., Serverless Network File Systems, ACM TOCS 1996

41. A Frangipani Configuration Information Management NTU Source: C.A. Thekkath et al., Frangipani: A Scalable Distributed File System, ACM SOSP 1997