Fundamentals Stream Session 5: Distributed File Systems

1. Distributed Systems Fundamentals Stream Session 5:Distributed File Systems CSC 253 Gordon Blair, François Taïani

2. Overview of the Session • Introduction to distributed file systems • What is a file system? • From file systems to distributed file systems • A debate: The motion is that “AFS is technically superior to NFS” • The case for AFS • The case for NFS • Other significant distributed file systems • From AFS to Coda • Samba • Recent advances in distributed file systems Associated Reading: Tanenbaum and van Steen, chapter 10 (see also CDK, chapter 8 and ACM Computing Surveys 22(4)) G. Blair/ F. Taiani

3. What is a File System? • A fundamental service in any systems environment • Supporting sharing and concurrent access to persistent data • Also support for meta-data • Hierarchical file systems • Files organised into directories • Directories can contain files or other directories (creating a tree structure) • Possibility to create/ delete new files or directories • Naming either relative to current directory or in absolute terms by way of a pathname • Also potential to mount self-contained hierarchies in positions in a more global hierachy G. Blair/ F. Taiani

4. Figure 8.4 UNIX file system operations filedes = open(name, mode) Opens an existing file with the given name. filedes = creat(name, mode) Creates a new file with the given name. Both operations deliver a file descriptor referencing the open file. The mode is read, write or both. status = close(filedes) Closes the open file filedes. count = read(filedes, buffer, n) Transfers n bytes from the file referenced by filedes to buffer. Transfers n bytes to the file referenced by filedes from buffer. count = write(filedes, buffer, n) Both operations deliver the number of bytes actually transferred and advance the read-write pointer. pos = lseek(filedes, offset, Moves the read-write pointer to offset (relative or absolute, whence) depending on whence). status = unlink(name) Removes the file name from the directory structure. If the file has no other names, it is deleted. status = link(name1, name2) Adds a new name (name2) for a file (name1). status = stat(name, buffer) Gets the file attributes for file name into buffer. Typical File System Operations G. Blair/ F. Taiani

5. From File Systems to Distributed File Systems • Typical file system functionality • But how should this be distributed • C.f. multi-tiered architectures See lecture on design of distributed systems G. Blair/ F. Taiani

6. Tranparencies Access: Same operations Location: Same name space after relocation of files or processes Mobility: Automatic relocation of files is possible Performance: Satisfactory performance across a specified range of system loads Scaling: Service can be expanded to meet additional loads Concurrency properties Isolation File-level or record-level locking Other forms of concurrency control to minimise contention • Replication properties • File service maintains multiple identical copies of files • Load-sharing between servers makes service more scalable • Local access has better response (lower latency) • Fault tolerance • Full replication is difficult to implement. • Caching (of all or part of a file) gives most of the benefits (except fault tolerance) Heterogeneity properties Service can be accessed by clients running on (almost) any OS or hardware platform. Design must be compatible with the file systems of different OSes Service interfaces must be open - precise specifications of APIs are published. • Fault tolerance • Service must continue to operate even when clients make errors or crash. • at-most-once semantics • at-least-once semantics • requires idempotent operations • Service must resume after a server machine crashes. • If the service is replicated, it can continue to operate even during a server crash. Consistency Unix offers one-copy update semantics for operations on local files - caching is completely transparent. Difficult to achieve the same for distributed file systems while maintaining good performance and scalability. • Security • Must maintain access control and privacy as for local files. • based on identity of user making request • identities of remote users must be authenticated • privacy requires secure communication • Service interfaces are open to all processes not excluded by a firewall. • vulnerable to impersonation and other attacks Efficiency Goal for distributed file systems is usually performance comparable to local file system. Distributed File Systems: Goals • Transparency • Concurrency • Replication • Heterogeneity • Fault tolerance • Consistency • Security • Efficiency.. * G. Blair/ F. Taiani

7. Client computer Server computer Directory service Application Application program program Flat file service Client module Distributed File Systems: Model Architecture G. Blair/ F. Taiani

8. Debate • The motion • This house believes that “AFS is technically far superior to NFS” • In proposition • Gordon Blair • In opposition • Francois Taiani G. Blair/ F. Taiani

9. The Case for AFS

10. Background on AFS • AFS (the Andrew File System) is a distributed, client-server, file system used to provide file-sharing services • Some properties of AFS are that it: • Provides transparent access to files. Files in AFS may be located on different servers, but are accessed the same way as files on your local disk regardless of which server they are on; • Provides a uniform namespace. A file's pathname is exactly the same from any Unix host that you access it from; • Provides secure, fine-grained access control for files. You can control exactly which users have access to your files and the rights that each one has. • Resources • http://www.openafs.org/ • http://www.angelfire.com/hi/plutonic/afs-faq.html G. Blair/ F. Taiani

11. AFS Implementation G. Blair/ F. Taiani

12. AFS Implementation (continued) G. Blair/ F. Taiani

13. AFS: Neat Idea #1 (Whole File Caching) • What is whole file caching? • When a file (or directory) is first accessed from the server (Vice) it is cached as a whole file on Venus • Subsequent read and write operations are performed on the cache • The server is updated when a file is closed • Cached copies are retained for further opens • Supported by callback mechanism to invalidate cache on concurrent updates • This is therefore a stateful approach • Why is this a good idea? • Scalability, scalability and scalability! • By off-loading work from servers to clients, servers can deal with much larger numbers of clients (e.g. 5,000) • Ask Francois how NFS scales! G. Blair/ F. Taiani

14. Empirical Evidence in Support of Whole File Caching • Measured update patterns • Reads are much more common than writes (factor of x6) • Most files are read and written by only one user (and if shared it is often only one user who modifies it) • The end result is that local cached copies for most files will remain valid for a very long time • Measured access patterns • Files are referenced in bursts (if a file has been referenced recently, there is a high probability of it being referenced soon) • More generally, the principle of locality as observed in memory management also applies to file stores • This implies that caches of a moderate size can be highly effective G. Blair/ F. Taiani

15. AFS: Neat Idea #2 (A Common View of the Global Namespace) G. Blair/ F. Taiani

16. Other Key Features • Clean update semantics • Close approximation to “one copy semantics” as supported by, for example, Unix • Pragmatic relaxation to enhance scalability • Compare this to the rather messy, time dependent semantics in NFS • Supported by lightweight RPC mechanism • Implemented directly on top of UDP • Strong security • Authentication and secure transmission provided in the underlying RPC package • Naturally extensible to support mobile (disconnected) operation • See later! G. Blair/ F. Taiani

17. My Case in a Nutshell • Whole file caching is central to the success of AFS • Support for scalability • Clean update semantics in spite of distribution • This strategy is supported by empirical evidence • Actual patterns of update and usage point towards such a strategy • Having a single view of the global namespace is also a big win • And they anticipated the future • Natural support for the mobile user G. Blair/ F. Taiani

18. The Case for NFS

19. NFS • Originally developed by SUN for its UNIX implementation • Since then available for many other systems • Started in 1982, latest version NFSv4 in 2003 • It’s an Internet standard • Controlled by the Internet Engineering Task Force (IETF) • Series of RFC (Request for Comment) documents: RFC 3530, etc. • Extremely important for interoperability • NFS specifies a collection of protocols • Similar to CORBA • Many implementation to choose from • Very good in heterogeneous environments • But essentially found on Unix systems (Linux, Solaris, BSD, etc.) G. Blair/ F. Taiani

20. NFS Architecture • Traditional Client Server: uses a remote access model • Based on Open Network Computing RPC system (ONC RPC) • Originally developped by SUN (SunRPC), now an Internet standard too • Originally stateless (until version 3) • File stays on server, file operations provided as RPC file operation Server Client File interface • Stateless approaches has been abandoned in version NSFv4 • More on this later G. Blair/ F. Taiani

21. NFS Operations (V3 & V4) G. Blair/ F. Taiani

22. Stateless or Stateful? • NFS up to version 3 allowed servers to be stateless • Server did not have to maintain information about clients • All operations self-contained and independent • Advantages • Simplicity: No need for reconciliation in cases of crashes, no additional bookkeeping • Disadvantages • Some functionalities hard to implement:i.e. file locking, authentication • file locking implemented outside of NFSv3 by a lock manager • Abandoned in NFSv4 (strongly inspired by AFS) • Necessary to support caching efficiently (delegations) • But client information on server kept to a minimal G. Blair/ F. Taiani

23. File Locking • Admittedly one of the long standing weakness of NFSv3 • Implemented by a separate file locking protocol • (Stateful) lock manager required on top of NFS • Complex and buggy implementations • Everything is far better with NFSv4 • Locking protocol integrated with the rest of the NFS norm • Fine granularity: down to a range of bytes in a file • Read and Read/Write locks • Fairness: client can choose to be put in a FIFO waiting list G. Blair/ F. Taiani

24. Neat Idea #1: Flexible Naming • Client must use a ‘mount’ operation • They can ‘mount’ exported directories anywhere in their local file system (very flexible) • Servers don’t need to export a whole file system (i.e. partition) G. Blair/ F. Taiani

25. Not Atomic! ! Neat Idea #2: Compound Operations • Available since NFSv4 • Several operations can be grouped in one message exchange • Increases performance in WAN (high latency per message) • Reading data from a file in NFS version 3. • Reading data using a compound procedure in version 4. G. Blair/ F. Taiani

26. Consistency in NFS • Client-side caching is used for performance reasons • Early versions had a session semantics (write on close) • Abandoned for performance reasons • Replaced by a time-out based mechanisms • Admittedly no clear semantics • Rather best effort • However • Trade-off between performance and guarantees • NFSv4: improved by delegation mechanism (heuristics based) • “ Most files are read and written by only one user (and if shared it is often only one user who modifies it) ” • Locking is available for stronger consistency guarantees G. Blair/ F. Taiani

27. My Case in a Nutshell • An Industrial Standard (same standardisation body as TCP/IP) • OS and network independent • Very good for heterogeneous environments • Many implementation, ubiquitous on Unix systems • With NSFv4 best of ASF included in NSF • Caching • Strong security (using Kerberos) • Improved sharing guarantees • Very flexible • Client can choose not to use the same naming space • A machine can be client & server at the same time I use it on my laptop when running Mac OS and Linux at the same time to share files between both OSs. Real life system with real life successes! G. Blair/ F. Taiani

28. References • NFS on Wikipedia • http://en.wikipedia.org/wiki/Network_File_System • Interesting account on how AFS and its derivatives (championed by IBM among others) lost to NFS • Tanenbaum & van Steen, Distributed Systems, Chapter 10 • Section 10.1 “Sun Network File System” (NFS’ old name) • How the NFS Service Works, Sun on-line documentation • http://docs.sun.com/app/docs/doc/816-4555/6maoquib3?a=view G. Blair/ F. Taiani

29. Other Significant Distributed File Systems • Coda • An extension to AFS supporting disconnected operation • SMB/CIFS • Microsoft’s file sharing protocol. Works best on LAN. • Plan 9 • Resources unified as files • XFS • A serverless file system • SFS • Scalable security G. Blair/ F. Taiani

30. An Introduction to Coda • What is Coda? • A distributed file system, based on AFS • Designed esp. for disconnected operation • Replication strategy • File system consists of n volume storage groups • Whole files also cached (and hoarded) locally • Optimistic replication strategy in that files may be modified (in cache) when disconnected • Conflicts detected for manual repair G. Blair/ F. Taiani

31. State Transition Diagram for Coda G. Blair/ F. Taiani

32. Introducing SMB • SMB is Microsoft’s protocol to share files and printers • Also renamed CIFS (Common Internet File System) • Client/Server, no location transparency • Not the same as Samba: an open source implementation of SMB primarily found on UNIX systems (Linux) • SMB usually runs on NetBIOS (naming + sessions + datagram) • NetBIOS + SMB developed for LAN use • A number of other services run on top of SMB • In particular MS-RPC, a modified variant of DCE-RPC • Authentication for SMB handled by the NT Domainssuite of protocols, running on top of MS-RPC NT-Domain MS-RPC SMB NetBIOS TCP/IP To know more: Timothy D Evans, NetBIOS, NetBEUI, NBF, NBT, NBIPX, SMB, CIFS Networkinghttp://timothydevans.me.uk/nbf2cifs/nbf2cifs.pdf G. Blair/ F. Taiani

33. Recent Advances in Distributed File Systems • Improvements in storage techniques • Emergence of RAID technology (Redundant Arrays of Inexpensive Disks) • Log-structured file systems • New design approaches • Striping of files across multiple servers • The emergence of peer-to-peer file systems • PAST • BitTorrent • Freenet • Kazaa G. Blair/ F. Taiani

34. Expected Learning Outcomes At the end of this session: • You should understand the concept of a distributed file system and in particular what such a system aims to achieve when compared to local file systems • You should have a strong understanding of key design choices in distributed file systems relating to caching, consistency and naming • You should have a clear appreciation of the architectures of AFS and NFS and also of the relative strengths and weaknesses of each approach • You should be aware of other key developments in distributed file systems including other important systems and also emerging techniques G. Blair/ F. Taiani