Distributed File Systems

Distributed File Systems Synchronization – 11.5 Consistency and Replication - 11.6 Fault Tolerance – 11.7

11.5: Synchronization • File System Semantics • File Locking

Synchronization • Is an issue only if files are shared • Sharing in a distributed system is often necessary, and at the same time can affect performance in various ways. • In the following discussion we assume file sharing takes place in the absence of process-implemented synchronization operations such as mutual exclusion.

UNIX File Semantics • In a single-processor (or SMP) system, any file read operation returns the result of the most recent write operation. • Even if two writes occur very close together, the next read returns the result of the last write. • It is as if all reads and writes are time-stamped from the same clock. Operation order is based on strict time ordering.

UNIX Semantics in DFS • Possible to (almost) achieve IF… • There is only one server • There is NO caching at the client • In this case every read and write goes directly to the server, which processes them in sequential order. • Even so, the order of simultaneous writes depends on network transmission times. • Any file read operation returns the result of the most recent write operation as seen by the server.

Caching and UNIX Semantics • Single-server + no client caching leads to poor performance, so most file systems allow users to make local copies of files (or file blocks) that are currently in use. • Now UNIX semantics are problematic: a write executed on a local copy only will not be seen by another client that reads the file from the server, or from other clients that have the file cached.

Write-Through • A partial solution is to require all changes to local copies to be immediately written to the server. Now a new user will see all the changes. • Still inefficient – caching is no longer as useful • Not a total solution: what happens when two users have the same file cached?

Consistency Models • Recall discussion of consistency models in Chapter 7 • Realistically, strict consistency can’t be achieved without synchronization techniques such as transactions or locks • Here we consider what the file system can do in the absence of user-enabled methods.

Session Semantics • Instead of trying to implement UNIX semantics where it is not possible, define a new semantic: • Local changes to a file are not made permanent until the file is closed. In the meantime, if another user opens the file, she gets the original version. • This approach is common in DFS’s.

Simultaneous Caching • What if two users concurrently cache and modify the same file? How do we determine the “new” state of the file? • Most likely option: • The most recently closed file becomes the new “official” version

File foo X = 10 Open file foo Write x = x+6 to foo Close foo Process 1 Process 2 Close foo Open foo Write x=4*x to foo Using session semantics, what is the final value of x in file foo after both processes close the file? Assume the horizontal lines represent real time & close operations are seen at the server in the order shown

Immutable Files • The only operations on a file are, effectively, create,read, and replace. • Once a file is created it can be read but not changed. • A new file (incorporating changes to a current file) can be created and placed in the directory instead of the original version. • If several users try to replace an existing file at the same time, one is chosen: either the last to close, or non-deterministically.

Review: File System Semantics • UNIX semantics • Session semantics • Immutable files • Every file operation is instantly visible to all processes • No changes are visible until the file is closed. • No updates are possible; files can only be replaced

Transaction Semantics • Transactions are a way of grouping several file operations together and ensuring that they are either all executed or none is executed. • We say they are atomic. • The transaction system is responsible for ensuring that all of the operations are carried out in order, without any interference from concurrent transactions.

The Transaction Model • Transaction: a set of operations which must be executed entirely, or not at all. • Processes in a transaction can fail at random • Failure causes: hardware or software problems, network problems, lost messages, etc. • Transactions will either commit or abort: • Commit => successful completion (All) • Abort => partial results are undone (Nothing)

Transaction Model • Transactions are delimited by two special primitives: Begin_transaction // or something similar transaction operations (read, write, open, close, etc.) End_transaction • If the transaction successfully reaches the end statement, it “commits” and all changes become permanent; otherwise it aborts.

ACID Properties of Transactions • Atomic: either all or none of the operations in a transaction are performed • Consistent: the transaction doesn’t affect system invariants; e.g., no money “lost” in a banking system • Isolated (serializable): one transaction can’t affect others until it completes • Durability: changes made by a committed transaction are permanent, even if the process or server fails.

Atomicity • An atomic action is one that appears to be “indivisible and instantaneous” to the rest of the system. For example, machine language instructions. • Transactions support the execution of multiple instructions as if they were a single atomic instruction.

Consistent • A state is consistent if invariants hold • An invariant is a predicate which states a condition that must be true. • Invariants for the airline ticket example: • seatsLeft = seatsTotal – seatsSold • seatsLeft >=0 • In the bank case (simplified) • balancefinal = balanceoriginal – withdrawals + deposits

Isolated • No other transaction will see the intermediate results of a transaction. • Concurrent transactions have the same effect on the database as if they had run serially. Notice the similarity to critical sections, which do run serially. • This characteristic is enforced through special concurrency control measures.

AD Properties • ACID is a commonly used term, but somewhat redundant. • Transactions that execute atomically will be consistent and isolated. • Atomicity and durability capture the essential qualities.

UNIX semantics Session semantics Immutable files Transactions Every file operation is instantly visible to all processes No changes are visible until the file is closed. No updates are possible; files can only be replaced All changes occur and are visible atomically – or not at all Semantics of File Sharing in Distributed Systems

File Locking • UNIX file semantics are not possible in DFS • Session semantics and immutable files do not always support the kind of sharing processes need. • Transactions have a heavy overhead. • Thus some additional form of synchronization is desirable to enable the server to enforce mutual exclusion on writes.

Locking in NFS • Early versions of NFS (through V3) were stateless, and so could not implement locks. • An add-on, NLM (Network Lock Manager) worked in the NFS environment to enforce advisory locking • If one process has locked a byte sequence, any other process requesting a lock on that sequence will be denied.

File Locking in NFSv4 • NFSv4 added a similar locking discipline to the basic protocols. • Lock managers in NSF, as in other file systems, are based on the centralized scheme discussed in Chapter 6 • Client requests lock • Lock manager grants lock (if it is free) • Client releases lock (or it expires after a time) • In NSF, if a client requests a lock which cannot be granted, the client is not blocked – must try again later.

Denied Requests • If a client’s request for a lock is denied, it receives an error message. • Poll the server later for lock availability • Clients can request to be put on a FIFO queue; when a lock is released it is reserved for the first process on the queue; if that process polls within a certain amount of time it gets the lock. • How is this different from the centralized mutual exclusion algorithm in the textbook?

File locking in NFS • Two types of locks: • Reader locks, which can be held simultaneously, • Writer locks, which guarantee exclusive access. • The lock operation is applied to consecutive byte sequences in the file, rather than to the whole file.

Operation Lock Lockt Locku Renew Description Create a lock for a range of bytes Test whether a conflicting lock has been granted Remove a lock from a range of bytes Renew the lease on a lock NFSv4 Lock Related Operations

Leases • Locks are granted for a specific time interval. • What problem does this address? • At the end of that interval the lock is removed unless the client has requested an extension.

Share Reservations in NFS • An open request specifies the kind of access the application requires: READ, WRITE, BOTH • It also specifies the kind of access that should be denied other clients: NONE, READ, WRITE, BOTH • If requirements can’t be met, open fails • Share reservations = implicit locking • Used in NFS for Windows-based systems

Share Reservations - Example • Client tries to open a file for reading and writing, and deny concurrent write access. • If no other client has the file open, the request succeeds. • If another client has opened the file for reading (and hasn’t blocked write access), the request succeeds • If another client has opened the file for writing, the request fails. • If another client has the file open and has denied read or write access, the request fails.

11.5 Summary • UNIX semantics not possible in DFS • Session semantics is a common choice • File locking (usually advisory) can provide additional protection if needed. • In parallel programs, mutual exclusion techniques can be used to protect file operations. • In database systems, transactions are used.

11.6: Consistency and Replication • Client-Side Caching • Server-Side Replication • Replication in P2P Systems

Introduction • Replication (and caching) => multiple copies of something • Two reasons for replication: • Reliability (protection against failure, corruption) • Performance (size of user base, geographical extent of system) • Replication can cause inconsistency: at least one copy is different from the rest.

Caching in a DFS • Caching in any DFS reduces access delays due to disk access times or network latency. • Caches can be located in the main memory of either the server or client and/or in the disk of the client

Caching in a DFS • Client-side caching (memory or disk) offers most performance benefits, but also leads to potential inconsistencies. • However, because in practice file sharing is relatively rare, client-side caching remains a popular way to improve performance in a DFS.

Cache Consistency Measures • Server-initiated consistency: server notifies client if its data becomes stale • e.g., another client closes its copy of the file, which was opened for writing. • Client-initiated consistency: client is responsible for consistency of data • e.g., client side software can periodically check with server to see if file has been modified.

Caching in NFS • NFSv3 did not define a caching protocol. • Individual implementations made decisions • “Stale” data could exist for periods ranging from a few seconds to ½ minute • NFSv4 made some improvements but many details are still implementation dependent. • General structure of NFS cache model follows

Client Side Caching in NFS Figure 11-21. Memory Cache NFS server Client applica-tion Disk cache Network

What Do Clients Cache? • File data blocks • File handles – for future reference • Directories • Two approaches to caching in NFS • Caching with server control • Caching with open delegation

Caching Data with Server Control • The simplest approach to caching allows the server to retain control over the file. • Procedure • Client opens file • Data blocks are transferred to the client (by read ops) • Client can read and write data in the cache. • When the file closes, flush changes back to server • Session semantics & NFS: the last (most recent) process to close a file has its changes become permanent. Changes made by processes that run concurrently but close earlier are lost.

Caching with Server Control • In caching with server control • All clients on a single machine may read and write the same cached data if they have access rights • data remaining in the cache after a file closes doesn’t need to be removed, although changes must be sent to server. • If a new client on the same machine opens a file after it has been closed, the client cache manager usually must validate local cached data with the server • If the data is stale, replace it.

Caching With Open Delegation • Allows a client machine to handle some local open and close operations from other clients on the same machine. • Normally the server decides if a client can open a file • Delegation can improve performance by limiting contact with the server • The client machine gets a copy of the entire file, not just certain blocks.

Open delegation – Examples* • Suppose a client machine has opened a file for writing, and has been delegated rights to control the file locally. • If another local client tries to lock the file, the local machine can decide whether or not to grant the lock • If a remote client tries to lock the file (at the server) the server will deny file access • If a client has opened the file for reading, only, local clients desiring write privileges must still contact the server.

Delegation and Callbacks • Server may need to “undelegate” the file – perhaps when another client needs to obtain access. • This can be done with a callback, which is essentially an RPC from server to client. • Callbacks require the server to maintain state (knowledge) about clients – a reason for NFS to be stateful.

Caching Attributes* • Clients can cache attributes as well as data. • (size of file, number of links, last date modified, etc.) • Cached attributes are kept consistent by the client, if at all • No guarantee that the same file cached at two sites will have the same attributes at both sites • Attribute modifications should be written through to the server (write through cache coherence policy), although there’s no requirement to do so

Leases* • Lease: cached data is automatically invalidated after a certain period of time. • Applies to file attributes, file handles (mapping of name to file handle), directories, and sometimes data. • When lease expires, must renew data from server • Helps with consistency and protects against errors.

An Implementation of Leases* • Data blocks have time-stamps applied by the server that indicate when they were last modified. • When a block is cached at a client, the server’s time-stamp is also cached. • After a period of time, the client confirms the validity of the data • Compare timestamp at the client to timestamp at server • If server timestamp is more recent, invalidate client data

CodaA Prototype Distributed File System • Developed at CMU – M. Satarayanan • Started in 1987 as an improvement on the Andrew file system ( a classic research FS) • Andrew strongly influenced NFSv4 and some versions of Linux • Most recent version of Coda (6.9.4) was released 1/05/2009 (http://www.coda.cs.cmu.edu/news.html )

Objectives of Coda • Support disconnected operation (server goes down, laptop is disconnected from network, etc.) • Client side caching is extensive • Uses client disk cache • Replication contributes to availability, fault tolerance, scalability

Distributed File Systems