The Google File System

1. The Google File System Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak Leung Presenter: Gladon Almeida Year: OCT’2003 Google File System

2. Key Design Considerations Google File System Component failures are the norm rather than the exception. Files are huge by traditional standards. Most files are mutated by appending new data rather than overwriting existing data. Co-designing the applications and the file system API.

3. Assumptions Google File System Inexpensive commodity hardware that often fail. Modest number of large files(typically ~100MB) will be stored. Small files supported but not optimized for. Large streaming read / small random reads Large sequential write that append data to files System must efficiently implement well-defined-semantics for multiple clients that concurrently append to the same file High sustained bandwidth is more important than low latency

4. Architecture Google File System • Single master and multiple chunk servers • Each is a typical commodity Linux machine running a user level server process • Files are divided into fixed sized chunks each of which is identified by a globally unique 64-bit chunk handle • Typically 3 replica’s of each chuck spread over different chunk servers • Master maintains all file system metadata • Namespace • Access control information • Mapping from files to chunks • Current locations of chunk • Master also control system-wide activities like: • Lease management • Garbage collection of orphaned chunks • Chunk migration between chunk servers • Periodic handshake messages with chunk servers • GFS client code uses this file system API to communicate with Master and chunk servers

5. Architecture Figure: GFS Architecture Google File System

6. Design Overview Google File System • Chuck Size (64MB - Large) • Reduces client-master interactions • Reduces network overhead • Reduces size of metadata on the master. • Metadata: • Namespace and file-to-chunk mappings persistent storageas mutation logs on master (as well as remote locations) • Operation Logs: • Historical record of critical metadata changes • Defines the order of concurrent operations • Critical: • Replication on multiple remote machines • Changes are 1stmade persistent on local as well as remote location and then made visible to client. • Fast recovery: (1 minute for few million files) • Replay operation logs • Checkpoints (B-tree like form)

7. Consistency Model Google File System • Consistent region: If all the clients will always see the same data, regardless of which replicas they read from. • Defined region: After a file data mutation if it is consistent and clients will see what the mutation writes in its entirety.

8. Consistency Model – contd. Google File System • After a sequence of successful mutations, the mutated is guaranteed to be defined by: • Applying mutations to all replicas in the same order • Using chunk version number to detect any replica that becomes stale • What if client caches stale chunk location? • Such window limited by the cache entry’s timeout • Most files are append-only – Stale replica returns a premature end of chunk rather than outdated data

9. System Interactions: Leases and Mutation order Google File System Leases: used to maintain a consistent mutation order across replicas. Lease is granted by the master to one of the replicas called the primary The primary picks a serial order for all mutations to the chunk. Initial timeout of lease of 60 sec which can be extended Extension requests piggybacked on heartbeat messages between master and chunk server

10. Control and Data flow for a write Google File System

11. Decoupling of Data and Control Flow Google File System • Control flow: • Master -> Client -> Primary Chunk -> Secondary Chunks • Data flow: • Decoupled from control flow to use the network efficiently • Data pushed linearly in pipeline fashion • Each machine forwards data to the closest machine (Determined by the IP address) • Outbound b/w is fully utilized (No tree structure) • Switched networks and full-duplex links

12. Atomic Record Appends (record append) Google File System • Concurrent serializable appends • Client specifies only data (No offset) • GFS uses append-at-least-once-atomically policy • GFS appends the data and returns the offset to the client • Heavily used: • Multiple-producer / Single-Consumer queues • Concurrent merged results from many different clients • If failure, the client retries the operation • GFS doesn’t guarantee that each chunk is byte-wise identical, it only guarantees that data is written at least once as an atomic unit. • Successful operations: Defined regions • Intervening regions: Undefined regions

13. Snapshots Google File System • Why ? • Makes copy of a file or directory almost instantaneously • Copy-on-write technique • Steps when a snapshot request is received: • Revoke nay outstanding leases • Log the operation to the disk • Duplicate the metadata for source file / directory tree • When write request to these chucks is received • Notices that reference count is greater than 1 • Creates copy of chunk locally • Informs other replicas to do the same • Returns new chunk handle

14. Master Operations - Namespace Management and Locking Google File System • No per-directory data structure • No support for aliases • lookup table mapping full path names to metadata • Each node in namespace tree (file/directory) has associated read/write lock • Why locks ? • Example: to lock /d1/d2/d3/leaf for write • Example: How mechanism prevents a file /home/user/foo from being created while /home/user is being snapshotted to /save/user • Snapshot: • Read locks on /home, /save • Write locks on /home/user, /save/user • Create: • Read lock on /home, /home/user • Write lock on /home/user/foo

15. Policies: Google File System • Chunk replica replacement: • Maximize data reliability and availability • Maximize network bandwidth utilization • Spread replicas across machine as well as racks • New chunk creation: • New replicas created on below-average disk utilization • Limit the number of “recent” creations on each chunk server • spread replicas of a chunk across racks • Re-replication: • soon as the number of available replicas falls below a user-specified goal. Priority considering: • How far from replication goal? • Is chunk blocking client? • Is file live? • Occasionally rebalance replicas

16. Garbage Collection Google File System • Lazy garbage collection • Steps: • Log the deletion immediately • Rename file to hidden name (Deleted in 3 days) • During regular scans remove the such hidden files • Master’s regular scans of chunk namespace • Identify orphaned chunks and erase metadata • During heartbeat message exchange this info with the chunk servers • Chunk servers delete these chunks • Stale replica detection: • Using version numbers

17. Fault Tolerance: Google File System • High Availability: • Fast recovery: Servers designed to restart fast. • Chunk Replication: • Different replication levels for different parts on namespace • Default level = 3 • Master Replication: • Operation log and checkpoint replicated remotely • Master process externally monitored. • On failure new process started using remotely saved checkpoint and logs • Use of canonical names • Shadow Masters • Data Integrity: • chunk broken into 64 KB blocks. Each has 32 bit checksum. • Chunk server verifies before returning – Hence, no error propagation

18. Relating it to CSCI-572 ! Google File System • GFS led to development of Hadoop Distributed File System • HDFS ideal for large workloads which can use the Map-Reduce framework for high degree of parallelism using commodity hardware. • Ideal for search engine workloads like: • Crawling, • Generating inverted index, • PageRank calculation, etc. • Other: • Large-scale machine learning problems • Clustering problems • Large scale graph computation

19. Pros and Cons Google File System • Pros: • Assumptions at the beginning of the paper are later backed up by experiment results. • Very important paper: Led to development of Hadoop Distributed File System (HDFS). • Cons: • Only talks about workloads which are sequential reads and file appends. GFS is not suitable for random read/writes. • The authors don’t provide any performance results for random read/writes

20. Conclusion: Google File System • GFS supports large-scale data processing workloads in commodity hardware • Design decisions specific to Google's needs but many may apply to data processing tasks of a similar magnitude and cost consciousness. • Component failures are the norm rather than exception • Optimization priority: • Concurrent appends • Read • Fault tolerance by monitoring, replication and fast recovery • High aggregate throughput to many concurrent readers