The Google File System

1. The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 Presented by WenhaoXu University of British Columbia

2. Outline • Background • Assumption and Google Workloads • Architecture • System Interaction • Master • Relaxed Consistency Model • Experiment • Conclusion • Discussion

3. Background • In 2003, • 50+ GFS clusters • Each with thousands of storage nodes • Managing petabytes of data • GFS is under lots of applications and other systems in Google, such as bigtable, etc. • Built on top of commodity machine and disks Tailored for Google workloads

4. Assumption & Google Workloads High availability: Replicas, Heartbeat, Fast recovery, checksums • Failures occur everywhere • Disks, Network, Machines • Store a modest number of large files • A few million files; Each typically 100MB or larger in size • Read: Large streaming reads and small random reads • Typically read hundreds of KBs of an individual operation, more commonly 1MB or more • Small reads typically read a few KBs at arbitrary offsets, which are usually batched and sorted to avoid seeking back and forth • Mutation: Mostly appending new data • Concurrent clients • Design flexibility: Co-designing the application and the file system API Large data chunk & No Client data cache Throughput is more important Make appending fast Relaxed consistency model & Atomicity Record Append Customizable API & Don’t need to conform to POSIX semantic!

5. Architecture • Single Master • Single point of Failures?  Shadow masters, operation log • Performance bottlenecks?  Less involvment • A Chunk • Size: 64MB • 64 bit handler • Stored as a Linux file, lazy space allocation • 3+ replicas • Simplicity: concurrency control, metadata management, etc. • Global knowledge: make sophisticated placement and replication decisions.

6. System interaction – Mutation Data flow • Leases – Primary replica • Master is only involved in 1 & 2 • Separated data flow and control flow • Employs network topology to transfer data • Easily to control the operation order

7. System Interaction - Heartbeat • What is a heartbeat message like? • “hey, I am still healthy. And I have chunks a, b, c …. Their versions are v1, v3, v9, …” • What is heartbeat for? • Chunkserver status • Chunk placement • Garbage collection • Stale Replica deletion • Lease extension

8. Master - Metadata Management • Three major types • File and chunk namespaces • (File, chunk) mapping • Location of each chunk’s replicas • All metadata is kept in the memory • Possibility: (<64B/chunk, 226 (64M) * 64 = 4G, a few millions of files) • Fast access & scanning • Operation log • Store First two types of metadata • Any changes to the first two types of metadata are first written to the operation log • “It is replicated on multiple remote machines and respond to a client operation only after flushing the corresponding log record to disk both locally and remotely” • Checkpoints are made to avoid larger log and long starting up time for masters.

9. Master – Namespace Management & locking • Namespace is organized into a lookup table • (file name, metadata) mapping • Does not support aliases for the same file or directory • Mutation lock • Read lock of all parents directory • Write lock of mutation directory • E.g: /d1/d2/…/dn/leaf creation • Read lock: /d1, /d1/d2, …, /d1/d2/…/dn • Write lock: /d1/d2/…/dn/leaf

10. Master - Replica Creation and Placement • Consideration • Balance disk space utilization • Performance: Limit the number of “recent” creations (hot pot) on each chunkserver • Availability: different racks; as far as possible • Maximize network utilization: as less switches as possible; exploit the aggregate bandwidth of multiple racks; • Multiple racks: • Tradeoff: read bandwidth & write traffic • Re-replication & rebalancing

11. Relaxed Consistency model • Two state definitions: • Consistent: All the clients will always see the same data, regardless of which replicas they read from • Defined: Consistent & Clients will see what the mutation writes in its entirety • The state after mutations depends on • whether the ops succeeds or fails • Whether there are concurrent mutations • Move work to applications/clients • Error code  check consistency • Application checksum  check if it is defined • Identifier for record  figure out duplicates • Retry • File namespace mutation are atomic • Handled exclusively by the master

12. Atomic Record Appends • Used heavily by Google applications • Clients specify data • Not specify offset in files • GFS appends it to the files • At least once atomically • Successfully written regions are defined • Does not guarantee that all replicas are identical • Intervening regions are inconsistent

13. Experiments

14. Conclusion • Gfs is designed for Google applications • It is simple, but good enough for Google applications • It is important for Google to store data for other applications.

15. Discussion (1/5) • It is designed specifically to serve search needs. For what other applications this filesystemis suitable for. And for what applications, it is not optimal? • Reply • Depends on: • Major file size (small or large) • Major file reads (streaming or not) • Major writing modes (appending or not)

16. Discussion (2/5) • When the Master decides on which replicas should a particular chunk goes in, shouldn't it consider the potential applications that is going to access that chunk ? • Reply: Not really. More replicas if it is hot spot. • We don't want to see unbalanced disk usages. In the paper it mentioned that it "rebalances" for load balancing but a way of doing it is not mentioned. • Reply : Master knows the disk usages of all chunkservers • In case of a shadow master is operating, it does not provide any mutation operations. But the processes that got lease from the primary before primary got crashed will continue on writing to the disks right ? So how can we keep track of those writes that is taken place while the shadow master is in operation ? • Reply : It can run until finished.

17. Discussion (3/5) • What happens in GFS if the primary master is down? Will one of the “shadow” masters become a new primary master? If so, how to pick one from several “shadow” ones? • Reply : Maybe. • Will the centralized master-server architecture degrade the GFS's scalability? • Reply : Seems not under Google’s workload. • GFS uses the "closest" client for replication. However, the term "closest" is never explained - is it based on network latency, or does it factor in network load on the closest chunkserver, does it factor in the system latency? How does one chunkserver figure out the closest server on the fly? • Reply : Deduced from IP addresss

18. Discussion (4/5) • The concept of 'records' appears frequently in the paper. GFS can guarantee that a record is appended atomically, and can associate an application defined checksum with each record. This seems to imply that GFS keeps metadata for each record. How is this done? • Reply : The checksum is in the record data, not in the metadata. • Clients cache chunk locations, so if one chunk server cannot contact the master it may unwittingly serve stale data to clients. Comments? • Reply : The master includes the chunk version number when it informs clients.

19. Discussion (5/5) • What do you think about the performance gain of “Concurrent appends to the same file operation” and the “relaxed” consistency model under normal workload? (I believe this is only suits well to google). Since appending is key to GFS write performance in a multi-writer environment, it might be that GFS would give up much of its performance advantage even in large serial writes. What do you think about this? • Reply: • It is a tradeoff between complexity between GFS and its clients. • I think it is good for performance. • It is tailored for google’s workloads (appending).

20. Thanks for your attention!