The Google File System

1. The Google File System Sanjay Ghemawat, Howard Gobioff,Shun-Tak Leung

2. Paper highlights • Presents a DFS tailored to a very specific workload • Mostly huge files • Mostly append-only updates • Mostly sequential reads • Client will try to order non-sequential read • Non-standard client API

3. The environment Component failures are frequent • Large system built from commodity parts Files are huge • Multi-GB are frequent Most files are mutated by appending • Large repositories, data streams, archival data GFS API co-designed along with applications

4. Design assumptions (I) Built from many cheap commodity components • Frequent failures Modest number of very large files • 100 MB or more Two kinds or reads • Large sequential reads (100's of KB and more) • Small random reads • Often batched and sorted by applications

5. Design assumptions Many large sequential writes • Append data to files • Random small writes are rare Must implement well-defined semantics for multiple clients that concurrently append data to the same file Sustained bandwidth is more important than latency

6. Interface Familiar POSIX API • create, delete, open, close, read and write • Two additional operations • snapshot • Creates a copy of a file or a directory • record append: • Allows multiple clients to append data to the same file at the same time • Guarantees the atomicity of updates

7. Architecture (I) Single master + multiple chunkservers Files divided into fixed-size chunks Each chunk has a unique immutable 64-bit chunk handle • Assigned by the master at chunk creation Chunks are • Stored as Linux files • Replicated (default is three replicas)

8. Architecture (II) The master • Maintains all metadata: • Namespace, access control, mapping from files to chunks, chunk locations • Controls all system-wide activities: • Garbage collection, chunk migration • Communicates to chunkservers through HearthBeats Having a single master simplify the design

9. Architecture (III) Clients interact with • Master for all system metadata operations • Directly to chunkservers for all data transfers No data caching anywhere in the system • Most applications stream through huge files • Would be ineffective Clients cache metadata

10. Overview

11. A typical interaction Client translates byte offset into chunk index within file • Easy because chunks have fixed-sizes Sends request to master with file name + chunk index Master replies with chunk handle and chunk replica locations Typical requests are for multiple chunks

12. Chunk sizes 64 MB (that's megabytes!) Advantages • Reduces need for clients to interact with master • Many applications exhibit spatial locality Disadvantage: • May cause hot spots if many clients have to access the same one-chunk file • Only happened for some executables • Solution was to have more replicas

13. Metadata Master stores in memory • File and chunk namespaces • Mapping from files to chunks • Locations of each chunk replicas First two are kept persistent by logging mutations on a operation log • Stored on the master's local disk • Replicated on remote hosts Chunk replica locations are collected from chunk servers each time the master reboots

14. In-memory data structures Make master operations fast Let master perform periodic maintenance operations in an efficient manner • Chunk garbage collection • Re-replication after a chunkserver failure • Chunk migration to balance load and disk space Less than 64 bytes per 64MB chunk

15. Chunk locations (I) Non-persistent • Can be stored in main memory Master polls chunkservers • At startup • Periodically after that (HeartBeat messages) Greatly simplifies the system design • No need to keep master and chunkservers synchronized

16. Chunk locations (II) Main principle • Each chunkserver has the final word over what chunks it stores on does not sore on its local disk • Eliminates consistency issues when a chunkserver crashes or another integrates the system

17. Operation log (I) Historical record of critical metadata changes Only persistent record of metadata Provides a logical timeline • File chunks and their versions are identified by the logical time at which they were created Replicated on several machines Changes cannot be visible to clients until metadata changes ae made persistent

18. Operation log (II) When master needs to recover the state of its file system, it will replay the log To minimize recovery times, log must kept short Master checkpoints its state • Each time the log reaches a maximum size Checkpoint is a B-tree • Can be directly loaded in main memory

19. Consistency model Relaxed consistency • A file region is consistent if all clients will always see the same data on all replicas • Product of concurrent successful writes • A file region is said to be defined if • It is consistent • Clients will see the mutation writes in their entirety • Product of serial writes

20. Namespace mutations Atomic • Handled exclusively by the master • Uses locks • Serial order defined by the master's operation log

21. Data mutations Can be • Writes • Record appends Writes write at an application specified write offset Record appends cause data to be appended at most once at an offset of GFS choosing • GFS may insert padding or duplicate appends

22. From the rest of the paper Shadow masters • Additional masters that "follow" the actual master by implementing the actions recorded on the master's operation log • Not perfect mirrors • Offer read-only access to GFS

23. You are not responsible for the rest of the paper