15-440 Distributed Systems

1. 15-440 Distributed Systems GFS/HDFS

2. Building Blocks • Google WorkQueue (scheduler) • Google File System • Chubby Lock service (paxos-based) • Two other pieces helpful but not required • Sawzall (languate) • MapReduce • BigTable: Build a more application-friendly storage service using these parts

3. Overview • Google File System (GFS) and Hadoop Distributed File System (HDFS) • BigTable

4. Google Disk Farm Early days… …today

5. Google Platform Characteristics • Lots of cheap PCs, each with disk and CPU • High aggregate storage capacity • Spread search processing across many CPUs • How to share data among PCs?

6. Google Platform Characteristics • 100s to 1000s of PCs in cluster • Many modes of failure for each PC: • App bugs, OS bugs • Human error • Disk failure, memory failure, net failure, power supply failure • Connector failure • Monitoring, fault tolerance, auto-recovery essential

8. Data-Center Network 80Gb 160Gb 320Gb

9. Data-Center Network 160Gb 160Gb 320Gb

10. Google File System: Design Criteria & Assumptions • Detect, tolerate, recover from failures automatically • Large files, >= 100 MB in size • Large, streaming reads (>= 1 MB in size) • Read once • Large, sequential writes that append • Write once • Concurrent appends by multiple clients (e.g., producer-consumer queues) • Want atomicity for appends without synchronization overhead among clients

11. GFS: Architecture • One master server (state replicated on backups) • Many chunk servers (100s – 1000s) • Spread across racks; intra-rack b/w greater than inter-rack • Chunk: 64 MB portion of file, identified by 64-bit, globally unique ID • Many clients accessing same and different files stored on same cluster

12. Master Server • Holds all metadata: • Namespace (directory hierarchy) • Access control information (per-file) • Mapping from files to chunks • Current locations of chunks (chunkservers) • Delegates consistency management • Garbage collects orphaned chunks • Migrates chunks between chunkservers Holds all metadata in RAM; very fast operations on file system metadata

13. Chunkserver • Stores 64 MB file chunks on local disk using standard Linux filesystem, each with version number and checksum • Has no understanding of overall file system (just deals with chunks) • Read/write requests specify chunk handle and byte range • Chunks replicated on configurable number of chunkservers (default: 3) • Why not RAID? • No caching of file data (beyond standard Linux buffer cache) • Send periodic heartbeats to Master

14. Master/Chunkservers

15. Client • Issues control (metadata) requests to master server • Issues data requests directly to chunkservers • Caches metadata • Does no caching of data • No consistency difficulties among clients • Streaming reads (read once) and append writes (write once) don’t benefit much from caching at client

16. Client • No file system interface at the operating-system level (e.g., under the VFS layer • User-level API is provided • Does not support all the features of POSIX file system access – but looks familiar (i.e. open, close, read…) • Two special operations are supported. • Snapshot: is an efficient way of creating a copy of the current instance of a file or directory tree. • Append: allows a client to append data to a file as an atomic operation without having to lock a file. Multiple processes can append to the same file concurrently without fear of overwriting one another’s data

17. GFS: Architecture

18. Client Read • Client sends master: • read(file name, chunk index) • Master’s reply: • chunk ID, chunk version number, locations of replicas • Client sends “closest” chunkserver w/replica: • read(chunk ID, byte range) • “Closest” determined by IP address on simple rack-based network topology • Chunkserver replies with data

19. Client Write • 3 replicas for each block  must write to all • When block created, Master decides placements • Default: two within single rack, third on a different rack • Why? • Access time / safety tradeoff

20. Client Write • Some chunkserver is primary for each chunk • Master grants lease to primary (typically for 60 sec.) • Leases renewed using periodic heartbeat messages between master and chunkservers • Client asks master for primary and secondary replicas for each chunk • Client sends data to replicas in daisy chain • Pipelined: each replica forwards as it receives • Takes advantage of full-duplex Ethernet links

21. Client Write (2) Send to closest replica first

22. Client Write (3) • All replicas acknowledge data write to client • Don’t write to file  just get the data • Client sends write request to primary (commit phase) • Primary assigns serial number to write request, providing ordering • Primary forwards write request with same serial number to secondaries • Secondaries all reply to primary after completing write • Primary replies to client

23. Client Record Append • Google uses large files as queues between multiple producers and consumers • Same control flow as for writes, except… • Client pushes data to replicas of last chunk of file • Client sends request to primary • Common case: request fits in current last chunk: • Primary appends data to own replica • Primary tells secondaries to do same at same byte offset in theirs • Primary replies with success to client

24. Client Record Append (2) • When data won’t fit in last chunk: • Primary fills current chunk with padding • Primary instructs other replicas to do same • Primary replies to client, “retry on next chunk” • If record append fails at any replica, client retries operation • So replicas of same chunk may contain different data—even duplicates of all or part of record data • What guarantee does GFS provide on success? • Data written at least once in atomic unit

25. GFS: Consistency Model • Changes to namespace (i.e., metadata) are atomic • Done by single master server! • Master uses log to define global total order of namespace-changing operations

26. GFS: Consistency Model (2) • Changes to data are ordered as chosen by a primary • But multiple writes from the same client may be interleaved or overwritten by concurrent operations from other clients • Record append completes at least once, at offset of GFS’s choosing • Applications must cope with possible duplicates • Failures can cause inconsistency • Behavior is worse for writes than appends

27. Logging at Master • Master has all metadata information • Lose it, and you’ve lost the filesystem! • Master logs all client requests to disk sequentially • Replicates log entries to remote backup servers • Only replies to client after log entries safe on disk on self and backups!

28. Chunk Leases and Version Numbers • If no outstanding lease when client requests write, master grants new one • Chunks have version numbers • Stored on disk at master and chunkservers • Each time master grants new lease, increments version, informs all replicas • Master can revoke leases • e.g., when client requests rename or snapshot of file

29. What If the Master Reboots? • Replays log from disk • Recovers namespace (directory) information • Recovers file-to-chunk-ID mapping (but not location of chunks) • Asks chunkservers which chunks they hold • Recovers chunk-ID-to-chunkserver mapping • If chunk server has older chunk, it’s stale • Chunk server down at lease renewal • If chunk server has newer chunk, adopt its version number • Master may have failed while granting lease

30. What if Chunkserver Fails? • Master notices missing heartbeats • Master decrements count of replicas for all chunks on dead chunkserver • Master re-replicates chunks missing replicas in background • Highest priority for chunks missing greatest number of replicas

31. File Deletion • When client deletes file: • Master records deletion in its log • File renamed to hidden name including deletion timestamp • Master scans file namespace in background: • Removes files with such names if deleted for longer than 3 days (configurable) • In-memory metadata erased • Master scans chunk namespace in background: • Removes unreferenced chunks from chunkservers

32. Limitations • Security? • Trusted environment, trusted users • But that doesn’t stop users from interfering with each other… • Does not mask all forms of data corruption • Requires application-level checksum

33. Limitations • Master biggest impediment to scaling • Performance bottleneck • Holds all data structures in memory • Takes long time to rebuild metadata • Must vulnerable point for reliability • MapReduce: Create many files at once – • Solution: • Have systems with multiple master nodes, all sharing set of chunk servers. • Not a uniform name space. • Large chunk size. • Can’t afford to make smaller, since this would create more work for master. • Mitigated by move to BigTable

34. GFS: Summary • Success: used actively by Google to support search service and other applications • Availability and recoverability on cheap hardware • High throughput by decoupling control and data • Supports massive data sets and concurrent appends • Semantics not transparent to apps • Must verify file contents to avoid inconsistent regions, repeated appends (at-least-once semantics) • Performance not good for all apps • Assumes read-once, write-once workload (no client caching!) • Replaced in 2010 by Colossus • Eliminate master node as single point of failure • Reduce block size to 1MB • Few details public 

35. MapReduce: Execution overview

36. MapReduce: Refinements Locality Optimization • Leverage GFS to schedule a map task on a machine that contains a replica of the corresponding input data. • Thousands of machines read input at local disk speed • Without this, rack switches limit read rate

37. HDFS Hmm… looks familiar

38. GFS vs. HDFS

39. Overview • Google File System (GFS) and Hadoop Distributed File System (HDFS) • BigTable

40. BigTable • Distributed storage system for managing structured data. • Designed to scale to a very large size • Petabytes of data across thousands of servers • Used for many Google projects • Web indexing, Personalized Search, Google Earth, Google Analytics, Google Finance, … • Flexible, high-performance solution for all of Google’s products

41. Motivation • Lots of (semi-)structured data at Google • URLs: • Contents, crawl metadata, links, anchors, pagerank, … • Per-user data: • User preference settings, recent queries/search results, … • Geographic locations: • Physical entities (shops, restaurants, etc.), roads, satellite image data, user annotations, … • Scale is large • Billions of URLs, many versions/page (~20K/version) • Hundreds of millions of users, thousands or q/sec • 100TB+ of satellite image data

42. Basic Data Model • A BigTable is a sparse, distributed persistent multi-dimensional sorted map (row, column, timestamp) cell contents • Good match for most Google applications

43. WebTable Example • Want to keep copy of a large collection of web pages and related information • Use URLs as row keys • Various aspects of web page as column names • Store contents of web pages in the contents: column under the timestamps when they were fetched. • Anchors: is a set of links that point to the page

44. Rows • Name is an arbitrary string • Access to data in a row is atomic • Row creation is implicit upon storing data • Rows ordered lexicographically • Rows close together lexicographically usually on one or a small number of machines

45. Columns • Columns have two-level name structure: • family:optional_qualifier • Column family • Unit of access control • Has associated type information • Qualifier gives unbounded columns • Additional levels of indexing, if desired

46. Timestamps • Used to store different versions of data in a cell • New writes default to current time, but timestamps for writes can also be set explicitly by clients • Lookup options: • “Return most recent K values” • “Return all values in timestamp range (or all values)” • Column families can be marked w/ attributes: • “Only retain most recent K values in a cell” • “Keep values until they are older than K seconds”

47. SSTable • Immutable, sorted file of key-value pairs • Chunks of data plus an index • Index is of block ranges, not values SSTable 64K block 64K block 64K block Index

48. Tablet • Contains some range of rows of the table • Built out of multiple SSTables Start:aardvark End:apple Tablet SSTable SSTable 64K block 64K block 64K block 64K block 64K block 64K block Index Index

49. Tablets • Large tables broken into tablets at row boundaries • Tablet holds contiguous range of rows • Clients can often choose row keys to achieve locality • Aim for ~100MB to 200MB of data per tablet • Serving machine responsible for ~100 tablets • Fast recovery: • 100 machines each pick up 1 tablet for failed machine • Fine-grained load balancing: • Migrate tablets away from overloaded machine • Master makes load-balancing decisions

50. Table • Multiple tablets make up the table • SSTables can be shared • Tablets do not overlap, SSTables can overlap Tablet Tablet apple boat aardvark apple_two_E SSTable SSTable SSTable SSTable