1 / 14

HDFS: Hadoop Distributed FS

HDFS: Hadoop Distributed FS. Steve Loughran, Hortonworks stevel@hortonworks.com @steveloughran ATLAS workshop, June 2013. What is a Filesystem?. Persistent store of data: write, read, probe, delete Metadata for organisation: locate, change A conceptual model for humans

shayna
Download Presentation

HDFS: Hadoop Distributed FS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDFS: Hadoop Distributed FS Steve Loughran, Hortonworks stevel@hortonworks.com @steveloughran ATLAS workshop, June 2013

  2. What is a Filesystem? • Persistent store of data: write, read, probe, delete • Metadata for organisation: locate, change • A conceptual model for humans • API for programmatic access to data & metadata • Unix is the model & POSIX its API

  3. Unix is the model & POSIX its API • directories and files: directories have children, files have data • API: open, read, seek, write,stat, rename, unlink, flock • Consistency: all sync()'d changes are globally visible • Atomic metadata operations: mv, rm, mkdir • Features are also constraints

  4. Relax constraints  scale and availability S3+cross-siteappendmetadata ops consistency HDFS+data locality (seek+write)locks Scale and availability NFS+cross hostlocks, sync ext4 Distance from Unix Filesystem model & API

  5. HDFS: what • Java code on Linux, Unix, Windows • Open Source: hadoop.apache.org • Replication rather than RAID • break file into blocks • store across servers and racks • delivers bandwith and more locations for work • Background work handles failures • replication of under-replicated blocks • rebalancing of unbalanced servers • checksum verification of stored files Location data for the Job Scheduler

  6. HDFS: why? • Store Petabytes of web data: logs, web snapshots • Keep per-node costs down to afford more nodes • Commodity x86 servers, storage (SAS), GbE LAN • Accept failure as a background noise • Support computation in each server Written for location aware applications -MapReduce, Pregel/Giraph & others that can tolerate partial failures

  7. Some of largest filesystemsever An emergent software stack

  8. HDFS: what next? • Exabytes in a single cluster • Cross cluster, cross-sitewhat constraints can be relaxed here? • Data Provenance, tainting • Evolving application needs. • Power budgets

  9. HDD  HDD+ SSD  SSD • New solid state storage technologies emerging • When will HDDs go away? • How to take advantage of mixed storage • SSD retains the HDD metaphor, hides the details (access bus, wear levelling) We need to give the OS and DFS control of the storage, work with the application

  10. Download and Play! http://hadoop.apache.org http://hortonworks.com

  11. P.S: we are hiring http://hortonworks.com/careers/

  12. Hadoop HDFS: replication is the key Switch ToR Switch ToR Switch ToR Switch fileblock1block2block3 … DataNode DataNode DataNode DataNode DataNode DataNode DataNode DataNode 2ary Name Node (Job Tracker) Name Node

  13. Replication handles data integrity • CRC32 checksum per 512 bytes • Verified across datanodes on write • Verified on all reads • Background verification of all blocks (~weekly) • Corrupt blocks re-replicated • All replicas corrupt  operations team intervention 2009: Yahoo! lost 19 out of 329M blocks on 20K servers –bugs now fixed

  14. Rack/Switch failure Switch ToR Switch ToR Switch ToR Switch fileblock1block2block3 … DataNode DataNode DataNode DataNode DataNode DataNode DataNode DataNode 2ary Name Node (Job Tracker) Name Node

More Related