Introduction to Data Center Computing

1. Introduction to Data Center Computing Derek Murray October 2010

2. What we’ll cover • Techniques for handling “big data” • Distributed storage • Distributed computation • Focus on recent papers describing real systems

3. Example: web search Crawling Indexing Querying WWW

4. A system architecture? Network Computers Storage

5. Data Center architecture Core switch Rack switch Rack switch Rack switch Server Server Server Server Server Server Server Server Server

6. Distributed storage • High volume of data • High volume of read/writerequests • Fault tolerance

7. Brewer’s CAP theorem (2000)

8. The Google file system (2003) GFS Master Client Chunk server Chunk server Chunk server

9. Dynamo (2007) Client

10. Distributed computation • Parallel distributed processing • Single Program, Multiple Data (SPMD) • Fault tolerance • Applications

11. Task farming Master Worker Worker Worker Storage

12. MapReduce(2004)

13. Dryad (2007) • Arbitrarydirected acyclic graph (DAG) • Vertices and channels • Topological ordering

14. DryadLINQ(2008) • Language Integrated Query (LINQ) vartable = PartitionedTable.Get<int>(“…”); varresult = from x in table selectx * x; intsumSquares = result.Sum();

15. Scheduling issues • Heterogeneous performance • Sharing a cluster fairly • Data locality

16. Percolator (2010) • Built on Google BigTable • Transactions via snapshot isolation • Per-column notifications (triggers)

17. Skywriting and Ciel(2010) • Universal distributed execution engine • Script language for distributed programs • Opportunities for student projects…

18. References • Storage • Ghemawatet al., “The Google File System”, Proceedings of SOSP 2003 • DeCandiaet al., “Dynamo: Amazon’s Highly-Available Key-value Store”, Proceedings of SOSP 2007 • Computation • Dean and Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, Proceedings of OSDI 2004 • Isardet al., “Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks”, Proceedings of EuroSys 2007 • Yu et al., “DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language”, Proceedings of OSDI 2008 • Olstonet al., “Pig Latin: A Not-So-Foreign Language for Data Processing”, Proceedings of SIGMOD 2008 • Murray and Hand, “Scripting the Cloud with Skywriting”, Proceedings of HotCloud 2010 • Scheduling • Zahariaet al., “Improving MapReduce Performance in Heterogeneous Environments”, Proceedings of OSDI 2008 • Isardet al., “Quincy: Fair Scheduling for Distributed Computing Clusters”, Proceedings of SOSP 2009 • Zahariaet al., “Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling”, Proceedings of EuroSys2010 • Transactions • Peng and Dabek, “Large-Scale Incremental Processing using Distributed Transactions and Notifications”, Proceedings of OSDI 2010

19. Conclusions • Data centers achieve high performance with commodity parts • Efficient storage requires application-specific trade-offs • Data-parallelism simplifies distributed computation on the data

20. Questions • Now or after the lecture • Email • Derek.Murray@cl.cam.ac.uk • Web • http://www.cl.cam.ac.uk/~dgm36/