1 / 21

Dryad

Dryad. Distributed Data-Parallel Programs from Sequential Building Blocks. Michael Isard , Mihai Budiu , Yuan Yu, Andrew Birrell , Dennis Fetterly of Microsoft Research, Silicon Valley Presented by: Thomas Hummel. Agenda. Introduction System Overview Dryad Graph

chelsey
Download Presentation

Dryad

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dryad Distributed Data-Parallel Programs from Sequential Building Blocks Michael Isard, MihaiBudiu, Yuan Yu, Andrew Birrell, Dennis Fetterly of Microsoft Research, Silicon Valley Presented by: Thomas Hummel

  2. Agenda • Introduction • System Overview • Dryad Graph • Program Development • Program Execution • Experimental Results • Future Work

  3. Introduction • Problem • How to write efficient distributed programs easily? • Environment • Parallel Processors • High Speed Links • Administered Domain • Ignore Low Level Issues

  4. Introduction • Parallel Execution • Faster Execution • Automatic Specification • Manual Specification • GPU Shader • Distributed Databases • MapReduce

  5. Introduction • Graph Model • Verticies Are Programs • Edges Are Communication Links • Forced Parallelism Mindset • Necessary Abstraction

  6. Introduction • GPU Shader • Low Level • Hardware Specific • MapReduce • Simplicity Paramount • Performance Sacrificed • Database • Implicit Communication • Algebra Optimized

  7. Introduction • Dryad • Fine Communication Control • Multiple Input/Output Sets • Must Consider Resources • Execution Engine • Executes DAG Of Programs • Outputs Directed To Inputs • No Recursion

  8. System Overview • Dryad Job • DAG • Data Passed On Edges • Vertex is a Program • Message Structure • User Defined • Shared Memory • TCP • Files

  9. System Overview • Dryad Job • DAG • Data Passed On Edges • Vertex is a Program • Message Structure • User Defined • Shared Memory • TCP • Files

  10. System Overview • System Organization • Job Manager • Name Server • Dameon (Work Nodes)

  11. Dryad Graph • Graph Description Language • “Embedded” in C++ • Combine Sub-Graphs • C++ Class • Inherited By Vertex Program • Program Name • Program Factory

  12. Dryad Graph • Vertex Creation • C++ Class • Inherited By Vertex Program • Program Name • Program Factory • One Vertex Is a Graph • Factory Called • Program Specific Arguments Applied

  13. Dryad Graph • Edge Creation • Composition (Combine) Operation • Two Graphs • Varying Assignment Methods

  14. Dryad Graph

  15. Dryad Graph • Communication Channel • File I/O By Default • TCP • Shared Memory • Pitfall: Connected Vertices Must Be On Same Process • Deadlock Avoidance • DAG Architecture

  16. Program Development • Vertex Program Development • C++ Base Classes • Status And Errors Reported to Job Manager • Standard “Main” Method • Channel Readers/Writers • Supplied Via Argument List • Legacy Programs • C++ Wrapper

  17. Program Development • Pipelined Execution • Assuming Sequential Code • Event Based Programming • Channels Are Asynchronous • Thread Pool • Optimized For Verticies

  18. Program Execution • Job Manager • Job Ends If JM Machine Fails • Different Schemes Possible To Avoid This • Versioning System For Execution Instances • Vertex Execution • Starts When All Input Channels Ready • User Can Specify Execution Machine • Can Be Re-Run On Failures • Job Ends After All Verticies Have Run

  19. Program Execution • Fault Tolerance • Re-Run Vertex If Failed • Channel Re-Creation (File Recreation) • TCP/Shared Memory Failures Cause Failures On All Connected Vertices • Staged Execution Allows Intermediate Error Checking

  20. Experimental Results • SQL Operation • 10 Computer Cluster • Gigabit Connections • Data Mining Operation • 1800 Computer Cluster • 10 TB Data Set • 11 Minute Execution Time

  21. Future Work • Scripting Language • Nebula • Additional Abstraction • SISS Integration • SQL Server Integration • Distributed SQL Queries • Query Optimizer

More Related