Pregel : A System for Large-Scale Graph Processing

1. Pregel: A System for Large-Scale Graph Processing Presented by Dylan Davis Authors: GrzegorzMalewicz, Matthew H. Austern, Aart J.C. Bik, James C. Dehnert, Ilan Horn, NatyLeiser, GrzegorzCzajkowski (GOOGLE, INC.)

2. Overview • What is a graph? • Graph Problems • The Purpose of Pregel • Model of Computation • C++ API • Implementation • Applications • Experiments

3. What is a graph? G = (V, E) Binary Tree

4. Graph Problems Social Network Connections Network Routing

5. The Purpose of Pregel • Google was interested in applications that could perform internet-related graph algorithms, such as PageRank, so they designed Pregel to perform these tasks efficiently. • It is a scalable, general-purpose system for implementing graph algorithms in a distributed environment. • Focus on “Thinking Like a Vertex” and parallelism

6. Model of Computation

7. Model of Computation (Vertex) Vertex ID Edge Value Vertex ID Edge Value Vertex ID Vertex Value

8. Model of Computation (Superstep) Superstep 1 Superstep 2 Superstep 0 Compute() Compute() Compute() Compute() Compute() Compute() Compute() Compute() Compute() Execution Time

9. Model of Computation (Vertex Actions) A vertex can: • Modify its values • Receive messages from previous superstep • Send messages • Request topology changes Vertex ID Vertex Value

10. Model of Computation (State Machine)

12. C++ API

14. C++ API (Message Passing) Destination Vertex ID Message Value Message Buffer 2 1 2 57

15. C++ API (Combiners & Aggregators) Combiner Aggregator

16. C++ API (Topology Mutations) V Superstep

17. C++ API (Input and Output) 0 1 2 3 4 0 0 0 1 1 0 1 0 0 0 1 1 2 1 1 0 1 1 3 0 1 1 0 1 4 1 1 1 0 0

18. Implementation

19. Implementation (Basic Architecture)

20. Implementation (Program Execution) Flow: • Copy user program – Master copy & worker copies • Master assigns graph partitions • Master takes user input data, assigns to workers – load vertex data • Supersteps (Compute() and send messages) • Save output

21. Implementation (Fault Tolerance) Recover Checkpoint Worker Save() Worker Recompute() Worker Save() X Worker Worker Recompute() Worker Save()

22. Implementation (Worker) Worker Worker

23. Implementation (Master) List of Workers Master Partitions

24. Applications

25. Applications (Shortest Path) 1 2 5 3

26. Experiments

27. Experiments (Description) • Test the execution times of Pregel running the Single-Source Shortest Path algorithm. • Use a cluster of 300 multicore commodity PCs. • Run Pregel with Binary Tree graphs, and with a more realistic, randomly-distributed graph. • Results do not include initialization, graph generation, and result verification times. • Failure Recovery is not included (reduces overhead)

31. Conclusion • Pregel is a model suitable for large-scale graph computing with a production-quality, scalable and fault tolerant implementation. • Programs are expressed as a sequence of iterations, in each of which a vertex can receive messages sent in the previous iteration, send messages to other vertices, and modify its own state and that of its outgoing edges. • This implementation is flexible enough to express a broad set of algorithms.