The Conductor: A Framework for Safe and Distributed Computing

1. Conductor A Framework for Distributed, Type-checked Computing Matthew Kehrt

2. Distributed Computing • Today, people are dealing with larger and larger computational problems

3. Distributed Computing • Today, people are dealing with larger and larger computational problems • Scientific computations often analyze enormous amounts of data • Radio telescope data

4. Distributed Computing • Today, people are dealing with larger and larger computational problems • Scientific computations often analyze enormous amounts of data • Radio telescope data • Some problems have large search spaces • Factoring • Password cracking

5. Distributed Computation • A solution – split large computations across multiple computers

6. Distributed Computation • A solution – split large computations across multiple computers • Some projects already do this • SETI@home • folding@home (protein folding) • evolution@home (genetics)

7. Distributed Computation • A solution – split large computations across multiple computers • Some projects already do this • SETI@home • folding@home (protein folding) • evolution@home (genetics) • However, these examples are all for very specific problems

8. Grid Computing • We want something more generic

9. Grid Computing • We want something more generic • A general framework for distributing computation • A way of submitting work to a collection of machines

10. Grid Computing • We want something more generic • A general framework for distributing computation • A way of submitting work to a collection of machines • We want a peer-to-peer network to distribute code

11. Safety? • We want to run code on remote hosts

12. Safety? • We want to run code on remote hosts • How do we guarantee code is benign? • Running code in a safe VM is expensive

13. Safety? • We want to run code on remote hosts • How do we guarantee code is benign? • Running code in a safe VM is expensive • Native code is faster

14. Safety? • We want to run code on remote hosts • How do we guarantee our code is benign? • Running code in a safe VM is expensive • Native code is faster • Can we do static checks for safety on code to run?

15. TAL • Type systems can make static guarantees about program behavior

16. TAL • Type systems can make static guarantees about program behavior • However, we want to send object code over the network

17. TAL • Type systems can make static guarantees about program behavior • However, we want to send object code over the network • Reduces work that needs to be done at runtime • How about a typed assembly language which need be only linked?

18. TAL • A good type system allows us to distribute guaranteed safe code

19. TAL • A good type system allows us to distribute guaranteed safe code • Assembly language means little work needs to be done to prepare distributed code for running.

20. The Conductor • Each machine on the grid runs a Conductor • The Conductor • Maintains a queue of work to be done • Locates peers through a gossip protocol • Queries known peers for others on the network • Downloads, checks and runs work • Returns results

21. The Conductor • Allows for multiple safety policies (different type systems) selected by the user • Publishes code and results in a distributed hashtable • Caches results for quick recomputation • Is tolerant to machines joining and leaving the network

22. Summer Goals • A network running on machines at CMU

23. Summer Goals • A network running on machines at CMU • A larger network running on machines owned by the Pittsburgh public school system

24. Summer Goals • A network running on machines at CMU • A larger network running on machines owned by the Pittsburgh public school system • Adding a new type system to the conductor (TAL Two, or TALT)

25. Summer Goals • A network running on machines at CMU • A larger network running on machines owned by the Pittsburgh public school system • Adding a new type system to the conductor (TAL Two, or TALT) • Ability to merge two grids