Outline for today

1. Outline for today • Structured overlay as infrastructures • Survey of design solutions • Analysis of designs

2. Questions: • How many DHTs will there be? • Can all applications share one DHT?

3. Benefits of Sharing a DHT • Amortizes costs across applications • Maintenance bandwidth, connection state, etc. • Facilitates “bootstrapping” of new applications • Working infrastructure already in place • Allows for statistical multiplexing of resources • Takes advantage of spare storage and bandwidth • Facilitates upgrading existing applications • “Share” DHT between application versions

4. K V K V K V K V K V K V K V K V K V K V The DHT as a Service DHT

5. The DHT as a Service DHT Clients

6. The DHT as a Service DHT

7. The DHT as a Service What is this interface? OpenDHT

8. It’s not lookup() lookup(k) Challenges: Distribution Security What does this node do with it? k

9. What about put/get? • Works easily for storage applications • Easy to share • No upcalls, so no code distribution or security complications • But does it work for rendezvous? • Chat? Sure: put(my-name, my-IP) • What about the others?

10. A Case for Common APIs • Lots and lots of peer to peer applications • Decentralized file systems, archival backup • Group communication / coordination • Routing layers for anonymity, attack resilience • Scalable content distribution • A number of scalable, self-organizing overlays • E.g. CAN, Chord, Pastry, Tapestry, Kademlia, etc… • Semantic differences • Store/get data, locate objects, multicast / anycast • How do these functional layers relate? • What is the smallest common denominator?

11. How are DHT’s used – Some Real Applications • Many possibilities • Storage, media delivery, resilient routing… • Storage • Distributed file systems • Automatic backup services • Distributed CVS • Media delivery • Wide-area multicast systems, pub-sub • Video-on-demand systems • Content distribution networks (CDNs) • Multicast, anycast •  Rendezvous • Resilient routing • Route around Internet failures

12. Some Abstractions • Distributed Hash Tables (DHT) • Simple store and retrieve of values with a key • Values can be of any type • Decentralized Object Location and Routing (DOLR) • Decentralized directory service for endpoints/objects • Route messages to nearest available endpoint • Multicast / Anycast (CAST) • Scalable group communication • Decentralized membership management

13. Tier 1 Interfaces

14. Structured P2P Overlays CFS PAST i3 SplitStream Bayeux OceanStore Tier 2 Tier 1 DHT CAST DOLR Key-based Routing Tier 0

15. The Challenges • Maintenance • Many components, many administrative domains • Constant change • Must be self-organizing • Must be self-maintaining • All resources virtualized—no physical names • Security • High availability is a hacker’s target-rich environment • Must have end-to-end encryption • Must not place too much trust in any one host

16. The Big Picture • For durability (archival layer) • Apply Erasure coding • Replicate copies of fragments across network • Periodically check for level of redundancy • For quick access (dissemination layer) • Maintain small # of copies replicated in network • Use access tracking to move copies closer to clients • For attack resilience (Byzantine layer) • Various solutions • Key decisions made per file by “inner-ring” of servers, Distributed decisions verified by a “threshold signature” • Signed data • Self-certifying data (SFS File System)