Outline

1. Pastry: Scalable, decentralized object location androuting for large-scale peer-to-peer systemsAntony Rowstron and Peter Druschel, Middleware 2001

2. Outline • Introduction • Design of Pastry • Node state & routing • Pastry API • Self-organization and adaptation • Locality • Experimental Results • Discussion

3. Introduction: What is Pastry? • It’s a scalable, distributed, decentralized object location and routing substrate • Serves as a general substrate for building P2P applications: SCRIBE, PAST,…etc. • Seeks to minimize distance messages travel • Pastry’s main capability

4. Pastry Node • Represented by 128-bit randomly chosen nodeId (Hash of IP or public key) • NodeId is in base 2b (b is a configuration parameter; b typical value 2 or 4) • Evenly distributed nodeIds along the circular namespace (0-2128 – 1 space). • Routes a message in O(log N) steps to destination • N: size of network • Node state contains: • Leaf Set ( L ) • Routing table ( R ) • Neighborhood Set ( M)

5. Leaf set: L/2 Numerically closest nodes (L is a configuration parameter = 16, 32 typically ) Routing Table (Prefix-based) Neighborhood Set: M physically closest nodes Design of Pastry:Node state

6. Pastry node state (Leaf Set) • Serves as a fall back for routing table and contains: • L/2 numerically closest and larger nodeIds • L/2 numerically closest and smaller nodIds • Size of L is typically 2b or 2 x 2b • Nodes in L are numerically close (could be geographically diverse)

7. Pastry node state: Neighborhood set (M) • Contains the IP addresses and nodeIds of closest nodes according to proximity metric • Size of |M| is typically 2b or 2x2b • Not used in routing, but instead for maintaining locality properties

8. Node state:Routing Table • Matrix of Log2bN rows and 2b – 1 columns (N is the number of nodes in the network) • Entries in row n match the first n digits of current nodeId AND • Column number follows matched digits: Format: matched digits–column number–rest of ID • Log2b N populated on average

9. Node10233102 (2), (b = 2, l = 8)

10. Routing(2): • If message with key D is within range of leaf set, forward to numerically closest leaf • Else forward to node that shares at least one more digit with D in its prefix than current nodeId • If no such node exists, forward to node that shares at least as many digits with D as current nodeId but numerically nearer than current nodeId

11. Routing Messages (1) Node is in the leaf set (2) Forward message to a closer node (Better match) (3) Forward towards numerically Closer node (not a better match) D: Message Key Li: ith closest NodeId in leaf set shl(A, B): Length of prefix shared by nodes A and B Rij: (j, i)th entry of routing table Source: Rowstron & Drushel, 2001

12. Routing Example: Source: www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf

13. Routing Performance: • (1) If key is within leaf set: • target one hop away • (2) If key has to be routed: • Number of nodes with longer prefix decreases by 2b • (3) Key is not covered by the leaf set (i.e., failed) • With high probability, one more hop needed • Thus: Number of routing steps needed log 2b N

14. Pastry API: • operations exported by pastry: • nodeId = pastryInit(Credentials, Application ) • Causes a Pastry node to join the network with state initialization. Other application specific information is also established. • route(msg, key) • Routes the message to another node which is numerically closest to the key

15. Pastry API • Operations exported by the application working on top of Pastry • deliver(msg,key) • Called when local node is numerically closest to the key • forward(msg, key, nextId) • forward a message from the local nodeId to the next nodeId ( nextId = Null if local node is final) • newLeafs( leafSet): • Updates the leaf set

16. Pastry node join • X = new node, Z = numerically closest node, A = bootstrap (A is close in proximity space to X) • X sends a join message to A with target nodeId X • A forwards to BC… • Stops at Z, numerically closest to X’s nodeId • In process, A,B,…,Z send their state tables to X

17. Node Join • X’s neighborhood set (NS) = A’s NS • X’s Leaf Set = Z’s leaf set • X’s routing table is filled as follows: • X’s Row 0 = A’s row 0 (X0 = A0) • X’s Row 1 = B’s row 1 (X1 = B1) • …etc. • X sends its state to every node in its state tables ( Leaf set, neighborhood set, and routing table)

18. Node Join: Example www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf Source: www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf

19. Node departure (2) • Invalid nodes in leaf set: detected by heartbeat monitor • Repair by inserting node from another leaf’s LS • Heartbeat for neighborhood set (NS) • Query all NS members for their NS tables, choose replacement according to proximity metric • Invalid routing entries detected when attempting to route • Query nodes in row for replacement entry, if failed • Query successive rows until success

20. Node failure in routing table: exampleIf node in red fails Source: www.cs.cornell.edu/courses/ cs514/2003fa/CS514-fa03-lec26v0.pdf

21. Locality in Pastry • Based on proximity metric (i.e., No. of IP hops, geographic distance) • Proximity space is assumed to be Euclidean • The route chosen for a message is likely to be “good“ with respect to the proximity metric • We will discuss locality regarding: • Routing table locality • Route locality • Locating the nearest among k nodes

22. Locality in Routing tables • Invariant: “all routing table entries refer to a node that is near the present node, according to the proximity metric, among all live nodes with a prefix appropriate for the entry.” • We wish to maintain the invariant when adding new nodes. • X joins; A is close to X; X0= A0, so locality holds in X’s routing table • X1 = B1. Entires in B1 (row 1 of X) are close to B, but are they necessarily close to X?

23. Locality in routing table • Entries of B1 are reasonable close to X Why? • A is much closer to B than entry in B1 to B because every time we choose from an exponentially decreasing set of nodes • To improve proximity approximation: • X Queries nodes in routing table and neighborhood set for their state • Compares distances (from routing table entries) and update route entries with closer nodes if found.

24. Route locality • At each routing step the message is moved closer to the destination in the: • nodeId space (numerically closer nodes) • proximity space: message travels the least possible distance • Given that: • A message routed from A to B at a distance dcannot be routed to a node with a distance of less than dfrom A. (follows from routing procedure) • Expected distance traveled increases exponentially • Though shortest path is not guaranteed, we still get a good route.

25. Locality among k nodes • In some Pastry-based applications, object is replicated on k nodes on its route (during insertion) • In prefix-base routing: goal is to reach any of k numerically closest nodes that has a copy of object • May miss nearby nodes with different prefix • Use heuristic to determine when close to k nearest nodes • Based on density of nodeIds that store object; using local info • Switch to numerically closest address

26. Arbitrary node failure • Node continues to be responsive, but behaves incorrectly or maliciously. • Repeated queries fail each time because they normally take the same route. • How to solve it? Use randomized routing • The choice among multiple nodes that satisfy the routing criteria should be made randomly

27. Routing Performance |L|=16 * b=4 * |M|=32 * 200,000 lookups Source: Rowstron & Drushel, 2001

28. Pastry routing Source: Rowstron & Drushel, 2001

29. Routing with failures Source: Rowstron & Drushel, 2001

30. Pastry locality |L|=16 * b=4 * |M|=32 * 200,000 lookups Source: Rowstron & Drushel, 2001

31. Summary • Pastry is a generic P2P object location and routing substrate • Distributed, and scales well • Used in developing applications like file storage, global file sharing,...etc. • Considers locality when routing messeges

32. References (1) A. Rowstron and P. Druschel, "Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems".  IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), Heidelberg, Germany, pages 329-350, November, 2001 (2) Jeff Odom slides:http://x1.cs.umd.edu/818/docs/pastry.ppt