SkipNet: A Scalable Overlay Network with Practical Locality Properties

SkipNet: A Scalable Overlay Network with Practical Locality Properties Nick Harvey, Mike Jones, Stefan Saroiu, Marvin Theimer, Alec Wolman Microsoft Research University of Washington

Overlay Networks • Overlays have achieved several goals: • Scalable and decentralized infrastructure • Uniform and random load and data distribution • But, at the price of data controllability • Data may be stored far from its users • Data may be stored outside its domain • Local accesses leave local organization • Basic trade-off: data controllability vs. data uniformity • SkipNet: • Traditional overlay functionality • Provides an abstraction to control this trade-off: • Constrained load balancing (CLB)

Talk Outline • Practical data locality requirements • Basic SkipNet design • SkipNet locality properties • Performance evaluation • Conclusions

Key Locality Properties and Abstraction • In practice, two properties are important: • Content Locality – ability to explicitly place data • Placement on a single node or on a set of nodes • Path Locality – ability to guarantee that local traffic remains local • One abstraction is important – CLB: • SkipNet abstraction to control the trade-off • Multiple DHT scopes within one single overlay

Practical Requirements • Data Controllability: • Organizations want control over their own data • Even if local data is globally available • Manageability: • Data control allows for data administration, provisioning and manageability • Data center/cluster = constrained set of nodes • CLB ensures load balance across data center/cluster

Practical Requirements (cont’d) • Security: • Content and path locality are key building blocks for dealing with certain external attacks • Data availability • Local data survives network partitions • Performance • Data can be stored near clients that use it

SkipNet • Key property: two address spaces • Name ID space: nodes are sorted by their names (e.g. DNS names) • Numeric ID space: nodes are randomly distributed • Combining both spaces achieves • Content + Path locality • Other uses could emerge: range queries [AS ’03] • Scalable peer-to-peer overlay network • O(log N) routing performance in both spaces • O(log N) routing state per node

X Z T A D V O M SkipNet Ring • Pointers at level h skip over 2h nodes • Nodes are ordered by names

S Z H A E M G F SkipNet Ring • Pointers at level h skip over 2h nodes • Nodes are ordered by names

Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 O M D A T Z V X M O D Ring 01 Ring 00 Ring 10 Ring 11 A T V Z X M O D Ring 1 Ring 0 T A Z V X SkipNet Global View L = 3 L = 2 L = 1 M D O Root Ring Level: L = 0 T A Z V X

Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 M O D A T Z V X M O D Ring 01 Ring 00 Ring 10 Ring 11 A T V Z X M O D Ring 1 Ring 0 T A Z V X SkipNet Global View L = 3 L = 2 L = 1 M D O Root Ring Level: L = 0 T A Z V X

Two Address Spaces • SkipNet can route efficiently in both address spaces: • Name ID space (e.g. DNS names) • Numeric ID space

Node A’s Routing Table Routing by Name ID Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 M O D L = 3 T A V X Z M D Ring 01 Ring 00 Ring 10 Ring 11 A T L = 2 O V Z X M O D Ring 1 Ring 0 T L = 1 A Z V X M D O Root Ring T A Level: L = 0 Z V X • Example: route from A to V • Simple Rule: Forward the message to node that is closest to dest, without going too far.

Routing by Name ID Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 M O D L = 3 T A V X Z M D Ring 01 Ring 00 Ring 10 Ring 11 A T L = 2 O V Z X M O D Ring 1 Ring 0 T L = 1 A Z V X M D O Root Ring T A Level: L = 0 Z V X • Example: route from A to V • Simple Rule: Forward the message to node that is closest to dest, without going too far.

Node T’s Routing Table Routing by Name ID Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 M O D L = 3 T A V X Z M D Ring 01 Ring 00 Ring 10 Ring 11 A T L = 2 O V Z X M O D Ring 1 Ring 0 T L = 1 A Z V X M D O Root Ring T A Level: L = 0 Z V X • Example: route from A to V • Simple Rule: Forward the message to node that is closest to dest, without going too far.

Routing by Name ID Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 M O D L = 3 T A V X Z M D Ring 01 Ring 00 Ring 10 Ring 11 A T L = 2 O V Z X M O D Ring 1 Ring 0 T L = 1 A Z V X M D O Root Ring T A Level: L = 0 Z V X • Example: route from A to V • Simple Rule: Forward the message to node that is closest to dest, without going too far.

Routing by Numeric ID • Provides the basic DHT primitive • To store file “Foo.c” • Hash(“Foo.c”)  a random numeric ID • Find highest ring matching that numeric ID • Store file on node in that ring • Log N routing efficiency

Foo.c DHT Example Ring 100 Ring 101 Ring 110 Ring 111 Ring 000 Ring 001 Ring 010 Ring 011 • Store file “Foo.c” from node A • Hash(“Foo.c”) = 101… • Route from A to V in numeric space M O D L = 3 T A V Z X M D Ring 01 Ring 00 Ring 10 Ring 11 A T L = 2 O V Z X M O D Ring 1 Ring 0 T L = 1 A Z V X M D O Root Ring T A Level: L = 0 Z V X

Numeric Routing Name Routing Constrained Load Balancing (CLB) • Multiple DHTs with differing scopes using a single SkipNet structure • A result of the ability to route in both address spaces • Divide data object names into 2 partsusing the ‘!’ special character CLB DomainCLB Suffix microsoft.com!skipnet.html

skipnet. html CLB Example • To read file “com.microsoft!skipnet.html” • Route by name ID to “com.microsoft” • Route by numeric ID to Hash(“skipnet.html”)within the “com.microsoft” constraint com.microsoft com.sun gov.irs edu.ucb

com.microsoft.research SkipNet Path Locality • Organizations correspond to contiguous SkipNet segments • Internal routing by NameID remains internal • Nodes have left / right pointers com.microsoft com.sun gov.irs edu.ucb

Fault Tolerance • Many failures occur along organizational boundaries: • Gateway/firewall failure, BGP misconfig, physical network cut, … • SkipNet handles organizational disconnect gracefully • Results in two well-connected, partitioned SkipNets • Efficient remerging algorithms • Node independent failures • Same resiliency as systems such as Chord and Pastry • Similar approach to repair (Leaf Set)

Primary Security Benefit & Weakness • SkipNet + name access control mechanism: • Content locality ensures that content stays within organization • Path locality prevents: • malicious forwarders • analysis of internal traffic • external tampering • Easier to target organizations: • Someone creates one million nodes with name prefixes microsofa.com and microsort.com • Most traffic to/from Microsoft will go through a microsofa / microsort intermediate node

Methodology • Packet-level, event-driven simulator: • SkipNet implementation • Basic SkipNet • Full SkipNet = Basic SkipNet + network proximity • Pastry and Chord implementation • Uses Mercator and GT-ITM network topologies • Experimentally evaluated: • Name ID routing performance • Tolerance to organizational disconnect

Methodology • Packet-level, event-driven simulator: • SkipNet implementation • Basic SkipNet • Full SkipNet = Basic SkipNet + network proximity • Pastry and Chord implementation • Uses Mercator and GT-ITM network topologies • Experimentally evaluated: • Name ID routing performance • Tolerance to organizational disconnect • Numeric ID routing performance • Effectiveness of network proximity optimizations • Effectiveness of CLB routing optimizations

Routing by Name ID Performance Benefits come at no extra cost

Surviving Organizational Disconnect Disconnected Org Size = 15% of all nodes

Conclusions • SkipNet: • Traditional overlay functionality • Explicit control of data placement • Constrained load balancing • Content + Path Locality are basic ingredients to: • Data controllability • Manageability • Security • Data availability • Performance

Questions? http://research.microsoft.com/sn/Herald/

SkipNet: A Scalable Overlay Network with Practical Locality Properties