1 / 56

Anycast for Any Service

Anycast for Any Service. Michael J. Freedman Karthik Lakshminarayanan David Mazières http://oasis.coralcdn.org/. What’s the replica-selection problem?. mycdn. Client needs to choose a “good” replica server Performance and cost dependent on replica selection. ?. What do we currently do?.

blanca
Download Presentation

Anycast for Any Service

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Anycast for Any Service Michael J. Freedman Karthik Lakshminarayanan David Mazières http://oasis.coralcdn.org/

  2. What’s the replica-selection problem? mycdn • Client needs to choose a “good” replica server • Performance and cost dependent on replica selection ?

  3. What do we currently do?

  4. How bad can it get?

  5. Anycast is the solution mycdn • Anycast = automated “good” replica selection • OASIS is a flexible anycast system for multiple services ?

  6. The need for anycast • Internet systems rely on replicated content and services • Distributed mirrors: Web servers, FTP servers, … • Content Distribution Networks: Akamai, CoralCDN, … • Internet Naming Systems: DNS, SFR, DOA, … • Distributed File Systems: CFS, Shark, …. • Routing Overlays: RON, Detour, i3, … • Distributed Hash Storage Systems: OpenDHT, … • All could benefit from anycast service

  7. How should one implement anycast?

  8. C D A B E Strawman: probe & find nearest mycdn I ICMP

  9. C D A B E , , E I D Strawman: probe & find nearest mycdn I ICMP  Result highly accurate Lots of probing Slow to compute

  10. C D A B E , , E I D Strawman: probe & find nearest mycdn I ICMP  Result highly accurate Lots of probing Slow to compute

  11. C D A B E , , E I D Avoid probing on-demand mycdn I ICMP  Result highly accurate Lots of probing Slow to compute

  12. C D A B E , , E I D Avoid probing on-demand mycdn • [IMC05] shows IP prefixes often preserve locality • ( 99% of /24s by stub AS at the same location ) I ICMP  Result highly accurate Lots of probing Slow to compute 18.0.0.0/8 • This is the problem • Akamai must solve

  13. E M I N D O What about yourcdn? mycdn yourcdn 18.0.0.0/8  Result highly accurate Lots of probing Slow to compute

  14. (42N,71W) (42N,71W) Idea: Use geographic coordinates mycdn yourcdn 18.0.0.0/8  Amortize costs  Stable across services, time, and failures  Result highly accurate Lots of probing Slow to compute • Assume all replicas • know geo-coords

  15. OASIS provides…  Amortize costs • Balances tension between: • Performance: finding nearest replica • Cost: minimizing 95% bandwidth usage  Stable across time, services, and failures  Result highly accurate  Fast response time  Supports flexible anycast policies

  16. Outline • Architecture and design decisions • Detailed design • Evaluation • Deployment and integration lessons • OASIS deployed since November 2005 • Currently in use by 10 services

  17. mycdn replica proxy = = DNS OASIS node RPC Two-tier architecture mycdn OASIS core • Large set of replicas that assist in measurement • Reliable core of hosts that implement anycast

  18. Using OASIS via DNS mycdn OASIS core 2 1 Resolver Client • Client issues DNS request for mycdn.nyuld.net • OASIS redirects client to nearby application replica

  19. Using OASIS via HTTP mycdn OASIS core 3 2 1 Client • Client issues HTTP request • Web cgi-bin issues RPC to OASIS core • Client redirected to nearby application replica

  20. request IP addr name bucketing service IP prefix policy coords replicas proximity response Using OASIS via HTTP How does core answer anycast?

  21. replicas How does core answer anycast? request IP addr name 18.71.0.3 mycdn.nyuld.net bucketing service IP prefix policy 18.0.0.0/8 mycdn coords proximity 18.26.4.9 171.66.3.181 216.165.109.81 18.26.4.9 response

  22. location accuracy How to map IP prefix to coords? proximity ( Lat, Lng, RTT distance ) IP prefix

  23. location accuracy How to map IP prefix to coords? proximity ( Lat, Lng, RTT distance ) IP prefix (42N,71W) (42N,71W) 18.0.0.0/8 : • Two-pronged approach • Find closest replica proxy

  24. location accuracy How to map IP prefix to coords? proximity ( Lat, Lng, RTT distance ) IP prefix (42N,71W) (42N,71W) 18.0.0.0/8 : , 6.0 ms • Two-pronged approach • Find closest replica proxy • Use closest replica’s geo-coords + error RTT as location

  25. Find replica nearest prefix efficiently “Probe 18.0.0.0/8” 18.168.0.23 18.0.0.0/8 • Two-pronged approach • Find closest replica proxy with less probing • Use closest replica’s geo-coords + error RTT as location

  26. Find replica nearest prefix efficiently 18.168.0.23 [ Meridian 05 ] • Two-pronged approach • Find closest replica proxy with less probing • Use closest replica’s geo-coords + error RTT as location

  27. 18.0.0.0/8 : (42N,71W) , 6.0 ms Find replica nearest prefix efficiently 18.168.0.23 [ Meridian 05 ] • Two-pronged approach • Find closest replica proxy with less probing • Use closest replica’s geo-coords + error RTT as location

  28. Geographic distance vs. RTT • Strong correlation b/w geographical distance and RTT

  29. Geographic distance vs. RTT • Strong correlation b/w geographical distance and RTT • RTT accuracy has real-world meaning • Check if new coordinates improve accuracy vs. old coords

  30. 18.0.0.0/8 18.0.0.0/8 18.0.0.0/8 : : : (42N,71W) (42N,72W) (42N,72W) , 6.0 ms , 3.0 ms , 3.0 ms Geographic distance vs. RTT • Strong correlation b/w geographical distance and RTT • RTT accuracy has real-world meaning • Check if new coordinates improve accuracy vs. old coords [ Meridian 05 ]

  31. [ Meridian 05 ] 18.0.0.0/8 18.0.0.0/8 18.0.0.0/8 : : : (42N,71W) (42N,72W) (42N,72W) , 6.0 ms , 3.0 ms , 3.0 ms Geographic distance vs. RTT • Strong correlation b/w geographical distance and RTT • RTT accuracy has real-world meaning • Check if new coordinates improve accuracy vs. old coords

  32. Geographic distance vs. RTT • Strong correlation b/w geographical distance and RTT • RTT accuracy has real-world meaning • Check if new coordinates improve accuracy vs. old coords • Useful for sanity check for network peculiarities • Do multiple results satisfy constraints (e.g., speed of light) ?

  33. Outline • Architecture and design decisions • Detailed design • Evaluation • Deployment and integration lessons • OASIS deployed since November 2005 • Currently in use by 10 services

  34. OASIS core mycdn opendht • OASIS core • Global membership view • Epidemic gossiping • Scalable failure detection • Spread policies, prefix→coords • Consistent hashing • Divide up responsibility for prefixes • Service replicas • Heartbeats to OASIS node • Form global Meridian overlay for probing

  35. replicas How to find “nearby” nodes? request IP addr name 18.26.4.9 mycdn.nyuld.net bucketing service IP prefix policy 18.0.0.0/8 mycdn coords proximity 18.26.4.9 171.66.3.181 216.165.109.81 Local info from gossiping (stale data okay) 18.26.4.9 response

  36. replicas How to find “nearby” nodes? request IP addr name 18.26.4.9 mycdn.nyuld.net bucketing service IP prefix policy 18.0.0.0/8 mycdn coords proximity 18.26.4.9 171.66.3.181 216.165.109.81 Local info from gossiping (stale data okay) Clients react poorly to stale data 18.26.4.9 response

  37. Aggregate replica information H(srv) OASIS OASIS mycdn • Define service’s rendezvous node via consistent hashing • Service replicas send keepalives to nearby OASIS nodes • Update rendezvous when replicas join, leave, large load change

  38. Aggregate replica information Bottleneck? H(srv) OASIS OASIS mycdn • Define service’s rendezvous node via consistent hashing • Service replicas send keepalives to nearby OASIS nodes • Update rendezvous when replicas join, leave, large load change

  39. Aggregate replica information H(srv) OASIS OASIS mycdn • Aggregate over k nodes for scalability • Rendezvous gossip liveness state for loose consistency • k can be dynamic for better scalability

  40. A client’s view: Finding a nameserver H(dns) OASIS OASIS Client • Core lookup: Contacts 1 of 13 nameservers for .nyuld.net • OASIS “uses itself” to discover replica for service dns

  41. A client’s view: Finding a nameserver H(dns) OASIS OASIS Client • Core lookup: Contacts 1 of 13 nameservers for .nyuld.net • OASIS “uses itself” to discover replica for service dns • Returns nearby nameservers for subsequent requests

  42. A client’s view: Finding a replica H(dns) OASIS OASIS H(mycdn) R Client • Replica lookup: Client contacts nearby nameserver • OASIS discover replica for service mycdn • Returns nearby replicas for application

  43. Evaluation • Deployed on PlanetLab since November 2005 • How much end-to-end benefit from OASIS? • How accurate is OASIS? • Effective for load balancing? • What are OASIS’s bandwidth costs?

  44. E2E download of web page 290% faster than Meridian 500% faster than RR Cached virtual coords highly inaccurate

  45. Client RTT to chosen replica Outperforms Meridian 60% of time

  46. OASIS minimizes bandwidth spikes 95% bandwidth usage per replica (MB) • 8 clients in CA repeatedly request 1 MB file • Replicas report load as log (95% bandwidth per 1-min slot) loc metric

  47. OASIS minimizes bandwidth spikes 95% bandwidth usage per replica (MB) • 8 clients in CA repeatedly request 1 MB file • Replicas report load as log (95% bandwidth per 1-min slot) loc metric

  48. Bandwidth costs: OASIS v. on-demand 1-2 orders of magnitude # DNS reqs to CoralCDN

  49. Outline • Architecture and design decisions • Detailed design • Evaluation • Deployment and integration lessons • OASIS deployed since November 2005 • Currently in use by 10 services

  50. Sanity check for network peculiarities • Employ measurement redundancy • Easy visualization significantly helped debugging

More Related