1 / 60

EPIDEMIC TECHNIQUES

EPIDEMIC TECHNIQUES. Ki Suh Lee. OUTLINE. Epidemic Protocol Epidemic Algorithms for Replicated Database Maintenance Astrolabe: A Robust and scalable technology for distributed system monitoring, management, and data mining . EPIDEMIC PROTOCOL.

keyanna
Download Presentation

EPIDEMIC TECHNIQUES

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EPIDEMIC TECHNIQUES Ki Suh Lee

  2. OUTLINE • Epidemic Protocol • Epidemic Algorithms for Replicated Database Maintenance • Astrolabe: A Robust and scalable technology for distributed system monitoring, management, and data mining

  3. EPIDEMIC PROTOCOL • Gossip Protocol: efficient and robust for data dissemination • From Mathematical Theory of Epidemiology • Direct-mail: Each update is mailed to all sites • Anti-entropy: Every site regularly chooses another site at random and resolves differences • Rumor mongering: Periodically chooses another site at random and ensures the update. When a site has tried to share a hot rumor with too many sites that have already seen it, stop propagating it

  4. EXAMPLES OF GOSSIP • Anti-entropy • Bayou(Xerox, 1995) • Failure Detection Systems(Cornell, 1998) • Bimodal Multicast(Cornell, 1998) • Astrolabe(Cornell, 1999) • Lightweight Probabilistic Broadcast(2003) • Kelips(Cornell, 2003) • Bamboo(Berkeley, 2003) • Rumor Mongering • Trickle(2004)

  5. EPIDEMIC ALGORITHM FOR REPLICATED DATABASE MAINTENANCE Alan Demers, Dan Greene, Carl Hauser, Wes Irish, John Larson, Scott Shenker, Howard Sturgis, Dan Swinehart, and Doug Terry

  6. AUTHORS Dan Greene Palo Alto Research Center Alan Demers Cornell Univ. Wesley Irish Coyote Hill Consulting LLC Carl Hauser Washington State Univ. John Larson Howard Sturgis Dan Swinehart Doug Terry MS Research Silicon Valley Scott Shenker EECS Berkley

  7. MOTIVATION • Clearinghouse Servers on Xerox Corporate Internet(CIN) (1987) • Several thousand workstations, servers, and computing hosts connected by gateways and phone lines • Performance problems due to traffic created to achieve consistency on replicated domains • Used direct mail and anti-entropy. • Anti-entropy couldn’t be completed because of the load on the network.

  8. INTRODUCTION • Goal: • To design algorithms for distributing updates and driving replicated database toward consistency • Efficient, robust, and scalable • Two Important Considerations • The time for an update to propagate to all sites • The network traffic generated in propagating a single update

  9. EPIDEMIC ALGORITHMS • Direct Mail • Anti-entropy • Rumor mongering TERMINOLOGY • “infective”: a site holding an update it is willing to share • “susceptible”: a site that has not yet received an update • “removed”: a site that has received an update but is no longer willing to share

  10. DIRECT MAIL • Notify all other sites of an update • When update received, check timestamp • Timely and reasonably efficient • Not completely reliable. • Messages may be discarded • Source may not know all other nodes • n messages per update

  11. DIRECT MAIL

  12. DIRECT MAIL May not know this node

  13. DIRECT MAIL Messages may be dropped

  14. ANTI-ENTROPY(SIMPLE EPIDEMIC) • every site regularly chooses another site at random • by exchanging content, resolves any differences between them • Eventually infect the entire population • Extremely reliable, butexpensive • Propagates more slowly than direct mail • O(log n) to eventually infect all nodes • Differences are resolved using: • Push: infective -> susceptible • Pull: susceptible -> infective • Push-pull: depending on timestamp, chooses either push or pull

  15. ANTI-ENTROPY

  16. ANTI-ENTROPY

  17. ANTI-ENTROPY

  18. ANTI-ENTROPY

  19. ANTI-ENTROPY

  20. ANTI-ENTROPY

  21. ANTI-ENTROPY

  22. ANTI-ENTROPY

  23. ANTI-ENTROPY

  24. ANTI-ENTROPY

  25. PUSH VS. PULL • pi – Probability that a node is susceptible after the ith round • Pull: • Push: • Pull converges faster than push = pi e-1

  26. ANTI-ENTROPY: OPTIMIZATIONS • Comparing two complete copies is expensive • For performance improvement, we can use • Checksum • Recent Update Lists • Inverted Index by timestamp

  27. COMPLEX EPIDEMICS • Sites are initially “susceptible” • If a node receives an update(“hot rumor”), it becomes “infective” • Randomly choose a node and share the update • If chosen node is “infective”, the node becomes “removed” with probability 1/k • k=1, 20% miss the rumor • k=2, 6% miss the rumor • Requires fewer resources • There is a chance that an update will not reach all sites.

  28. RUMOR MONGERING

  29. RUMOR MONGERING

  30. RUMOR MONGERING

  31. RUMOR MONGERING

  32. RUMOR MONGERING

  33. VARIATIONS OF COMPLEX EPIDEMIC • Criteria to judge epidemics • Residue: remaining susceptibles when the epidemic finishes • Traffic: Total update traffic / # of sites • Delay: • tavg : Average of the difference between the time of the initial injection and the arrival of the update at a given site • tlast : the delay until the last site receives the update

  34. VARIATIONS (CONT’) • Blind vs. Feedback • Counter vs. Coin • Push vs. Pull • Minimization • Connection Limit • Hunting

  35. Table 1. Push, Feedback & Counters Table 2. Push, Blind & Coin Table 3. Pull, Feedback & Counters

  36. COMBINING ANTI-ENTROPY & COMPLEX EPIDEMIC • Complex Epidemic • (+) spread updates rapidly • (+) low network traffic • (-) sometimes can fail • Anti-entropy • (+) robust • (-) expensive • Complex Epidemic for an initial distribution / redistribution and Anti-entropy for an infrequent backup • Every update eventually reaches every site • Can significantly reduce the cost of distribution

  37. DELETION AND DEATH CERTIFICATES • Replace deleted items with death certificates(DC) which carry timestamps • When old copies of deleted items meet death certificates, remove them • Delete death certificates • When every site has seen it • After holding it for fixed threshold of time

  38. DORMANT DEATH CERTIFICATES • Tradeoff between the amount of space and the risk of obsolete items being resurrected • After threshold t1, delete old DCs at most sites, retaining “dormant” at only a few cites • When obsolete update encounters a dormant DC, propagate DC again • After t1+t2, discard dormant DCs.

  39. SPATIAL DISTRIBUTIONS • Nonuniform spatial distribution can significantly reduce the traffic • Favor nearby neighbors • Trade off between convergence time and average traffic per link Europe North America Bushey CIN topology

  40. SPATIAL DISTRIBUTION: ANTI-ENTROPY • Can significantly reduce traffic on critical links • Slow convergence doesn’t change the total amount of traffic • Robust: With probability 1, update converges.

  41. SPATIAL DISTRIBUTION:RUMOR MONGERING • Less robust than anti-entropy • Increase k to give 100% distribution • Worthwhile improvements

  42. CONCLUSION • Characteristics of Anti-entropy and Rumor Mongering • Anti-entropy • can spread updates to all node with probability 1 • Reduces average link traffic • Rumor Mongering • If combined with anti-entropy, can greatly reduce the cost of redistribution • Requires Membership though

  43. ASTROLABE: A ROBUST AND SCALABLE TECHNOLOGY FOR DISTRIBUTED SYSTEM MONITORING, MANAGEMENT, AND DATA MINING Robbert van Renesse, Kenneth P. Birman, and Werner Vogels

  44. AUTHORS Ken Birman Cornell Univ. Werner Vogels Amazon.com CTO Robbert van Renesse Cornell Univ.

  45. OVERVIEW • Distributed Information Management System • Monitors the dynamically changing state of resources • Reports summaries of information • Design principles • Scalability through hierarchy • Flexibility through mobile code • Robustness through a randomized p2p protocol • Security through certificates • Now used in Amazon.com

  46. ZONE AND MIB • Zone name: path of zone identifiers from the root • Management Information Base(MIB): attribute list containing the information associated with the zone • Each agent has a local copy of the root MIB, the MIBs of each child of the root • Each agent maintains MIB list of child zones

  47. GOSSIP PROTOCOL • Epidemic p2p protocol to propagate information • Each zone elects a set of representative agents to gossip on behalf of that zone. • An agent may represent more than one zone • Each agent periodically gossips • Picks one of the child zones at random, • Picks a random agent from child’s list • Sends attributes of all the child zones up to root level. • Gossip spreads quickly, O(log n) • For scalability, robustness, and rapid dissemination of updates, eventual consistency is adopted.

  48. EVENTUAL CONSISTENCY • Probabilistic consistency • Given an aggregate attribute X that depends on some other attribute Y • When an update u is made to Y, either u itself, or an update to Y made after u, is eventually reflected in X • With probability 1

  49. Name Name Time Time Load Load Weblogic? Weblogic? SMTP? SMTP? Word Version Word Version swift swift 2003 2003 .67 .67 0 0 1 1 6.2 6.2 falcon falcon 1976 1976 2.7 2.7 1 1 0 0 4.1 4.1 cardinal cardinal 2231 2201 3.5 1.7 1 1 1 1 6.0 6.0 ASTROLABE IS A FLEXIBLE MONITORING OVERLAY swift.cs.cornell.edu Periodically, pull data from monitored systems cardinal.cs.cornell.edu From Ken’s slide http://www.cs.cornell.edu/courses/cs614/2006fa/Slides/Epidemics.ppt

  50. Name Time Load Weblogic? SMTP? Word Version swift 2003 .67 0 1 6.2 falcon 1976 2.7 1 0 4.1 cardinal 2201 3.5 1 1 6.0 STATE MERGE: CORE OF ASTROLABE EPIDEMIC swift.cs.cornell.edu cardinal.cs.cornell.edu From Ken’s slide http://www.cs.cornell.edu/courses/cs614/2006fa/Slides/Epidemics.ppt

More Related