1 / 26

Indexing data-oriented overlay networks

2005. Indexing data-oriented overlay networks. Presented by: Anwitaman Datta Joint work with Karl Aberer, Manfred Hauswirth, Roman Schmidt Ecole Polytechnique Fédérale de Lausanne (EPFL). Patrons:. NCCR-MICS: www.mics.ch/.

belva
Download Presentation

Indexing data-oriented overlay networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2005 Indexing data-oriented overlay networks Presented by: Anwitaman Datta Joint work with Karl Aberer, Manfred Hauswirth, Roman Schmidt Ecole Polytechnique Fédérale de Lausanne (EPFL) Patrons: NCCR-MICS: www.mics.ch/ Swiss National Centres of Competence in Research Mobile Information & Communication Systems Evergrow: www.evergrow.org/ EC FP6, IST priority “Complex System Research” Contract no. 001935 (FET-IP) Ever-growing global scale-free networks, their provisioning, repair and unique functions.

  2. Structured overlays ♫ Associate each peer with some part of the load, i.e., a partition of the key-space ♪e.g. as in Distributed Hash Tables (DHT) ♫ Provide an efficient routing mechanism to locate peer responsible for a particular part of the key-space ♪ Various choice of topology possible

  3. Structured overlay maintenance ♫ Dynamics ♪Churn: Peers Join/Leave ♪New data inserted ♫ Standard maintenance mechanisms ♪ Correspond to updating database index ♪ Traditionally: Overlay evolution has been studied for incremental peer population Challenge #1: Fast construction of structured overlay from scratch

  4. Overlays for data-oriented applications ♫Hash Tables give constant time look-ups ♪ At the cost of losing ordering information ♪ DHTs need log(n) network hops ♫ Can we preserve (semantic) ordering information? ♪ Skewed load-distribution Challenge #2: The structured overlay should deal with arbitrary skew of load

  5. Toy example: Distributing skewed load 1 3 8 2 4 5 7 6 0 1 Key-space Load-distribution

  6. A globally coordinated recursive bisection approach 1 2 3 8 6 4 5 7 ♫Key-space can be divided in two partitions ♪ Assign peers proportional to the load in the two sub-partitions 0 1 Load-distribution

  7. A globally coordinated recursive bisection approach 2 6 8 1 5 4 7 3 ♫ Recursively repeat the process to repartition the sub-partitions 0 1 Load-distribution

  8. A globally coordinated recursive bisection approach 2 6 1 7 4 3 5 8 ♫ Partitioning of the key-space s.t. there is equal load in each partition ♪ Uniform replication of the partitions ♪ Important for fault-tolerance ♫ Note: A novel and general load-balancing problem. 1 0 Load-distribution

  9. Lessons from the globally coordinated algorithm Achieves an approximate load-balance. ♫ The intermediate partitions may be such that they can not be perfectly repartitioned. ♪There’s a fundamental limitation with any bisection based approach, as well as for any fixed key-space partitioned overlay network. ♫ Limit of dealing with load skews ♫ Nonetheless practical ♪ For realistic load-skews and peer populations

  10. pid 1 3 3 1 * * 0: 1 Routing table 1: 3 Keys 000,010,100 000,010,001 101,100 101,001 Legend (only part of the prefix is shown) 1 step: Distributed proportional partitioning - for overlay construction Randominteraction partitioning ♫ Given: ♪ A mechanism to meet other random peers ♪ A parameterp for partitioning the space ♫Proportional partitioning: Peers partition proportional to the load distribution ♪ In a ratio p:1-p ♪ Lets say: we call the sub-partitions as 0 and 1 ♫Referential integrity: Obtain reference to the other partition ♪ Needed to enable overlay routing ♫Sorting the load/keys: Peers exchange the locally stored keys in order to store only keys for its own partition. 0 1

  11. Heuristic 1: Autonomous partitioning (AUT) ♫ Make a priori probabilistic decision (parameterized by p) for a sub-partition ♪ proportionality constraint automatically met ♫ Find a peer from the other partition ♪ In order to meet referential integrity constraint ♫ Markovian asymptotic analysis of the process (for p = 0.5) ♪2 log(2) interactions (on an average) per peer

  12. Heuristic 2: Eager partitioning (for p = 0.5) ♫Undecided peers initiate contact with other random peer ♪If contacted peer is also undecided, contacting and contacted peers decide for different partitions (Balanced split) ♪If contacted peer has already decided, contacting peer decides for the other partition (Unblanced split) ♫Markovian asymptotic analysis of the process (for p = 0.5) ♪log(2) interactions (on an average) per peer ♫AUT is relatively inefficient ♪AUT wastes interactions in order to find a suitable peer Challenge: Can we have a strategy which works for all values of p, and is as efficient as eager partitioning when p = 0.5?

  13. AEP: Adaptive eager partitioning (w.l.g, p ≤ 0.5) ♫ Undecided peers initiate contact with other random peers ♪ If contacted peer is also undecided, perform Balanced split with probability: ♪ Since we need more peers (a fraction of 1-p ) in sub-partition 1 ♪ If the contacted peer has already decided for 0, contacting peer decides for 1 ♪ If the contacted peer has already decided for 1, contacting peer decides for 0 with a probability: ♪ 1 otherwise, since we need more peers in sub-partition 1

  14. Adaptive eager partitioning: choice of parameters ♫ Markovian analysis of the interactions ♪ Parameterized equations for & ♫0 ≤ p ≤ 1-log(2) ♪ ♫1-log(2) ≤ p ≤ 0.5 ♪

  15. AEP: Without global knowledge of p ♫ If we only have local estimates of p ♪ Error analysis: What’s the distribution of the estimates, and how does it affect the partitioning process? ♪ Introduces systematic skew ♪ Favors larger partition ♪ Compensating the skew

  16. COR: Skew compensated for AEP

  17. Algorithmic Issues: Overlay Construction ♫ Initiating the indexing process ♫ Synchronizing and terminating the process ♪ Synchronizing replicas ♫ Complexity ♪ Latency: O(log(n)2) - linear for sequential processes ♪ Communication: O(n.log(n)2)- same as in sequential processes

  18. Simulation results ♫ Discrete time simulation ♪ Mathematica based proprietary simulator ♫ Workloads ♪ Uniform, Pareto, Normal, real text collection from IR apps. (EU project: Alvis) ♫ Evaluation ♪ Deviation w.r.to what is obtained by the globally coordinated algorithm ♪ Measured in terms of the Euclidian Distance

  19. Load-distributionU: UniformP: ParetoN: NormalA: Alvis IR proj. text Simulation results: How useful is the theory? Theory vs. Heuristic (256 peers) deviation Load distribution

  20. Expts: Population & Load distribution Load-distributionU: UniformP: Pareto N: Normal Quality of load-balancing w.r.to peer population Peer populations 256 512 1024 deviation

  21. Expts: Population & Load distribution Scalability interactions Interactions required per peer for overlay construction 256 512 1024 Load-distributionU: UniformP: ParetoN: NormalA: Alvis IR proj. text

  22. "All models are wrong, but some are useful." - George E.P. Box From theory to practice: PlanetLab experiments Structured overlay construction Experiments evaluating search performance Bootstrap the peers and form an unstructured network Churn ♫ PlanetLab Testbed ♪ 400+ computers spread over various organizations and continents (www.planet-lab.org) ♫ Java implementation integrated with P-Grid ♪ P-Grid is a full-fledged P2P software (www.p-grid.org) ♫ Workload ♪ Text from IR applications studied under EU project Alvis (www.alvis.info) Simulation vs. Expts deviation peers Expt period

  23. Overlay operational phase Bandwidth consumption Overlay construction phase ♪ Construction process involves sorting keys. ♪ Initially it has higher bandwidth requirement. ♪ (Later) In operational phase, the queries dominate the bandwidth consumption. Expt period

  24. Churn No churn query latency Overlay performance ♪ Overlay construction was complete and peers discovered all their replicas ♪ Plots show absolute query latency ♪ In terms of overlay hops, experiments match theory ♪ Churn leads to larger deviation, but 95% to 100% success rate Expt period

  25. Related work ♫ Mostly sequential construction ♪ Recent work on fast overlay construction [SPAA 2005] ♪ Does not deal with load-balancing ♫ Load-balancing ♪ Mostly addresses uniform load-distribution case ♪ Some work on skewed loads [e.g., VLDB 2004] ♪ Incremental load/peer population changes ♪ No dynamic adaptation of replication

  26. www.p-grid.org Java implementation source-code available for download ♫lso: Range query paper@ IEEE P2P 2005

More Related