200 likes | 287 Views
Performance Evaluation of a Communication Round over the Internet. Omar Bakr Idit Keidar MIT MIT/Technion PODC 2002. Communication Round. Exchange of information from all hosts to all hosts Part of many distributed algorithms, systems
E N D
Performance Evaluation of a Communication Round over the Internet Omar Bakr Idit Keidar MIT MIT/Technion PODC 2002
Communication Round • Exchange of information from all hosts to all hosts • Part of many distributed algorithms, systems • consensus, atomic commit, replication, ...
Common Metric for Evaluating Algorithms • Number of rounds (or steps) they require
Questions • What is the best way to implement a communication round over the Internet • decentralized vs. centralized • How long is a communication round over the Internet?
Prediction is Hard • Internet is unpredictable, diverse, … • Different answers for different topologies, different times • Different performance metrics • local running time one host is engaged in algorithm • overall running time from when first host starts to when last host finishes
“Communication Round” Primitive • Initiated by some host • Propagates data from every host to every other host connected to it
All-to-all Leader Secondary Leader Example Implementations
Experiment I • 10 hosts: Taiwan, Korea, US academia, ISPs • TCP/IP (connections always up) • Algorithms: • All-to-all • Leader (initiator) • Secondary leader (not initiator) periodically initiated at each host - 650 times over 3.5 days
Computing Overall Running Time • Elapsed time from initiation (at initiator) until all hosts terminate • Requires estimating clock differences • Clocks not synchronized, drift • We compute difference over short intervals • Compute 3 different ways • Achieve accuracy within 20 ms. on 90% of runs
All-to-all: 2 Leader: 3 Secondary Leader: 4 Teaser: Comparing Performance Based on Number of Steps
150+240 = 390 150+150+150 = 450 Predicting Overall Runnig Times From MIT • Ping-measured latencies (IP): • Longest link latency 240 milliseconds • Longest link to MIT 150 milliseconds
Measured Running Times Runs Initiated at MIT Running times in milliseconds
What’s Going On? • Loss rates on two links are very high • 42% and 37% • Taiwan to two ISPs in the US • Loss rates on other links up to 8% • Upon loss, TCP’s timeout is big • More than round-trip-time • All-to-all sends messages on lossy links • Often delayed by loss
All-to-all overall local Leader overall local Sec. Leader overall local Average (runs under 2 sec) 866 645 1120 844 679 607 % runs over 2 seconds 54% 24% 64% 43% 13% 7% Running Times Runs Initiated at Taiwan Running times in milliseconds
Taiwan MIT Hosts’s with bad links to Taiwan Leader Secondary Leader What’s Going On? Good link Lossy link Other Hosts All-to-all
Experiment II: Removing Taiwan • Overall running times much better • For every initiator and algorithm, less than 10% over 2 seconds (as opposed to 55% previously) • All-to-all overall still worse than others! • either Leader or Secondary Leader best, depending on initiator • loss rates of 2% - 8% are not negligible • all-to-all sends O(n2) messages; suffers • But, all-to-all has best local running times
Probability of Delay due to Loss • If all links would have same latency • assume 1% loss on all links; 10 hosts (n=10) • Leader sends 3(n-1) = 27 messages • probability of at least one loss: 1 -.9927 »24% • All-2-all sends n(n-1) = 90 messages • probability of at least one loss: 1 -.9990 »60% • In reality, links don’t have same latency • only loss on long links matters
Conclusions • Message loss causes high variation in TCP link latencies • latency distribution has high variance, heavy tail • Latency distribution determines expected time for receiving O(n) concurrent messages • Secondary leader helps • No triangle inequality, especially for loss • Different for overall vs. local running times • Number of rounds/steps not sufficient metric • One-to-all and all-to-all have different costs