Federated Distributed Systems: Concepts of Distributed Systems (1)

1. Federated Distributed Systems:Concepts of Distributed Systems (1) Anda Iamnitchi Fall 2005, CIS 6930

2. Outline • Formal definition • Characterizing distributed systems • Issues unique to distributed systems • Modeling distributed systems • Complexity measures • Time and synchronization • Cristian’s, Lamport’s, NTP, Berkeley, etc. • (Next time: Impossibility Results) Concepts of Distributed Systems (CIS 6930)

3. Distributed System: Definition A set of autonomous computing nodes that cooperate to achieve a well defined goal • Compared to Centralized systems, where at most one autonomous activity is carried on at any single moment: • Distributed systems may have many autonomous computing nodes active in the system at any time. Concepts of Distributed Systems (CIS 6930)

4. Characterizing Distributed Systems Size • 8-node cluster vs. Gnutella Geographical area covered • 8-node local cluster vs. the seeds and leeches of a torrent in BitTorrent Shared memory vs. message passing • Message passing: • Point-to-point • Broadcast communication (radio, Ethernet) Concepts of Distributed Systems (CIS 6930)

5. Characterizing Distributed Systems Coupling level: • Tightly coupled systems (e.g., parallel machines): • Tight synchrony • Shared memory • Fast and reliable communication • Exploit processor cooperation • Wide-area communication network: • More independent processes • Less synchronized communication • Less frequent communication • Limited cooperation, more individual goals Concepts of Distributed Systems (CIS 6930)

6. Issues Unique to Distributed Computing • Communication • Has a cost • Incomplete Knowledge • About the other participants • About the stage of their processing • About the global environment (communication delays, failures, network partitioning, etc.) • Coping with Failures • Timing and Synchrony • Synchronous vs. asynchronous systems Concepts of Distributed Systems (CIS 6930)

7. Synchronous Systems: Definition • Assumption: All link delays are bounded • Each processor keeps a local clock whose pulses must satisfy the following: • Msg sent at pulse p is received by p+1 • Consequences: • It seems the system is driven by a global clock • One can assume that the system evolves in lock-steps (rounds of computation) • Timeouts can be used for failure detection • Silence can be used for communication Concepts of Distributed Systems (CIS 6930)

8. Synchronous Distributed System Rounds: • send messages • receive messages sent in round 1 • change state (processing) Time Concepts of Distributed Systems (CIS 6930)

9. Asynchronous Distributed System • Link delays are finite but unpredictable • Consequences: • Impossible to tell whether a message has been delayed or has been lost • Impossible to rely on the order of message arrivals to infer the order of computational events • Asynchronous distributed systems are inherently nondeterministic • Because messages are delivered in arbitrary order • There are also partially synchronous distributed systems (see N. Lynch 96) Concepts of Distributed Systems (CIS 6930)

10. Algorithmic and Programming Difficulties • Model: undirected graph, weighted or not • Protocol: local program executed by a vertex in the graph • Distributed algorithm: n protocols • Synchronous system: the protocol specifies what happens at every step • Asynchronous system: the protocol is described as event-driven Concepts of Distributed Systems (CIS 6930)

11. Complexity Measures • Time complexity: • Defined as worst case scenario. • Synchronous systems? • Asynchronous systems? • Space complexity • Message complexity • Weighted communication cost (takes into account link capacity) Concepts of Distributed Systems (CIS 6930)

12. Distributed Systems in This Course • Asynchronous • Wide-area (most often) • Large scale (e.g., peer-to-peer networks) • Based on point-to-point message passing (most often) Concepts of Distributed Systems (CIS 6930)

13. But We Need Synchronization! • Example 1: Two sharpshooters in a multiplayer online game kill the same target. Which one gets the points? • Example 2: make • Can all clocks in a distributed system be synchronized? Concepts of Distributed Systems (CIS 6930)

14. Clock Synchronization Anda Iamnitchi’s adaptation of REK’s adaptation of Claypool’s adaptation of Tanenbaum’s Distributed Systems Chapter 5

15. Physical Clocks • It is impossible to guarantee that crystals in different computers all run at exactly the same frequency. This difference in time values is clock skew. • “Exact” time was computed by astronomers • The difference between two transits of the sun is termed a solar day. Divide a solar day by 24*60*60 yields a solar second. • However, the earth is slowing! (35 days less in a year over 300 million years) • There are also short-term variations caused by turbulence deep in the earth’s core. • A large number of days (n) were used used to the average day length, then dividing by 86,400 to determine the mean solar second. Concepts of Distributed Systems (CIS 6930)

16. Physical Clocks Computation of the mean solar day. Concepts of Distributed Systems (CIS 6930)

17. Physical Clocks • Physicists take over from astronomers and count the transitions of cesium 133 atom • 9,192,631,770 cesium transitions == 1 solar second • Accuracy: • of from 2 to 3 parts in 10 to the 14th, i.e. 0.00000000000002 Hz • or 2 nanoseconds per day • or one second in 1,400,000 years • 50 International labs have cesium 133 clocks. • The Bureau Internationale de l’Heure (BIH) averages reported clock ticks to produce the International Atomic Time (TAI). • The TAI is mean number of ticks of cesium 133 clocks since midnight on January 1, 1958 divided by 9,192,631,770 . Concepts of Distributed Systems (CIS 6930)

18. Physical Clocks • To adjust for lengthening of mean solar day, leap seconds are used to translate TAI into Universal Coordinated Time (UTC). • UTC is broadcast by NIST from Fort Collins, Colorado over shortwave radio station WWV. WWV broadcasts a short pulses at the start of each UTC second. [accuracy 10 msec.] • GEOS (Geostationary Environment Operational Satellite) also offer UTC service. [accuracy 0.5 msec.] Concepts of Distributed Systems (CIS 6930)

19. Clock Synchronization Algorithms • Computer timers go off H times/sec, and increment the count of ticks (interrupts) since an agreed upon time in the past. • This clock value is C. • Using UTC time, the value of clock on machine p is Cp(t). • For a perfect time, Cp(t) = t and dC/dt = 1. • For an ideal timer, H =60, should generate 216,000 ticks per hour. Concepts of Distributed Systems (CIS 6930)

20. Clock Synchronization Algorithms • But typical errors, 10–5, so the range of ticks per second will vary from 215,998 to 216,002. • Manufacturer specs can give you the maximum drift rate (). • Every t seconds, the worst case drift between two clocks will be at most 2t. • To guarantee two clocks never differ by more than , the clocks must re-synchronize every /2 seconds using one of the various clock synchronization algorithms. Concepts of Distributed Systems (CIS 6930)

21. Clock Synchronization Algorithms • Centralized Algorithms • Cristian’s Algorithm (1989) • Berkeley Algorithm (1989) • Decentralized Algorithms • Averaging Algorithms (e.g. NTP) • Multiple External Time Sources Concepts of Distributed Systems (CIS 6930)

22. Cristian's Algorithm • Assume one machine (the time server) has a WWV receiver and all other machines are to stay synchronized with it. • Every /2seconds, each machine sends a message to the time server asking for the current time. • Time server responds with message containing current time, CUTC. Concepts of Distributed Systems (CIS 6930)

23. Cristian's Algorithm Getting the current time from a time server Concepts of Distributed Systems (CIS 6930)

24. Cristian's Algorithm • A major problem – the client clock is fast  arriving value of CUTC will be smaller than client’s current time, C. • What to do? • One needs to gradually slow down client clock by adding less time per tick. Concepts of Distributed Systems (CIS 6930)

25. Cristian’s Algorithm • Minor problem – the one-way delay from the server to client is “significant” and may vary considerably. • What to do? • Measure this delay and add it to CUTC. • The best estimate of delay is (T1 – T0)/2. • In cases when T1 – T0is above a threshold, then ignore the measurement. {outliers} • Can subtract off I (the server interrupt handling time). • Can use average delay measurement or relative latency (shortest recorded delay). • Caveat • RTT estimated by the client using its own clock, thus affected by client’s clock drift • But this effect small if clock drift small relative to real RTT value Concepts of Distributed Systems (CIS 6930)

26. The Berkeley Algorithm • The time daemon asks all the other machines for their clock values. • The machines answer and the time daemon computes the average. • The time daemon tells everyone how to adjust their clock. Concepts of Distributed Systems (CIS 6930)

27. Averaging Algorithms • Every R seconds, each machine broadcasts its current time. • The local machine collects all other broadcast time samples during some time interval, S. • The simple algorithm: the new local time is set as the average of the value received from all other machines. Concepts of Distributed Systems (CIS 6930)

28. Averaging Algorithms • A slightly more sophisticated algorithm: Discard the m highest and m lowest to reduce the effect of a set of faulty clocks. • Another improved algorithm: Correct each message by adding to the received time an estimate of the propagation time from the ith source. • extra probe messages are needed to use this scheme. • One of the most widely used algorithms in the Internet is the Network Time Protocol (NTP). • Achieves worldwide accuracy in the range of 1-50 msec. Concepts of Distributed Systems (CIS 6930)

29. The Network Time Protocol (NTP) • Uses a network of time servers to synchronize all processors on a net. • Time servers are connected by a synchronization subnet tree. The root is adjusted directly . Each node synchronizes its children nodes. Primary server, direct synch. 1 Secondary servers, synched by the primary server 2 2 2 Synched by secondary servers 3 3 3 3 3 3 Concepts of Distributed Systems (CIS 6930)

30. Server B T T i -2 i-1 Time m m' Time Server A T T i - 3 i Messages Exchanged Between a Pair of NTP Peers Each message bears timestamps of recent message events: the local time when the previous NTP message was sent and received, and the local time when the current message was transmitted. Concepts of Distributed Systems (CIS 6930)

31. Server B T T i -2 i-1 Time m m' Time Server A T T i - 3 i Theoretical Base for NTP • oi: estimate of the actual offset • between the two clocks • o: true offset of the clock at B • relative to that at A •  B’s clock = A’clock + o • t and t’: actual transmission times • for m and m’ • di: estimate of accuracy of oi ; • total transmission times for m • and m’; d=t+t’ Concepts of Distributed Systems (CIS 6930)

32. Logical Clocks • For a certain class of algorithms, it is the internal consistency of the clocksthat matters. The convention in these algorithms is to speak of logical clocks. • Lamport [1978] showed clock synchronization need not be absolute. What is important is that all processes agree on the order in which events occur. Concepts of Distributed Systems (CIS 6930)

33. Logical Clocks • The order of events is important: define relation Happens-Before (): • On the same process:a  b, if time(a) < time(b) • If p1 sends m to p2:send(m)  receive(m) • Ifa  b and b  cthena  c • Lamport’s Algorithm uses this relationship to provide a partial ordering of events: • All processes use a counter (clock) with initial value of zero • The counter is incremented by and assigned to each event, as its timestamp. • A send (message) event carries its timestamp • For a receive (message) event the counter is updated by Max(receiver-counter, message-timestamp) + 1 Concepts of Distributed Systems (CIS 6930)

34. (impossible) Lamport Timestamps • Each processes with own clock with different rates. • Lamport's algorithm corrects the clocks. • Can add machine ID to break ties Concepts of Distributed Systems (CIS 6930)

35. Limitation of Lamport’s algorithm • If e1  e2 then e1.TS < e2.TS • But it is not necessarily the case that if e1.TS < e2.TS then e1  e2 e1.TS < e8.TS, but is e1 e8 true? Given two events, we cannot say whether they are causally related from their timestamps Concepts of Distributed Systems (CIS 6930)

36. The Near Future • Next class: • Reading (Overlay Networks) • Discussion leader: Xu • In a week from today: • Tools for research in distributed systems • Chris on Macedon, Anda on Globus • Future lectures: • Impossibility results • Locality-sensitive algorithms Concepts of Distributed Systems (CIS 6930)