Synchronization

Synchronization Chapter 5 Chapter 5  Synchronization 1

Synchronization • Multiple processes must not simultaneously access shared resource • Ordering may be important • Such as, msg 1 must come before msg 2 • Time • Absolute time vs relative time • May want one process to coordinate • Election algorithms Chapter 5  Synchronization 2

Synchronization • Special topics… • Distributed mutual exclusion • Protect shared resources from simultaneous access • Distributed transactions • Similar, but try to optimize access thru “advanced concurrency control” Chapter 5  Synchronization 3

What Time is It? • Easy to answer in a non-dist system • Spse A asks for time, then B • B’s time will be later than A’s • In dist system, this may not be true • Spse A checks time, then B • B’s time might not be later than A’s • That is, time on A and B might not agree • If time comes from a central location, network communication variation is a problem Chapter 5  Synchronization 4

What Time is It? • Why do we care about time? • Consider make example • Make used to compile and link multiple source files into one executable file • If file.o was last modified before file.c, then file.c must be recompiled • If file.o was last modified after file.c, then no need to recompile file.c • This breaks if time is not the same in distributed system Chapter 5  Synchronization 5

Clock Synchronization • Both machines have their own clock • Clocks differ by “2” • What will make do with output.c? • Oops! Chapter 5  Synchronization 6

Time • With single processor system • Doesn’t matter if time is incorrect • Relative time is what’s important • If more than one processor • Clock skew is inevitable • Multiple clock problems • How to synchronize with “real” clock? • How to synchronize clocks with each other? • But first we digress… Chapter 5  Synchronization 7

Physical Clocks • Time between 2 transits of the sun • Solar day • Solar second is 1/86400th solar day Chapter 5  Synchronization 8

Physical Clocks • Period of earth rotation not constant • Earth is slowing due to drag • Days are getting longer • Atomic clock invented 1948 • Official second is now • 9,192,631,770 transitions of cesium 133 • International Atomic Time (TAI) • Today, 86,400 TAI seconds is about 3 msec less than mean solar day! Chapter 5  Synchronization 9

Physical Clocks • Solar seconds are not of constant length • TAI seconds are of constant length • Leap seconds are used to keep in phase with sun • Add leap second when discrepancy > 800 msec • Otherwise noon would eventually be before breakfast  might cause riots! Chapter 5  Synchronization 10

Physical Clocks • TAI with leap seconds is known as • Universal Coordinated Time (UTC) • UTC replaces Greenwich Mean Time (GMT) • NIST operates radio WWV from Colorado • Sends out pulse at start of each UTC second • But only accurate to within 1 msec • Do to atmospheric effects, can vary by 10 msec • Some satellites offer similar service • In any case, must know relative position • To compensate for propagation delay Chapter 5  Synchronization 11

Clock Sync. Algorithms • Suppose one machine monitor WWV • How to keep other clocks in sync? • Let t be UTC time • Let Cp(t) be time on machine p • Ideally, want Cp(t) = t • We’ll be happy if dCp/dt = 1 Chapter 5  Synchronization 12

Clock Sync. Algorithms • Clocks drift • Suppose • One clock is slow and one is fast… • Drift apart at twice the drift rate Chapter 5  Synchronization 13

Clock Sync. Algorithms • Let Cp(t) be time on machine p • Ideally, want Cp(t) = t • Or dCp/dt = 1 • But processor clocks can drift • If maximum rate of drift is  • After t, two clocks could be 2  t apart • If you want clocks to differ by less than  • Must synchronize clocks every  / 2 seconds • How to synchronize? Chapter 5  Synchronization 14

Clock Sync. Algorithms • How to synchronize clocks? • Cristian’s algorithm • Pull protocol • Berkeley algorithm • Push protocol • Averaging algorithms • Decentralized approach • Network Time Protocol (NTP) • Multiple external time sources Chapter 5  Synchronization 15

Cristian's Algorithm • Spse time server has WWV time • Clients want to stay within  of others • Every  / 2 seconds or less… • Client asks time server for time • Somebody got an algorithm named after themselves for that? • See next slide Chapter 5  Synchronization 16

Cristian's Algorithm • What are the potential problems? • Time cannot run backwards • Takes (variable) time to get reply Chapter 5  Synchronization 17

Cristian's Algorithm • Time cannot run backwards • If clock is fast… • Increment time more slowly than usual • Must account for time to get reply • How to do this? • Educated guess! Roundtrip time divided by 2 • Account for time server takes to process, multiple roundtrip measurements, etc., etc. Chapter 5  Synchronization 18

Berkeley Algorithm • Cristian’s “algorithm” • Time server is passive • Berkeley algorithm • Time server is aggressive • Does not require server to know UTC • Server polls clients • Computes average time • Pushes result to clients Chapter 5  Synchronization 19

Berkeley Algorithm • Server asks others for their clock values • Machines answer • Server tells others how to adjust their clock Chapter 5  Synchronization 20

Averaging Algorithms • Cristian’s and Berkeley are centralized • Averaging (decentralized) approach… • All machines broadcast time • Everybody computes average • The usual refinements apply • When to broadcast? • Only practical on a LAN Chapter 5  Synchronization 21

Network Time Protocol • According to book, NTP uses • “advanced clock synchronization algorithms” • Accuracy range of 1 to 50 msec • But NTP is not very secure • NTP actually uses Marzullo’s Algorithm • Aka the Intersection Algorithm • Have a collection of times intervals • Example: time of 102 gives interval [8,12] Chapter 5  Synchronization 22

Network Time Protocol • Given collection of times intervals • Of the form [a,b] • Marzullo’s algorithm finds consistent interval • Efficient: linear in time and space • If no consistent interval, finds interval(s) consistent with the most sources • Marzullo takes center of resulting interval • Intersection Algorithm refines this • Use statistical info on confidence intervals • Selected time not necessarily midpoint Chapter 5  Synchronization 23

Multiple External Time Sources • Suppose very accurate time needed • Multiple UTC sources? • But these will not agree • So need to average (or similar) • Network delays • Processing delays, etc. • Not clear that this helps very much! Chapter 5  Synchronization 24

Use of Synchronized Clocks • Today, computers can be at or near UTC • How to make use of this? • To enforce “at most once delivery” • Traditional approach • Server keeps track of msg numbers • Checks list against incoming msg numbers • How long to keep list? What if server crashes? • Alternative is to use timestamps • We discuss other apps in later sections Chapter 5  Synchronization 25

Logical Clocks • Usually good enough to agree on time • Even if it’s not the actual time • Often sufficient to agree on order • Recall make example • Lamport time • Synchronize logical clocks • Vector timestamps • Extension of Lamport’s algorithm Chapter 5  Synchronization 26

Lamport Timestamps • “Happens before”: a  b • According to Tanenbaum: a  b if all processes agree that a came before b • Lamport actually defines “” as the “smallest” relation satisfying • If a occurs before b on same processor then a  b • If a == send, b == receive, a  b • Transitive: a  b and b  c implies a  c Chapter 5  Synchronization 27

Lamport Timestamps • “Happens before”: a  b • Does “happens before” equal “really happened before”? • If a and b are on same process and a occurs before b, then a  b • If a == msg sent, b == (same) msg received, then a  b • It takes time for message to be sent • If a b and b  a, msgs are concurrent / / Chapter 5  Synchronization 28

Lamport Timestamps • For event a, want timestamp C(a) • If a  b then C(a) < C(b) • C is a non-decreasing function • Time cannot go backwards! • Lamport’s solution • Each msg carries timestamp with it • If local time is less than timestamp, set local time to timestamp + 1 • Advance clock between any two events • Illustrated on next slide… Chapter 5  Synchronization 29

0 0 0 0 0 0 A A 10 10 6 8 6 8 20 20 12 16 12 16 B B 30 30 18 24 18 24 40 40 24 32 24 32 50 50 30 40 30 40 C C 60 60 36 48 36 48 70 70 42 56 42 61 D D 80 80 48 64 48 69 90 90 54 72 70 77 100 100 60 80 76 85 Lamport Timestamps • Three processes with different clocks • Lamport's algorithm corrects the clocks Chapter 5  Synchronization 30

Lamport Timestamps • Can also insure that no two events ever occur at exactly the same time • 40.1 for process 1 • 40.2 for process 2, etc. • With this refinement, we have a total ordering on all events in the system • If a  b on same process then C(a) < C(b) • If a == msg sent, b == msg received, then we have C(a) < C(b) • If a  b then C(a)  C(b) Chapter 5  Synchronization 31

Totally-Ordered Multicast • Consider replicated database • Spse replica in San Jose and in New York • Query goes to nearest copy • Updates are tricky • Must have updates in same order at replicas • For example: Interest calculation and deposit • For consistency, no “right” order • Just want updates to happen in same order • Correctness is a different story… Chapter 5  Synchronization 32

Non-Totally-Ordered Multicast • Assumptions • $1000 in acct, deposit is $1000, interest rate is 10% • On left, $2200, on right $2100 • Inconsistent! Deposit Interest Chapter 5  Synchronization 33

Totally-Ordered Multicast • Assume msgs received in order and no loss • Using Lamport timestamps… • Msgs timestamped with sender’s logical time • Multicast sent to all sites, including sender • Msgs go into local queue in timestamp order • Multicast ACK msgs (to yourself too) • Message only removed from queue if • It is at head of queue and • It has been ACKed • Does this work? See next slide… Chapter 5  Synchronization 34

Totally-Ordered Multicast • $1000 in acct, deposit is $1000, interest rate 10% • What happens in this case? Deposit Interest After 45… Deposit: 45 After 10… Interest: 10 30 0 Deposit Interest Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 45 10 60 20 75 45 ACK(I) ACK(D) 90 46 105 90 120 91 Chapter 5  Synchronization 35

Totally-Ordered Multicast • When is interest calculation done at each replica? • When is deposit made? Deposit Interest After 45… Deposit: 45 After 10… Interest: 10 30 0 Deposit Interest Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 45 10 60 20 75 45 ACK(I) ACK(D) 90 46 105 90 120 91 Chapter 5  Synchronization 36

8 9 1 2 3 P1 2 7 9 1 4 5 11 P2 10 3 4 1 P3 5 6 7 Scalar Timestamps • Scalar timestamps (such as Lamport timestamps) give total ordering using C(a) • But C(a) < C(b) does not mean that event a really happened before b • The “4” at P2 occurs before the “3” at P1 Chapter 5  Synchronization 37

Vector Timestamps • Lamport timestamps don’t reflect causality • Local events are causally ordered • Example: multicast news posting • Response might arrive before original posting • Vector timestamps do reflect causality • Must specify • Local data structures to represent logical time • Update mechanism/protocol • Tanenbaum’s description is confusing! Chapter 5  Synchronization 38

Vector Timestamps • Want vector timestamp such that • If VT(a) < VT(b) then a causally precedes b • Process Pi maintains vector Vi • Vi[i] is incremented for each event at i • Vi[j] is Pi’s current view of the number of events that have occurred at process Pj • Vi[i] is easy to maintain • Vi[j] is obtained from info sent with msgs • Each message includes vector timestamp Chapter 5  Synchronization 39

Vector Timestamps • Suppose Pj received msg m from Pi • Pi includes it’s vector timestamp, vt • Then Pj adjusts its values according to vt • Pj then knows the number of events on which m can depend • Tanenbaum claims… • Pj knows no. of messages it must receive before it has seen everything that m could depend on • Not true! Event  msg! Chapter 5  Synchronization 40

Vector Timestamps P1 P2 P3 Chapter 5  Synchronization 41

Vector Timestamp • Modified (useful) form of VT • Suppose Vi[i] counts msgs sent by Pi • Now consider multicast newsgroup • Suppose Pi post a message • Includes vector vt(a) • Suppose Pj posts a response • Includes vector vt(b) • Want vt(a) to reflect msgs known to Pi when a was sent, and similarly for vt(b) Chapter 5  Synchronization 42

Vector Timestamp • Let Vi[i] be number of msgs sent by Pi • Vi[j] = number of messages Pi received from Pj • Pi sends a with vt(a), later Pj sends b,vt(b) • Suppose Pk receives b before a • Pk waits to deliver msg b until • vt(b)[j] == Vk[j] + 1 • This is the next msg expected from Pj • vt(b)[i] <= Vk[i], all i  j • Ensures that Pk must have seen msg a Chapter 5  Synchronization 43

Vector Timestamp P1 P2 msg c P3 Queue msg a Queue msg b    c a b Chapter 5  Synchronization 44

Global State • Global state of distributed system • All local states plus msgs in transit • Definition of “state” can vary • Useful to know global state to • Know that computation is finished • Detect deadlock • How to record global state? • Distributed snapshot Chapter 5  Synchronization 45

Global State • Distributed snapshot • A consistent state “in which the system might have been” • For example, if Q received msg from P then must show that P sent the msg • P sent msg Q has not yet received is OK • Global state represented by a cut • Next slide… Chapter 5  Synchronization 46

Global State • Consistent cut • Inconsistent cut Chapter 5  Synchronization 47

Global State • Assume distributed system uses point-to-point unidirectional communication • Suppose P starts snapshot • P records its state • P sends “marker” to neighbors • When Q receives marker • First marker on any channel: Q records state • Append incoming messages from S until marker from S is received • Q is done when it has received marker on all incoming channels Chapter 5  Synchronization 48

Global State • This figure does not match algorithm! • See next few slides… Chapter 5  Synchronization 49

Global State • Consider the following example • Bank has 3 branches, A, B, C • Each branch connected to others by • Unidirectional point-to-point links • State consists of • Money in branch and… • …money in transit between branches Chapter 5  Synchronization 50

Synchronization