1 / 115

Coordination and Agreement

Coordination and Agreement. Ricardo Jiménez Peris, Marta Patiño Martínez Distributed Systems Laboratory Universidad Politécnica de Madrid (UPM) http://lsd.ls.fi.upm.es/lsd/lsd.htm La producción de este material ha sido financiada parcialmente por

Rita
Download Presentation

Coordination and Agreement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Coordination and Agreement Ricardo Jiménez Peris, Marta Patiño Martínez Distributed Systems Laboratory Universidad Politécnica de Madrid (UPM) http://lsd.ls.fi.upm.es/lsd/lsd.htm La producción de este material ha sido financiada parcialmente por Microsoft Research Cambridge (Award MS-2004-393) Lsd

  2. Index • Clocks and event ordering. • Reliable Multicast. • Consensus and related problems. • Distributed mutual exclusion. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  3. Bibliography • Chapter 11 Colouris. • 5 Mullender. • Chapters 2 and 7 Veríssimo. • M Raynal Distributed Algorithms and Protocols John Wiley 1988. • Lamport, L. Time, Clocks and the Ordering of Events in a Distributed System, Communications of the ACM, v. 21, n. 7, pp. 558-565, jul., 1978. • R. Vitenberg, I. Keidar, G. V. Chockler, D. Dolev. Group Communication Specifications: A Comprehensive Study. ACM Computing Surveys. v33, n4, pp. 427-489. Dec. 2001. • X. Defago, A. Schiper, P. Urban. Total order broadcast and multicast algorithms: Taxonomy and survey. ACM Computing Surveys, v. 36, n. 4, 2004, pp. 372-421. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  4. Bibliography • Birman, K. P. and R. Van Renesse, Reliable Distributed Computing with Isis Toolkit, IEEE Press, 1993. • R. Friedman and R. van Renesse, Strong and Weak Virtual Synchrony in Horus, CS Dep., Cornell Univ., TR95-1537, 1995. • Moser, L. E. and Y. Amir and Melliar-Smith, P. M. and Agarwal, D. A., Extended Virtual Synchrony, Proc. of the 14th IEEE Conf. on Distributed Computing Systems,pp. 56-65, june, 1994. • Özalp Babaoğlu, Alberto Bartoli, Gianluca Dini. Enriched View Synchrony: A Programming Paradigm for Partitionable Asynchronous Distributed Systems. IEEE Transactions on Computers. V. 46, n. 6, June 1997. pp. 642 - 658. 1997. • Jeremy Sussman Idit Keidar Keith Marzullo. Optimistic Virtual Synchrony. Proc. of the 19th IEEE Symp. on Reliable Distributed Systems (SRDS'00). 2000. • A. Schiper and A. Sandoz. Uniform Reliable Multicast in a Virtually Synchronous Environment. In Proc. of IEEE Int. Conf. on Distributed Systems (ICDCS), pages 561–568, 1993. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  5. Bibliography • Yair Amir, Claudiu Danilov, Michal Miskin-Amir, John Schultz, and Jonathan Stanton. The Spread Toolkit: Architecture and Performance. Johns Hopkins University, Center for Networking and Distributed Systems (CNDS) Technical report CNDS-2004-1. • Hayden, M., The Ensemble System, TR-98-1662, Department of Computer Science. Cornell University, jan., 1998. • Luís Rodrigues and Paulo Veríssimo. xAMp: a Multi-primitive Group Communications Service. October 1992. Proceedings of the 11th Symposium on Reliable Distributed Systems, Houston, Texas, USA. • Ezhilchelvan, P.D., Macêdo, R.A. and Shrivastava, S.K., "Newtop: A Fault-Tolerant Group Communication Protocol“. In Proc. of the 15th IEEE Int. Conf. on Distributed Computing Systems (ICDCS '95), 1995, pp. 296-306. • Moser, L.E. et a., Totem: A Fault-Tolerant Multicast Group Communication System, Communications of the ACM, v. 39, n. 4, pp. 54-63, apr. 1996. • D.A. Agarwal, L.E. Moser, P. M. Melliar-Smith, R.K. Budhia, The Totem Multiple-Ring Ordering and Topology Maintenance Protocol, Trans. on Computer Systems, v. 16, n. 2, pp. 93-132, 1998. • D. Dolev and D. Malki, The Transis Approach to High Availability Cluster Communication, Communications of the ACM, v. 39, n. 4, pp. 64-70, 1996. • JGroups: A Toolkit for Reliable Multicast Communication. http://www.jgroups.org. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  6. Bibliography • Michael J. Fischer, Nancy A. Lynch and Michael S. Paterson, "Impossibility of Distributed Consensus with One Faulty Process," Journal of the ACM, April 1985, 32(2):374-382. • Leslie Lamport, Robert Shostak, Marshall Pease. The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems (TOPLAS), n. 3, July 1982. pp. 382 - 401. 1982. • R. Guerraoui. Revisiting the Relationship between Non-Blocking Atomic Commitment and Consensus. In Proc. of Int. Workshop on Distributed Algorithms (WDAG), 1995. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  7. Bibliography • AGRAWAL, D. AND ABBADI, A. E. 1992. The Generalized Tree Quorum Protocol: An Efficient Approach for Managing Replicated Data. ACM Transactions on Database Systems 17, 4, 689–717. • AMIR, Y. AND WOOL, A. 1998. Optimal Availability Quorums Systems: Theory and Practice. Information Processing Letters 65, 5, 223–228. • CHEUNG, S. Y., AHAMAD, M., AND AMMAR, M. H. 1990. The grid protocol: a high performance scheme for maintaining replicated data. In Proc. of Int. Conf. on Data Engineering (ICDE). Los Angeles, USA, 438–445. • KUMAR, A. 1991. Hierarchical Quorum Consensus: A New Algorithm for Managing Replicated Data. IEEE Transactions on Computers 40, 9 (Sept.), 996–1004. • MAEKAWA, M. 1985. A sqrt(n) Algorithm for Mutual Exclusion in Decentralized Systems. ACM Transactions on Computer Systems 3, 2, 145–159. • NAOR, M. AND WOOL, A. 1998. The Load, Capacity, and Availability of Quorum Systems. SIAM Journal of Computing 27, 2 (Apr.), 423–447. • PELEG, D. AND WOOL, A. 1994. Crumbling Walls: A Class of Practical and Efficient Quorum Systems. Distributed Computing 10, 2, 87–97. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  8. Bibliography • THOMAS, R. H. 1979. A Majority Consensus Approach to Concurrency Control for Multiple Copy Databases. ACM Transactions on Database Systems 4, 9 (June), 180–209. • PELEG, D. AND WOOL, A. 1995. The Availability of Quorum Systems. Information and Computation 123, 2, pp. 210–223. • GIFFORD, D. K. 1979. Weighted Voting for Replicated Data. 7th ACM Symp. on Operating Systems, 150–162. • ERDOS, P. AND LOVASZ, L. 1975. Problems and results on 3-chromatic hypergraphs and some related questions. In Infinite and finite sets. Colloq. Math. Soc. J´anos Bolyai 10, 609–627. • LOVASZ, L. 1973. Coverings and colorings in hypergraphs. In Proc. of 4th South-eastern Conf. Combinatorics, Graph Theory and Computing. 3–12. • R. Jiménez-Peris, M. Patiño-Martínez, G. Alonso, and B. Kemme. Are Quorums an Alternative for Data Replication? ACM Transactions on Database Systems, Vol. 28, N. 3, Sept. 2003, pp. 257-294, ACM Press. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  9. Physical Clocks and Event Ordering • One alternative to order events in a distributed system lies in labeling them with timestamps obtained from a local clock. • The main obstacle is due to the lack of perfect synchronization among clocks that might even have different drifts. • An ordinary quartz clock has a drift of 1 second every 11-12 days. (10-6 seconds/second). • A high precision quartz clock has a drift of 10-7-10-8 seconds/second. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  10. Physical Clocks and Event Ordering • Coordinated Universal Time (UTC): • The international time is based on an atomic clock with drifts in the order of 10-13 seconds/second. • UTC is the time international standard. • It is based on atomic time, but it is adjusted ocassionally with the astronomic time. • It is multicast via radio stations and GPS. • A computer with a radio receiver or a GPS can synchronize its local clock via these signals. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  11. Physical Clocks and Event Ordering • Radio signals have a precision in the order of 0.1-10 ms. • GPS has a precision of around 1 µs. • There are time synchronization protocols with very good precisions: • Network Time Protocol (NTP) attains precisions of tens of ms in Internet and 1 ms in LANs. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  12. Logical Clocks and Causal Event Ordering • In a distributed system three events are distinguished in a communication between two nodes: sending, reception and delivery. • Sending is the physical submission of the message frin the origin. • Reception is the physical arrival of the message to the destination. • The delivery is when the message is given to the application layer. • Between reception and delivery a non-negligible amount of time can pass. • These events are used to establish the logical ordering of events. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  13. Logical Clocks and Causal Event Ordering • [Lamport cacm78] proposed the following causal ordering of events: • Two events, e1 and e2, executed sequentially by a process are casually ordered: • The sending of a message precedes its delivery. • Causal ordering is transitive: • Logical clocks (as opposed to physical clocks) are proposed to label events with a timestamp or logical time. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  14. Logical Clocks and Causal Event Ordering • Each process p has assocaited a logical clock (counter), lc. • Initially, it is 0 at all processes. • lc:= 0. • Each time a message is sent from p, its clock is incremented by1 and the message is labeled with it: • lc:= lc+1; ts:= lc. • When a message is received by p, its new logical clock is obtained as adding 1 to the maximum between its local logical clock and the timestamp: • lc:= max(lc,ts)+1. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  15. Logical Clocks and Causal Event Ordering 1 2 p 1 a b m 1 4 5 3 p 2 e f g m 3 m 2 6 1 2 p 3 d h c • If an event precedes another event, ev1  ev2, then the first one has a lower logical clock than the second, lc(ev1) < lc(ev2). • The reverse might not be true. lc(b) < lc(e), but b  e does not hold. • Logical clocks do not capture concurrency. • By themselves are not sufficient to implement causal delivery of messages. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  16. Coordination and Agreement • Properties used to characterize the problems of coordination and agreement: • Termination: Liveness property, the agreement will take eventually take place. • Validity: Actions performed by correct processes eventually affect the rest of processes (e.g. a message multicast by a correct process is eventually delivered by the receiver processes). • Agreement: The decision taken by different correct processes fulfills certain property (e.g. all processes take the same decision). • Integrity: Properties that aims to avoid illogical solutions (e.g. a process might only take a decision previously proposed by some other process, or only messages that have been previously sent can be delivered). Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  17. Group Communication Systems (GCS) • Basic building block for developing fault tolerant distributed systems (replicated processes, replicated and distributed databases, load balancing, …). • GCS provide group multicast with different properties and a membership service. • The membership service keeps a list of alive and connected processes. • The updated membership, or view, is delivered to the application everytime a change is detected. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  18. Multicast • Multicast vs. Broadcast. • A multicast sends a message to a process group. • A broadcast sends a message to all the processes in the system. • Advantages: • Allows an efficient use of the physical communication buses (e.g. IP multicast over Ethernet). • The sender spends less resources. • Reliability and ordering properties are provided that are fundamental to build reliable distributed systems. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  19. Multicast • Group types: • Closed: only group members can multicast messages to the group. • Open: Any process can multicast messages to the group. • Events in the multicast of a message: • Sending. • Reception. • Delivery. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  20. Multicast • Reliability levels: • Non-reliable multicast (e.g. ip-multicast). • Reliable multicast. • Uniform multicast. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  21. IP Multicast • Based on IP. • Allows to send an IP packet to a set of nodes that are part of a multicast group. • Dynamic membership. • Open groups. • The multicast is performed sending a UDP datagram to an IP multicast address. • To receive messages the process should join the multicast group (s.joinGroup(group)). • There are no reliability nor ordering guarantees (as with UDP unicast). Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  22. Reliable Multicast • Properties: • Integrity: a correct process delivers a message m at most once. Additionally, it only delivers messages that have been previously multicast by other processes. • Validity: If a correct process p multicasts a message m, m will be eventually delivered by p. • Agreement: If a correct process delivers a message m, the all correct processes will eventually deliver m. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  23. Uniform Multicast • The previous definition only refers to guarantees provided by correct processes. • If we are interested in constraining the behavior of incorrect processes, uniformity is needed: • Uniform agreement: If a process, correct or not, delivers a message m, then all correct processes eventually deliver m. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  24. Kinds of Ordering • No ordering. • FIFO: Messages from the same sender are delivered in the order they were sent: • Causal: • FIFO • If two messages are casually related they are delivered in causal order: • Total: Messages are delivered in the same order at all group members. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  25. m3 and m1 do not respect causality Causal Order p1 p2 p3 send(m1) → send(m3) delivery (m3) → delivery (m1) The latter does not fulfil causality m2 m3 m1 Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  26. m1 and m2 do not respect causal order in p3 send(m1) →delivery (m1) → send(m2) and delivery (m2) → delivery(m1) Causal Order p 1 m1 m1 m2 p 2 m2 p 3 Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  27. Views • Safety properties: • If a process installs a view, the process belongs to that view. • View identifiers are mononically increasing. • A membership service can be primary component or partitionable. • Under a primary component membership all the views are totally ordered. • Under a partitionable membership all views are partially ordered and multiple disjoint views might coexist concurrently. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  28. Views • A primary component service guarantees that there is always at least one process that transits from the current view to the next one. • In partitionable systems, processes are informed whether they belong or not to a primary view (Horus). • The sending view delivery property guarantees that messages are delivered in the view they were sent. (Isis, Totem). Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  29. Views • Implementing the sending view delivery property implies to block message multicasting during a view change. • To attain this the GCS sends a blocking event to all participants. • The application replies with a flush, indicating that it will not send any further messages till the new view is delivered (Horus). Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  30. Views • The same view delivery property indicates that if two processes deliver the same message, both do so in the same view (Transis). • Unlike sending view delivery, same view delivery does not require blocking the sending of messages during the view change. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  31. Virtual Synchrony • Virtual Synchrony in GCS guarantees that if two processes transit from a view, Vk, to the next view, Vl, both processes have delivered the same set of messages in viewVk. • This property was defined by Isis and it is provided by all GCS. • Strong virtual synchrony: Apart from virtual synchrony it also fulfils sending view delivery. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  32. Virtual Synchrony • Virtual synchrony is especially important to determine at which point the state can be transferred in a consistent way to new processes joining the group. • In this way, it is known which messages have been delivered and are included in the transferred state and guarantees that latter messages will be delivered by all new members (and none of the former messages). • There are many extensions to virtual synchrony: weak, optimistic, extended, enriched, … Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  33. Virtual Synchrony without “sending view delivery” P1 P2 P3 m1 m2 New view Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  34. Virtual Synchrony without “same view delivery” P1 P2 P3 m1 m2 New view Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  35. Multicast Protocols • Reliable Multicast: • The sender multicasts the message to the receiver group. • The sener waits for the reception of acks from all members of the receiver group. • If any ack is missing, the message is resent periodically till all acks are obtained or till processes with missing acks are suspected of having failed. • If the sender fails, one of the receivers of the message upon detecting the sender failure, will resend the message till it is received by all correct processes. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  36. Multicast Protocols • Waiting for the ack of every message before sending the next message generate high latencies. • An alternative is to use a sliding window protocol. • A maximum of n messages are sent without having received the ack. • When the ack of the first message is received, the window is slid. • The sliding window also serves as a flow control mechanism. • The sending of acks in multicast implies the reception of many messages at the sender (ack implosion). • It can be avoided by piggybacking acks in other multicast messages. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  37. Multicast Protocols • FIFO Order: • Each process has associated a counter. • Messages are labeled with an identifier of the sender and the current value of the counter. • Each time a message is sent, the local counter is incremented. • A message is only delivered is all previous messages have been delivered. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  38. Multicast Protocols: FIFO • Negative acknowledgements (nacks): • Each sender maintains a counter of sent messages that associates to each message sent. • Each receiver maintains for each sender the highest counter for which all messages with that counter or lower have been received. • When a message is sent, the counter is piggybacked. • When a message is received, if the receiver counter is consecutive with the message counter, the message is delivered and the receiver counter incremented. • Otherwise, it means messages have been lost and they are requested to the sender via a nack message and the message stored for later delivery. • If the message counter is lower than the receiver counter, it is a duplicate message and discarded. • If a node does not send message, it sends heartbeat messages periodically indicating the counter of the lattest sent message. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  39. Multicast Protocols: Causal Order p1 p2 p3 1 send(m1) → send(m3) (m3) → (m1) 2 m2 3 4 m3 5 m3 and m1do not fulfill causality. Logical clocks do not provide sufficient information m1 6 Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  40. Multicast Protocols: Causal Order • A vectorial clock, vci, of a process i is a vector with as many components as processes in the group. • Vectorial clocks are compared component by component. vck≤ vcj , if for all i, vck[i] ≤ vcj[i] • vck<vcj , sivck≤ vcj and exists i, vck[i] < vcj[i] • Vectorial clocks represent the causality relationship mi→mj,sivc(mi) <vc(mj). The reverse is also true. • Two events, mimj ,are concurrent, mi||mj, if neither vc(mi) ≤ vc(mj), nor vc(mj) ≤vc(mi). Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  41. Multicast Protocols: Causal Order • Causal order (vector clocks): • Eachprocesspi has associatedvci. • Initiallyvci=(0,…,0) at allprocesses. • Each time pi sends a message, theicomponent of vciisincremented and themessagelabeledwithvci: • vci[i]=vci[i]+1; ts= vci Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  42. Multicast Protocols: Causal Order • A message from pklabeled with ts is delivered at pi if • That is, is pihas delivered all previous messages from previous to pk and all messages that pk had delivered before sending the message. • When a message is delivered at pk it obtains a new timestamp resulting from selecting for each component the maximum between the vck and the message timestamp, ts, components Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  43. Cannot be delivered due to 0+1=v1(2)+1 ≠ v2,m3(2) =2 Multicast Protocols: Causal Order m1 v1(1,0,0) v1(0,0,0) v1(1,0,0) v2(1,1,0) v2(1,2,0) m2 m3 v2(0,0,0) v2(1,0,0) v1(1,0,0) v2(1,1,0) v2(1,2,0) v3(0,0,0) v3(1,0,0) v3(1,1,0) v3(1,2,0) Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  44. Cannot be delivered due to 1=v2(3) >v1(3)=0 Multicast Protocols: Causal Order v1(0,1,0) v1(0,0,0) v2(0,2,1) v2(0,1,0) m3 m1 v2(0,0,0) v2(0,1,0) v2(0,1,1) v3(0,1,1) v3(0,1,1) v2(0,1,0) v2(0,2,1) m2 v3(0,0,0) v3(0,1,0) v3(0,2,1) Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  45. Causal Order (concurrent messages) m1 v1(1,0,0) v1(1,1,0) v1(0,0,0) v1(1,0,0) v2(0,1,0) m2 v2(0,0,0) v1(1,0,0) v2(0,1,0) v3(0,0,0) v3(1,0,0) v3(1,1,0) Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  46. Multicast Protocols: Total Order • Where are orderedthemessages? • Sender: • Basedonhistory • Basedonprivileges (token-based) • Using a sequencer: • Fixed • Mobile • Receiver: • Agreementondestination Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  47. Total Order: basedonhistory • Use the Lamport relationship “happened before” to order messages with logical clocks. Given two messages m1 and m2, if rl(m1) < rl(m2), m1 will be delivered before m2. • The relationship “happened before” is a partial order. The senders have an order. If the messages are concurrent, the sender order is used. In this case, a message m1 is delivered before m2, if sender(m1) < sender(m2). • When a message, m, is sent, the logical clock is piggybacked. • A receiver delivers m when knows that it is not going to receive a message with an lc lower than m. Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  48. Total Order: basedonhistory Initialization: receivedp := Ø; { Messagesreceivedbyprocessp } deliveredp:= Ø ; { Messagesdeliveredbyprocessp } LCp[p1 . . . pn ] := {0 , . . . , 0 } {LCp[q]: logicalclock of processq, as seenbyprocessp } procedureTO-multicast(m) { ToTO-multicast a messagem } LCp[p] := LCp[p] + 1 ts(m) := LCp[p] send FIFO (m, ts(m)) toall Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  49. Total Order: history-based whenreceive (m, ts(m)) LCp[p] := max(ts(m), LCp[p]) + 1 LCp[sender(m)] := ts(m) receivedp := receivedpU {m} deliverable := Ø; foreachmessage m´ in receivedp \ deliveredpdo ifts(m´) ≤minq€ΠLCp[q] then deliverable := deliverableU {m´} deliverallmessages in deliverable, in increasingorder of (ts(m), sender(m)) deliveredp := deliveredpUdeliverable Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

  50. 1 (1,0) m1 cannot be delivered (1,2) 3 m1 can be delivered but not m2 M1 can be delivered (1,3) (4,3) m1 and m2 can be delivered Total Order: basedonhistory (0,0) m1 (0,0) 1 m2 Laboratorio de Sistemas Distribuidos, Universidad Politécnica de Madrid

More Related