From Fast to Lightweight Atomic Memory in Large Scale Dynamic Distributed Systems

1. From Fast to LightweightAtomic Memory in Large Scale Dynamic Distributed Systems Vincent Gramoli IRISA – Université de Rennes 1, France vgramoli@irisa.fr From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

2. Goal(s) • Atomic Consistency • Fault-tolerance • Scalability • Fast: low operation latency • Lightweight: low communication cost From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

3. Model • Distributed System • A set of connected nodes, each uniquely identified. • Dynamic System • Each node leaves, crashes, or joins at any time. • A joining node has a new id. • Asynchronous Communication • Messages can be arbitrarily delayed. (Eventually synchronous for liveness requirements.) • Memory • Each object is replicated at n nodes (servers). • Each object is accessed (read/wrote) by some nodes (clients). From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

4. Atomic DSM • Roughly speaking: “All nodes see the consequence of any operation according to the time this op. occurred.” • More precisely they see them in a specific order: • Non-concurrent operations are ordered as they occur. • Write ops. are totally ordered. • Read ops. are ordered w.r.t. write ops. • Each read returns the value written by the immediate preceding write (or default value). From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

5. Example P1 and P2 access the same object whose default value is v0. W(v1) P1 P2 R(v0) From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

6. Example W(v1) P1 P2 R(v0) the virtual occurrence time (a.k.a. serialization point) From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

7. Counter-example W(v1) P1 P2 R(v0) P3 R(v1) R(v0) < W(v1) < R(v1) by property 4. But R(v0) > R(v1) by property 1 From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

8. Quorum of Servers • Quorums: mutually intersecting sets. • Example: line+column of a grid of nodes (quorum of size O(√n) ) • A client contacts a quorum of servers during an operation. • There is at least one element to testify prior operations => Atomicity Guarantee [ABD95] From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

9. Facing Dynamism 1. Structured Reconfiguration • Failures accumulate Operations: time: O(1) msg: O(√n) Reconfiguration: time: O(1) msg: O(n) clients servers From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

10. Facing Dynamism 2. Improving Structured Reconfiguration • Periodic reconfiguration to cope with accumulated failures. • RAMBO • Rapid reconfiguration to cope with high dynamism. • RDS • Low-cost reconfiguration to diminish bandwidth congestion. • [GMS05] From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

11. Facing Scalability 1. Restricting Knowledge to Locality • Adaptive probe [NW03, SQUARE] Operations: time: O(√n) msg: O(√n) Reconfiguration: time: O(1) msg: O(1) clients servers From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

12. Facing Scalability 2. Probabilistic Structured Approach • Random walks on DeBruijn Graph [AM03] Operations: time: O(log n) msg: O(√n log n) Reconfiguration: time: O(log n) msg: O(log n) clients / servers From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

13. Facing Scalability 3. Probabilistic Unstructured Approach • Disseminating Solution [GKMRS] Operations: time: O(log √n) msg: O(√n) Reconfiguration: 0 if random com. graph. What are the topologies that can afford lightweight operations? Random Graph, Small word…? clients / servers From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

14. Conclusion 1. Summary Notations:  not, ~ pseudo. From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

15. Conclusion 2. Future work PREVIOUS CHALLENGE: OPERATION COST Static Systems require Replication Dynamic Systems require Reconfiguration/Adaptation Large-Scale Dynamic Systems can not afford CostlyReconfiguration/Adaptation NEW CHALLENGE: RECONF/ADAPTATION COST What could be the operations that encompass reconf/adaptation? From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems

16. References [ABD95] Sharing Memory Robustly in Message-Passing Systems. H. Attiya, A. Bar-Noy, and D. Dolev. J.ACM 42(1) 1995. [RAMBO] Reconfgurable Atomic Memory in Dynamic Systems. DISC’02. N. Lynch and A. Shvartsman [AM03] Probabilistic Quorums for Dynamic Systems. Dist.Comp.’03.I. Abraham and D. Malkhi [NW03] Scalable and Dynamic Quorum Systems. PODC’03. M. Naor and U. Wieder [RDS]Reconfigurable Distributed Storage. OPODIS’05G. Chockler, S. Gilbert, V. Gramoli, P. Musial, and A. Shvartsman [GMS05] Operation Liveness and Gossip Mgt. in Distr. Atomic Data Sce. PDCS’05. V. Gramoli, P. Musial, and A. Shvartsman [SQUARE] Scalable Quorum-based Atomic Memory w/ Local Reconfiguration. TR1805 IRISA 2006. E. Anceaume, V. Gramoli, and A. Virgillito. [GKMRS] Core Persistence in P2P Systems. RDDS’06.V.Gramoli, A.-M.Kermarrec, A.Mostefaoui, M.Raynal, and B.Sericola. From Fast to Lightweight Atomic Memory for Large-Scale Dynamic Distributed Systems