Distributed Systems CS 15-440

Distributed SystemsCS 15-440 Consistency and Replication – Part IV Lecture 13, Oct 23, 2013 Mohammad Hammoud

Today… • Last Session: • Consistency and Replication- Past III • Client-centric consistency models and replica management • Today’s Session: • Consistency and Replication – Part IV • Consistency Protocols • Programming Models – Part I • Announcements: • Project II is due today by 11:59PM • Project III will be posted by tonight (and is due on Nov 6, 2013)

Overview • Consistency Models • Replica Management • Consistency Protocols

Consistency Protocols • A consistency protocol describes the implementation of a specific consistency model • We are going to study three consistency protocols: • Primary-based protocols • One primary coordinator is elected to control replication across multiple replicas • Replicated-write protocols • Multiple replicas coordinate together to provide consistency guarantees • Cache-coherence protocols • A special case of client-controlled replication

Overview of Consistency Protocols

Primary-based protocols • In Primary-based protocols, a simple centralized design is used to implement consistency models • Each data-item xhas an associated “Primary Replica” • The primary replica is responsible for coordinating write operations • We will study one example of Primary-based protocols that implements Sequential Consistency Model • Remote-Write Protocol

Remote-Write Protocol • Rules: • All write operations are forwarded to the primary replica • Read operations are carried out locally at each replica • Approach for write ops: (Budhirajaet al. 1993) • Client connects to some replica RC • If the client issues write operation to RC: • RC forwards the request to the primary replica RP • RP updates its local value • RP forwards the update to other replicas Ri • Other replicas Riupdate, and send an ACK back to RP • After RP receives all ACKs, it informs RC that the write operation is completed • RC acknowledges the client, which in return completes the write operation x+=5 Client 1 Primary server R1 R2 R3 x1=5 x1=0 x2=0 x3=5 x2=5 x3=0 Data-store

Remote-Write Protocol – Discussion • The Remote-Write protocol provides • A simple way to implement sequential consistency • Guarantees that clients see the most recent write operations • However, latency is high in Remote-Write Protocols • Clients block until all the replicas are updated • Can a non-blocking strategy be applied? • Remote-Write Protocols are applied to distributed databases and file systems that require fault-tolerance • Replicas are placed on the same LAN to reduce latency

Replicated-Write Protocol • In a replicated-write protocol, updates can be carried out at multiple replicas • We will study one example on replicated-write protocols called Active Replication Protocol • Here, clients write at any replica • The modified replica will propagate updates to other replicas

Active Replication Protocol • When a client writes at a replica, the replica will send the write operation updates to all other replicas • Challenges with Active Replication • Ordering of operations cannot be guaranteed across the replicas x+=2 x*=3 Client 1 Client 2 W(x) R(x)2 x+=2 R1 R(x)6 R(x)0 R(x)2 R2 R1 R2 W(x) R3 R(x)2 R(x)6 x*=3 R3 x1=0 x1=2 x1=6 x2=0 x2=2 x2=6 x3=0 x3=2 x3=6 Data-store

Centralized Active Replication Protocol • Approach • There is a centralized coordinator called the sequencer (Seq) • When a client connects to a replica RC and issues a write operation • RC forwards the update to the Seq • Seqassigns a sequence number to the update operation • RC propagates the sequence number and the operation to other replicas • Operations are carried out at all the replicas in the order defined by the sequencer x-=2 x+=5 Client 1 Client 2 10 R1 Seq R2 R3 11 11 10 x-=2 x+=5 Data-store

Cache Coherence Protocols • Caches are special types of replicas • Typically, caches are client-controlled replicas • Cache coherence refers to the consistency of data stored in caches • How are the cache coherence protocols in shared-memory multiprocessor (SMP) systems different from those in Distributed Systems? • Coherence protocols in SMP assume cache states can be broadcasted efficiently • In DS, this is difficult because caches may reside on different machines

Cache Coherence Protocols (Cont’d) • Cache Coherence protocols determine how caches are kept consistent • Caches may become inconsistent when a data item is modified: • at the server replicas, or • at the cache

When Data is Modified at the Server • Two approaches for enforcing coherence: • Server-initiated invalidation • Here, server sends all caches an invalidation message (when data item is modified) • Server updates the cache • Server will propagate the update to the cache

When Data is Modified at the Cache • The enforcement protocol may use one of three techniques: • Read-only cache • The cache does not modify the data in the cache • The update is propagated to the server replica • Write-through cache • Directly modify the cache, and forward the update to the server • Write-back cache • The client allows multiple writes to take place at the cache • The client batches a set of writes, and will send the batched write updates to the server replica

Summary of Consistency Protocols

Consistency and Replication – Brief Summary • Replication improves performance and fault-tolerance • However, replicas have to be kept reasonably consistent

Back-up Slides

Local-Write Protocols • Can we make Remote-Write better (e.g., for mobile computers that are able to operate in disconnected mode) • Approach: • Client connects to some replica RC • If the client issues write op to RC: • RC becomes the primary replica RP • Rest of the protocol is similar to Remote-Write x+=5 Client 1 Primary server R2 R1 R1 R2 R3 x1=0 x1=5 x2=0 x2=5 x3=0 x3=5 Data-store

Local-Write Protocol • Advantages • Primary replica can propagate a batch of write updates instead of an individual update • Multiple, successive write operations can be carried out locally, while reading processes can still access their local copies • Scenarios where Local-Write is applicable: • Mobile computers when disconnected (all updates can be carried locally, while other processes can still perform read operations- but not updates) • Scenarios where Local-Write is inappropriate: • When (multiple) clients are writing at multiple replicas • Overhead of reassigning primary replica is high

Two aspects of Cache Coherence Protocols • In order to maintain consistent caches, we need to perform two operations: • Coherence detection strategies • Detect inconsistent caches • Coherence enforcement strategies • Update caches

Coherence Detection Strategies • Detection strategies deal with predicting when caches are inconsistent • Since different replicas may be written by client processes, the protocol has to dynamically detect cache inconsistency

Coherence Detection Strategies • In a distributed system, cache inconsistencies can be typically detected at three stages: • Verify coherence before every access • Before every read or write operation • Verify coherence before a write access • Cache coherence is checked before every write operation • Verify coherence after a write access • First, an update is performed, and later cache consistency is verified • If cache was inconsistent • The write operation is rolled-back, and re-performed

Coherence Enforcement Strategies • Enforcement strategies determine how caches are kept consistent • Caches may become inconsistent when data item is modified: • at the server replicas, or • at the cache

When Data is Modified at the Server • Two approaches for enforcing coherence: • Server-initiated invalidation • Here, server sends all caches an invalidation message when data item is modified • Server updates the cache • Server will propagate the update to the cache

Distributed Systems CS 15-440